package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

github.com Readme Changelog ISC License Edit opam file Versions (41)

Authors

  1. T
    Thomas Gazagnaire

Maintainers

  1. T
    thomas@gazagnaire.org

Sources

alcotest-1.0.1.tbz
sha256=0c8748838a89df6dee4850aa7ef5e46c573265a9bf1589dec255bd8156a793f6
sha512=f5f52dea5bb143e7001b8d0ac6131f8851389b080f46b9ad1ccacb95cc31a38143dd7122ccba59bb190abe559dbf81f33cc4dc3401ed95772d59be75fa566f19

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 13 Feb 2020

README

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Examples

A simple example (taken from examples/simple.ml):

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use ./simple.byte --help to see the runtime options.

$ ./simple.native
Testing Utils.
[OK]       string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
Test Successful in 0.001s. 4 tests run.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead. Test names may contain only alphanumeric characters, spaces, hyphens and underscores.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Screenshots

The following screenshots demonstrate the HTML testing output from the odoc project.

All tests passed Some tests failed Failed test with custom diffing
ok err diff

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)

  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (8)

  1. stdlib-shims
  2. re >= "1.7.2"
  3. uuidm
  4. cmdliner >= "1.0.3"
  5. astring
  6. fmt >= "0.8.6"
  7. ocaml >= "4.03.0"
  8. dune >= "1.11"

Dev Dependencies

None

  1. ahrocksdb
  2. albatross >= "1.5.0"
  3. alcotest-async != "1.0.0" & < "1.1.0"
  4. alcotest-lwt != "1.0.0" & < "1.1.0"
  5. alg_structs_qcheck
  6. ambient-context
  7. ambient-context-eio
  8. angstrom >= "0.7.0"
  9. ansi >= "0.6.0"
  10. anycache >= "0.7.4"
  11. anycache-async
  12. anycache-lwt
  13. archetype >= "1.4.2"
  14. archi
  15. arp
  16. arp-mirage
  17. arrakis
  18. art
  19. asak >= "0.2"
  20. asli >= "0.2.0"
  21. asn1-combinators >= "0.2.2"
  22. atd >= "2.3.3"
  23. atdgen >= "2.10.0"
  24. atdpy
  25. atdts
  26. base32
  27. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  28. bastet
  29. bastet_async
  30. bastet_lwt
  31. bechamel >= "0.5.0"
  32. bigarray-overlap
  33. bigstring >= "0.3"
  34. bigstring-unix >= "0.3"
  35. bigstringaf
  36. bitlib
  37. blake2
  38. bloomf
  39. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  40. bls12-381-hash
  41. bls12-381-js >= "0.4.2"
  42. bls12-381-js-gen >= "0.4.2"
  43. bls12-381-legacy
  44. bls12-381-signature
  45. bls12-381-unix
  46. blurhash
  47. builder-web < "0.2.0"
  48. bulletml
  49. bytebuffer
  50. ca-certs
  51. ca-certs-nss
  52. cactus
  53. caldav
  54. calendar >= "3.0.0"
  55. callipyge
  56. camlix
  57. camlkit
  58. camlkit-base
  59. capnp-rpc < "1.2.3"
  60. capnp-rpc-lwt < "0.3"
  61. capnp-rpc-mirage >= "0.9.0"
  62. capnp-rpc-unix >= "0.9.0" & < "1.2.3"
  63. carray
  64. carton
  65. cborl
  66. ccss >= "1.6"
  67. cf-lwt
  68. chacha
  69. channel
  70. charrua-client
  71. charrua-client-lwt
  72. charrua-client-mirage < "0.11.0"
  73. checkseum >= "0.0.3"
  74. cid
  75. clarity-lang
  76. class_group_vdf
  77. cohttp >= "0.17.0"
  78. cohttp-curl-async
  79. cohttp-curl-lwt
  80. cohttp-eio >= "6.0.0~beta2"
  81. colombe >= "0.2.0"
  82. color
  83. conan
  84. conan-cli
  85. conan-database
  86. conan-lwt
  87. conan-unix
  88. conduit = "3.0.0"
  89. conex < "0.10.0"
  90. conex-mirage-crypto
  91. conex-nocrypto
  92. cookie
  93. cow >= "2.2.0"
  94. css
  95. css-parser
  96. cstruct >= "3.3.0"
  97. cstruct-sexp
  98. ctypes-zarith
  99. cuid
  100. curly
  101. current_incr
  102. cwe_checker
  103. data-encoding
  104. datakit >= "0.12.0"
  105. datakit-bridge-github >= "0.12.0"
  106. datakit-ci
  107. datakit-client-git >= "0.12.0"
  108. decompress >= "0.8" & < "1.5.3"
  109. depyt
  110. digestif >= "0.8.1"
  111. dispatch >= "0.4.1"
  112. dkim
  113. dkim-bin
  114. dkim-mirage
  115. dns >= "4.0.0"
  116. dns-cli
  117. dns-client >= "4.6.0"
  118. dns-forward < "0.9.0"
  119. dns-forward-lwt-unix
  120. dns-resolver
  121. dns-server
  122. dns-tsig
  123. dnssd
  124. dnssec
  125. docfd >= "2.2.0"
  126. dog < "0.2.1"
  127. domain-name
  128. dot-merlin-reader >= "5.3~5.3preview"
  129. dream
  130. dream-pure
  131. duff
  132. dune-release >= "1.0.0"
  133. duration >= "0.1.1"
  134. emile
  135. encore
  136. eqaf >= "0.5"
  137. equinoxe
  138. equinoxe-cohttp
  139. equinoxe-hlc
  140. eris
  141. eris-lwt
  142. ezgzip
  143. ezjsonm >= "0.4.2" & < "1.3.0"
  144. ezjsonm-lwt < "1.3.0"
  145. FPauth
  146. FPauth-core
  147. FPauth-responses
  148. FPauth-strategies
  149. faraday != "0.2.0"
  150. farfadet
  151. fat-filesystem >= "0.12.0"
  152. ff
  153. ff-pbt
  154. fiat-p256
  155. flex-array
  156. fsevents-lwt
  157. functoria >= "2.2.0"
  158. functoria-runtime >= "2.2.0" & != "3.0.1" & < "4.0.0~beta1"
  159. geojson
  160. geoml >= "0.1.1"
  161. git = "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0" & < "3.0.0"
  162. git-mirage < "3.0.0"
  163. git-split
  164. git-unix >= "1.10.0" & != "2.1.0" & < "3.0.0"
  165. git_split
  166. gitlab-unix
  167. glicko2
  168. gmap >= "0.3.0"
  169. gobba
  170. gpt
  171. graphql
  172. graphql-async
  173. graphql-cohttp >= "0.13.0"
  174. graphql-lwt
  175. graphql_parser != "0.11.0"
  176. graphql_ppx >= "0.7.1"
  177. h1_parser
  178. h2
  179. hacl
  180. hacl_func
  181. hacl_x25519 >= "0.2.0"
  182. highlexer
  183. hkdf
  184. hockmd
  185. html_of_jsx
  186. http
  187. http-multipart-formdata < "2.0.0"
  188. httpaf >= "0.2.0"
  189. httpun
  190. httpun-ws
  191. hvsock
  192. icalendar >= "0.1.4"
  193. imagelib >= "20200929"
  194. index
  195. inferno >= "20220603"
  196. influxdb-async
  197. influxdb-lwt
  198. inquire < "0.2.0"
  199. interval-map
  200. iomux
  201. irmin < "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & < "2.3.0"
  202. irmin-bench >= "2.7.0"
  203. irmin-chunk < "1.3.0" | >= "2.3.0"
  204. irmin-cli
  205. irmin-containers
  206. irmin-fs < "1.3.0" | >= "2.3.0"
  207. irmin-git < "2.0.0" | >= "2.3.0"
  208. irmin-http < "2.0.0"
  209. irmin-mem < "1.3.0"
  210. irmin-pack >= "2.4.0" & != "2.6.1"
  211. irmin-pack-tools
  212. irmin-test >= "2.2.0" & < "3.0.0"
  213. irmin-tezos
  214. irmin-tezos-utils
  215. irmin-unix >= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
  216. irmin-watcher
  217. jekyll-format
  218. jerboa
  219. jitsu
  220. jose
  221. json-data-encoding >= "0.9"
  222. json_decoder
  223. jsonxt
  224. junit_alcotest
  225. jwto
  226. kdf
  227. ke >= "0.2"
  228. kkmarkdown
  229. lambda-runtime
  230. lambda_streams
  231. lambda_streams_async
  232. lambda_streams_lwt
  233. lambdapi >= "2.0.0"
  234. lambdoc >= "1.0-beta4"
  235. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  236. lmdb >= "1.0"
  237. logical
  238. logtk >= "1.6"
  239. lp
  240. lp-glpk
  241. lp-glpk-js
  242. lp-gurobi
  243. lru
  244. lt-code
  245. luv
  246. mbr-format >= "1.0.0"
  247. mdx >= "1.6.0"
  248. mec
  249. mechaml >= "1.0.0"
  250. merge-queues >= "0.2.0"
  251. merge-ropes >= "0.2.0"
  252. merlin >= "4.17.1-414" & < "5.0-502" | >= "5.2.1-502"
  253. merlin-lib >= "4.17.1-414" & < "5.0-502" | >= "5.2.1-502"
  254. metrics
  255. minicaml = "0.3.1" | >= "0.4"
  256. mirage >= "4.0.0~beta1"
  257. mirage-block-partition
  258. mirage-block-ramdisk >= "0.3"
  259. mirage-channel >= "4.0.0"
  260. mirage-channel-lwt
  261. mirage-crypto-ec
  262. mirage-flow >= "1.0.2" & < "1.2.0"
  263. mirage-flow-unix
  264. mirage-fs-mem
  265. mirage-fs-unix >= "1.2.0"
  266. mirage-kv >= "2.0.0"
  267. mirage-kv-mem
  268. mirage-kv-unix
  269. mirage-logs >= "0.3.0"
  270. mirage-nat
  271. mirage-net-unix >= "2.3.0"
  272. mirage-runtime >= "4.0.0~beta1" & < "4.5.0"
  273. mirage-tc
  274. mjson
  275. mmdb
  276. mnd
  277. monocypher
  278. mrmime >= "0.2.0"
  279. mrt-format
  280. msgpck >= "1.6"
  281. mssql >= "2.0.3"
  282. multibase
  283. multihash
  284. multihash-digestif
  285. multipart-form-data
  286. multipart_form
  287. multipart_form-eio
  288. multipart_form-lwt
  289. named-pipe
  290. nanoid
  291. nbd >= "4.0.3"
  292. nbd-tool
  293. nloge
  294. nocoiner
  295. non_empty_list
  296. OCADml >= "0.6.0"
  297. obatcher
  298. ocaml-index >= "1.1"
  299. ocaml-r >= "0.5.0"
  300. ocaml-version >= "3.1.0"
  301. ocamlformat >= "0.13.0" & != "0.19.0~4.13preview" & < "0.25.1"
  302. ocamlformat-rpc < "removed"
  303. ocamline
  304. ocluster < "0.3.0"
  305. odoc >= "1.4.0" & < "2.1.0"
  306. ohex
  307. oidc
  308. opam-0install
  309. opam-0install-cudf >= "0.5.0"
  310. opam-file-format >= "2.1.1"
  311. opentelemetry >= "0.6"
  312. opentelemetry-client-cohttp-lwt >= "0.6"
  313. opentelemetry-client-ocurl >= "0.6"
  314. opentelemetry-cohttp-lwt >= "0.6"
  315. opentelemetry-lwt >= "0.6"
  316. opium >= "0.15.0"
  317. opium-graphql
  318. opium-testing
  319. opium_kernel
  320. orewa
  321. ortac-core
  322. osx-acl
  323. osx-attr
  324. osx-cf
  325. osx-fsevents
  326. osx-membership
  327. osx-mount
  328. osx-xattr
  329. otoggl
  330. owl >= "0.6.0" & != "0.9.0" & != "1.0.0"
  331. owl-base < "0.5.0"
  332. owl-ode >= "0.1.0" & != "0.2.0"
  333. owl-symbolic
  334. passmaker
  335. patch
  336. pbkdf
  337. pecu >= "0.2"
  338. pf-qubes
  339. pg_query >= "0.9.6"
  340. pgx >= "1.0"
  341. pgx_unix >= "1.0"
  342. pgx_value_core
  343. pgx_value_ptime < "2.2"
  344. phylogenetics
  345. piaf
  346. polyglot
  347. polynomial
  348. ppx_blob >= "0.3.0"
  349. ppx_deriving_cmdliner
  350. ppx_deriving_rpc
  351. ppx_deriving_yaml
  352. ppx_graphql >= "0.2.0"
  353. ppx_inline_alcotest
  354. ppx_protocol_conv >= "5.0.0"
  355. ppx_protocol_conv_json >= "5.0.0"
  356. ppx_protocol_conv_jsonm >= "5.0.0"
  357. ppx_protocol_conv_msgpack >= "5.0.0"
  358. ppx_protocol_conv_xml_light >= "5.0.0"
  359. ppx_protocol_conv_xmlm
  360. ppx_protocol_conv_yaml >= "5.0.0"
  361. ppx_subliner
  362. ppx_units
  363. ppx_yojson >= "1.1.0"
  364. pratter
  365. prc
  366. preface
  367. pretty_expressive
  368. prettym
  369. proc-smaps
  370. producer < "0.2.0"
  371. prom
  372. prometheus < "1.2"
  373. prometheus-app
  374. protocell
  375. protocol-9p >= "0.3" & < "0.11.0" | >= "0.11.2"
  376. protocol-9p-unix
  377. psq
  378. qcheck >= "0.18" & < "0.22"
  379. qcheck-alcotest < "0.22"
  380. quickjs
  381. radis
  382. randii
  383. reason-standard
  384. red-black-tree
  385. reparse >= "2.0.0" & < "3.0.0"
  386. reparse-unix < "2.1.0"
  387. resp
  388. resp-unix >= "0.10.0"
  389. rfc1951 < "1.0.0"
  390. routes < "2.0.0"
  391. rpc >= "7.1.0"
  392. rpclib >= "7.1.0"
  393. rpclib-async
  394. rpclib-lwt >= "7.1.0"
  395. rpmfile < "0.3.0"
  396. rpmfile-eio
  397. rpmfile-unix
  398. rubytt
  399. SZXX >= "4.0.0"
  400. salsa20
  401. salsa20-core
  402. sanddb >= "0.2"
  403. scaml >= "1.5.0"
  404. scrypt-kdf
  405. secp256k1 >= "0.4.1"
  406. secp256k1-internal
  407. semver >= "0.2.1"
  408. sendmail
  409. sendmail-lwt
  410. sendmail-miou-unix
  411. sendmail-mirage
  412. sendmsg
  413. server-reason-react
  414. session-cookie
  415. session-cookie-async
  416. session-cookie-lwt
  417. sherlodoc
  418. slug
  419. sodium-fmt
  420. spin >= "0.6.0"
  421. squirrel
  422. ssh-agent
  423. ssl >= "0.6.0"
  424. stramon-lib
  425. styled-ppx
  426. swapfs
  427. syslog-rfc5424
  428. tcpip >= "2.4.2" & < "4.0.0" | >= "5.0.1" & < "7.0.0"
  429. tdigest < "2.1.0"
  430. term-indexing
  431. term-tools
  432. terminal_size >= "0.1.1"
  433. terminus
  434. terminus-cohttp
  435. terminus-hlc
  436. terml
  437. testo
  438. testo-lwt
  439. textrazor
  440. tezos-base-test-helpers < "13.0"
  441. tezos-client-base < "12.0"
  442. tezos-lmdb
  443. tezos-test-helpers < "11.0"
  444. tftp
  445. timedesc
  446. timere
  447. tls >= "0.12.0"
  448. toc
  449. topojson
  450. topojsone
  451. transept
  452. twostep
  453. type_eq
  454. type_id
  455. typebeat
  456. typeid >= "1.0.1"
  457. tyre >= "0.4"
  458. tyxml >= "4.0.0"
  459. tyxml-jsx
  460. tyxml-ppx >= "4.3.0"
  461. tyxml-syntax
  462. uecc
  463. ulid
  464. universal-portal
  465. unix-dirent
  466. unix-errno >= "0.3.0"
  467. unix-fcntl >= "0.3.0"
  468. unix-sys-resource
  469. unix-sys-stat
  470. unix-time
  471. unstrctrd
  472. user-agent-parser
  473. uspf
  474. uspf-lwt
  475. uspf-mirage
  476. uspf-unix
  477. utop >= "2.13.0"
  478. validate
  479. validator
  480. vercel
  481. vpnkit
  482. wcwidth
  483. websocketaf
  484. x509 >= "0.7.0"
  485. xapi-rrd >= "1.8.2"
  486. xapi-stdext-date
  487. xapi-stdext-encodings
  488. xapi-stdext-std >= "4.16.0"
  489. yaml < "3.2.0"
  490. yaml-sexp
  491. yocaml < "2.0.0"
  492. yocaml_syndication = "2.0.0"
  493. yocaml_yaml < "2.0.0"
  494. yojson >= "1.6.0"
  495. yojson-five
  496. yuscii >= "0.3.0"
  497. zar
  498. zed >= "3.2.2"
  499. zlist < "0.4.0"

Conflicts

None

OCaml

Innovation. Community. Security.

GitHub Discord Twitter Peertube RSS
About
Changelog Releases Industrial Users Academic Users Why OCaml
Ecosystem
Packages Community Events OCaml Planet Jobs
Resources
Install OCaml Get Started Platform Tools Language Manual Standard Library API Books Exercises Papers OCaml Playground Logo
Policies
Carbon Footprint Governance Privacy Code of Conduct