package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

Authors

Maintainers

Sources

alcotest-lwt-1.1.0.tbz
sha256=212827a49abf4008581c0da53e7ec78a9d639b415380dcb1fdeeb23f3ff083e2
sha512=c47d04b3c7100af703b470b93ff9fe9fe22790415370b6d5972736f46a5c83901717d67caf0c4115d01020d3078dc7f3063838578174921cab352546dad00148

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 03 Apr 2020

README

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Examples

A simple example (taken from examples/simple.ml):

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use ./simple.byte --help to see the runtime options.

$ ./simple.native
Testing Utils.
[OK]       string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
Test Successful in 0.001s. 4 tests run.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead. Test names may contain only alphanumeric characters, spaces, hyphens and underscores.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Screenshots

The following screenshots demonstrate the HTML testing output from the odoc project.

All tests passed Some tests failed Failed test with custom diffing
ok err diff

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)

  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (8)

  1. stdlib-shims
  2. re >= "1.7.2"
  3. uuidm
  4. cmdliner >= "1.0.3" & < "1.1.0"
  5. astring
  6. fmt >= "0.8.6"
  7. ocaml >= "4.03.0"
  8. dune >= "2.0"

Dev Dependencies (1)

  1. odoc with-doc

  1. ahrocksdb
  2. albatross >= "1.5.0"
  3. alcotest-async < "1.0.0" | = "1.1.0"
  4. alcotest-lwt < "1.0.0" | = "1.1.0"
  5. alg_structs_qcheck
  6. ambient-context
  7. ambient-context-eio
  8. ambient-context-lwt
  9. angstrom >= "0.7.0"
  10. ansi >= "0.6.0"
  11. anycache >= "0.7.4"
  12. anycache-async
  13. anycache-lwt
  14. archetype >= "1.4.2"
  15. archi
  16. arp
  17. arp-mirage
  18. arrakis
  19. art
  20. asak >= "0.2"
  21. asli >= "0.2.0"
  22. asn1-combinators >= "0.2.2"
  23. atd >= "2.3.3"
  24. atdgen >= "2.10.0"
  25. atdpy
  26. atdts
  27. base32
  28. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  29. bastet
  30. bastet_async
  31. bastet_lwt
  32. bech32
  33. bechamel >= "0.5.0"
  34. bigarray-overlap
  35. bigstring >= "0.3"
  36. bigstring-unix >= "0.3"
  37. bigstringaf
  38. bitlib
  39. blake2
  40. bloomf
  41. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  42. bls12-381-hash
  43. bls12-381-js >= "0.4.2"
  44. bls12-381-js-gen >= "0.4.2"
  45. bls12-381-legacy
  46. bls12-381-signature
  47. bls12-381-unix
  48. blurhash
  49. builder-web
  50. bulletml
  51. bytebuffer
  52. ca-certs
  53. ca-certs-nss
  54. cactus
  55. caldav
  56. calendar >= "3.0.0"
  57. callipyge
  58. camlix
  59. capnp-rpc < "1.2.3"
  60. capnp-rpc-lwt < "0.3"
  61. capnp-rpc-mirage >= "0.9.0"
  62. capnp-rpc-unix >= "0.9.0" & < "1.2.3"
  63. carray
  64. carton
  65. cborl
  66. ccss >= "1.6"
  67. cf-lwt
  68. chacha
  69. channel
  70. charrua-client
  71. charrua-client-lwt
  72. charrua-client-mirage < "0.11.0"
  73. checkseum >= "0.0.3"
  74. cid
  75. clarity-lang
  76. class_group_vdf
  77. cohttp >= "0.17.0"
  78. cohttp-curl-async
  79. cohttp-curl-lwt
  80. cohttp-eio >= "6.0.0~beta2"
  81. colombe >= "0.2.0"
  82. color
  83. conan
  84. conan-cli
  85. conan-database
  86. conan-lwt
  87. conan-unix
  88. conduit = "3.0.0"
  89. conex < "0.10.0"
  90. conex-mirage-crypto
  91. conex-nocrypto
  92. cookie
  93. cow >= "2.2.0"
  94. css
  95. css-parser
  96. cstruct >= "3.3.0"
  97. cstruct-sexp
  98. ctypes-zarith
  99. cuid
  100. curly
  101. current_incr
  102. cwe_checker
  103. data-encoding
  104. datakit >= "0.12.0"
  105. datakit-bridge-github >= "0.12.0"
  106. datakit-ci
  107. datakit-client-git >= "0.12.0"
  108. decompress >= "0.8" & < "1.5.3"
  109. depyt
  110. digestif >= "0.8.1"
  111. dispatch >= "0.4.1"
  112. dkim
  113. dkim-bin
  114. dkim-mirage
  115. dns >= "4.0.0"
  116. dns-cli
  117. dns-client >= "4.6.0"
  118. dns-forward < "0.9.0"
  119. dns-forward-lwt-unix
  120. dns-resolver
  121. dns-server
  122. dns-tsig
  123. dnssd
  124. dnssec
  125. docfd >= "2.2.0"
  126. dog < "0.2.1"
  127. domain-name
  128. dream
  129. dream-pure
  130. duff
  131. dune-release >= "1.0.0"
  132. duration >= "0.1.1"
  133. emile
  134. encore
  135. eqaf >= "0.5"
  136. equinoxe
  137. equinoxe-cohttp
  138. equinoxe-hlc
  139. eris
  140. eris-lwt
  141. ezgzip
  142. ezjsonm >= "0.4.2" & < "1.3.0"
  143. ezjsonm-lwt < "1.3.0"
  144. FPauth
  145. FPauth-core
  146. FPauth-responses
  147. FPauth-strategies
  148. faraday != "0.2.0"
  149. farfadet
  150. fat-filesystem >= "0.12.0"
  151. ff
  152. ff-pbt
  153. fiat-p256
  154. flex-array
  155. fsevents-lwt
  156. functoria >= "2.2.0"
  157. functoria-runtime >= "2.2.0" & != "3.0.1" & < "4.0.0~beta1"
  158. geojson
  159. geoml >= "0.1.1"
  160. git = "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0"
  161. git-mirage < "3.0.0"
  162. git-unix >= "1.10.0" & != "2.1.0"
  163. gitlab-unix
  164. glicko2
  165. gmap >= "0.3.0"
  166. gobba
  167. gpt
  168. graphql
  169. graphql-async
  170. graphql-cohttp >= "0.13.0"
  171. graphql-lwt
  172. graphql_parser != "0.11.0"
  173. graphql_ppx >= "0.7.1"
  174. h1_parser
  175. h2
  176. hacl
  177. hacl-star >= "0.6.0"
  178. hacl_func
  179. hacl_x25519 >= "0.2.0"
  180. highlexer
  181. hkdf
  182. hockmd
  183. html_of_jsx
  184. http
  185. http-multipart-formdata < "2.0.0"
  186. httpaf >= "0.2.0"
  187. hvsock
  188. icalendar >= "0.1.4"
  189. imagelib >= "20200929"
  190. index
  191. inferno >= "20220603"
  192. influxdb-async
  193. influxdb-lwt
  194. inquire < "0.2.0"
  195. interval-map
  196. iomux
  197. irmin < "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & != "2.3.0"
  198. irmin-bench >= "2.7.0"
  199. irmin-chunk < "1.3.0" | >= "2.3.0"
  200. irmin-cli
  201. irmin-containers
  202. irmin-fs < "1.3.0" | >= "2.3.0"
  203. irmin-git < "2.0.0" | >= "2.3.0"
  204. irmin-http < "2.0.0"
  205. irmin-mem < "1.3.0"
  206. irmin-pack >= "2.4.0" & != "2.6.1"
  207. irmin-pack-tools
  208. irmin-test >= "2.2.0" & < "3.0.0"
  209. irmin-tezos
  210. irmin-tezos-utils
  211. irmin-unix >= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
  212. irmin-watcher
  213. jekyll-format
  214. jerboa
  215. jitsu
  216. jose
  217. json-data-encoding >= "0.9"
  218. json_decoder
  219. jsonxt
  220. junit_alcotest
  221. jwto
  222. ke >= "0.2"
  223. kkmarkdown
  224. lambda-runtime
  225. lambda_streams
  226. lambda_streams_async
  227. lambdapi >= "2.0.0"
  228. lambdoc >= "1.0-beta4"
  229. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  230. letters
  231. lmdb >= "1.0"
  232. logical
  233. logtk >= "1.6"
  234. lp
  235. lp-glpk
  236. lp-glpk-js
  237. lp-gurobi
  238. lru
  239. lt-code
  240. luv
  241. mbr-format >= "1.0.0"
  242. mdx >= "1.6.0"
  243. mec
  244. mechaml >= "1.0.0"
  245. merge-queues >= "0.2.0"
  246. merge-ropes >= "0.2.0"
  247. metrics
  248. minicaml = "0.3.1" | >= "0.4"
  249. mirage >= "4.0.0~beta1"
  250. mirage-block-partition
  251. mirage-block-ramdisk >= "0.3"
  252. mirage-channel >= "4.0.0"
  253. mirage-channel-lwt
  254. mirage-crypto-ec
  255. mirage-flow >= "1.0.2" & < "1.2.0"
  256. mirage-flow-unix
  257. mirage-fs-mem
  258. mirage-fs-unix >= "1.2.0"
  259. mirage-kv >= "2.0.0"
  260. mirage-kv-mem
  261. mirage-kv-unix
  262. mirage-logs >= "0.3.0"
  263. mirage-nat
  264. mirage-net-unix >= "2.3.0"
  265. mirage-runtime >= "4.0.0~beta1"
  266. mirage-tc
  267. mjson
  268. mmdb
  269. mnd
  270. monocypher
  271. mrmime >= "0.2.0"
  272. mrt-format
  273. msgpck >= "1.6"
  274. mssql >= "2.0.3"
  275. multibase
  276. multihash
  277. multihash-digestif
  278. multipart-form-data
  279. multipart_form
  280. multipart_form-lwt
  281. named-pipe
  282. nanoid
  283. nbd >= "4.0.3"
  284. nbd-tool
  285. nloge
  286. nocoiner
  287. non_empty_list
  288. OCADml >= "0.6.0"
  289. ocaml-r >= "0.5.0"
  290. ocaml-version >= "3.1.0"
  291. ocamlformat >= "0.13.0" & != "0.19.0~4.13preview" & < "0.25.1"
  292. ocamlformat-rpc < "removed"
  293. ocamline
  294. ocluster < "0.3.0"
  295. odoc >= "1.4.0" & < "2.1.0"
  296. ohex
  297. oidc
  298. opam-0install
  299. opam-file-format >= "2.1.1"
  300. opentelemetry >= "0.6"
  301. opentelemetry-client-cohttp-lwt >= "0.6"
  302. opentelemetry-client-ocurl >= "0.6"
  303. opentelemetry-cohttp-lwt >= "0.6"
  304. opentelemetry-lwt >= "0.6"
  305. opium >= "0.15.0"
  306. opium-graphql
  307. opium-testing
  308. opium_kernel
  309. orewa
  310. ortac-core
  311. osx-acl
  312. osx-attr
  313. osx-cf
  314. osx-fsevents
  315. osx-membership
  316. osx-mount
  317. osx-xattr
  318. otoggl
  319. owl >= "0.6.0" & != "0.9.0" & != "1.0.0"
  320. owl-base < "0.5.0"
  321. owl-ode >= "0.1.0" & != "0.2.0"
  322. owl-symbolic
  323. passmaker
  324. patch
  325. pbkdf
  326. pecu >= "0.2"
  327. pf-qubes
  328. pg_query >= "0.9.6"
  329. pgx >= "1.0"
  330. pgx_unix >= "1.0"
  331. pgx_value_core
  332. pgx_value_ptime
  333. phylogenetics
  334. piaf
  335. polyglot
  336. polynomial
  337. ppx_blob >= "0.3.0"
  338. ppx_deriving_cmdliner
  339. ppx_deriving_rpc
  340. ppx_deriving_yaml
  341. ppx_graphql >= "0.2.0"
  342. ppx_inline_alcotest
  343. ppx_parser
  344. ppx_protocol_conv >= "5.0.0"
  345. ppx_protocol_conv_json >= "5.0.0"
  346. ppx_protocol_conv_jsonm >= "5.0.0"
  347. ppx_protocol_conv_msgpack >= "5.0.0"
  348. ppx_protocol_conv_xml_light >= "5.0.0"
  349. ppx_protocol_conv_xmlm
  350. ppx_protocol_conv_yaml >= "5.0.0"
  351. ppx_repr < "0.4.0"
  352. ppx_subliner
  353. ppx_units
  354. ppx_yojson >= "1.1.0"
  355. pratter
  356. prc
  357. preface
  358. pretty_expressive
  359. prettym
  360. proc-smaps
  361. producer < "0.2.0"
  362. progress < "0.2.0"
  363. prom
  364. prometheus < "1.2"
  365. prometheus-app
  366. protocell
  367. protocol-9p >= "0.3" & < "0.11.0" | >= "0.11.2"
  368. protocol-9p-unix
  369. psq
  370. qcheck >= "0.18"
  371. qcheck-alcotest
  372. qcheck-core >= "0.18"
  373. quickjs
  374. radis
  375. randii
  376. reason-standard
  377. reparse >= "2.0.0" & < "3.0.0"
  378. reparse-unix < "2.1.0"
  379. resp
  380. resp-unix >= "0.10.0"
  381. rfc1951 < "1.0.0"
  382. routes < "2.0.0"
  383. rpc >= "7.1.0"
  384. rpclib >= "7.1.0"
  385. rpclib-async
  386. rpclib-lwt >= "7.1.0"
  387. rubytt
  388. SZXX >= "4.0.0"
  389. salsa20
  390. salsa20-core
  391. sanddb >= "0.2"
  392. scaml >= "1.5.0"
  393. scrypt-kdf
  394. secp256k1 >= "0.4.1"
  395. secp256k1-internal
  396. semver >= "0.2.1"
  397. sendmail
  398. sendmail-lwt
  399. sendmsg
  400. server-reason-react
  401. session-cookie
  402. session-cookie-async
  403. session-cookie-lwt
  404. sherlodoc
  405. slug
  406. sodium-fmt
  407. spin >= "0.6.0"
  408. squirrel
  409. ssh-agent
  410. ssl >= "0.6.0"
  411. stramon-lib
  412. syslog-rfc5424
  413. tcpip >= "2.4.2" & < "4.0.0" | >= "5.0.1" & < "8.0.0"
  414. tdigest < "2.1.0"
  415. terminal_size >= "0.1.1"
  416. terminus
  417. terminus-cohttp
  418. terminus-hlc
  419. textrazor
  420. tezos-base-test-helpers < "13.0"
  421. tezos-bls12-381-polynomial
  422. tezos-client-base < "12.0"
  423. tezos-crypto >= "8.0" & < "9.0"
  424. tezos-lmdb
  425. tezos-plompiler = "0.1.3"
  426. tezos-plonk = "0.1.3"
  427. tezos-signer-backends >= "8.0" & < "13.0"
  428. tezos-stdlib >= "8.0" & < "12.0"
  429. tezos-test-helpers < "12.0"
  430. tftp
  431. timedesc
  432. timere
  433. tls >= "0.12.0"
  434. toc
  435. topojson
  436. topojsone
  437. transept
  438. twostep
  439. type_eq
  440. type_id
  441. typebeat
  442. typeid >= "1.0.1"
  443. tyre >= "0.4"
  444. tyxml >= "4.0.0"
  445. tyxml-jsx
  446. tyxml-ppx >= "4.3.0"
  447. tyxml-syntax
  448. uecc
  449. ulid
  450. unix-dirent
  451. unix-errno >= "0.3.0"
  452. unix-fcntl >= "0.3.0"
  453. unix-sys-resource
  454. unix-sys-stat
  455. unix-time
  456. unstrctrd
  457. user-agent-parser
  458. uspf
  459. uspf-lwt
  460. uspf-unix
  461. utop >= "2.13.0"
  462. validate
  463. validator
  464. vercel
  465. vpnkit
  466. websocketaf
  467. x509 >= "0.7.0"
  468. xapi-rrd >= "1.8.2"
  469. xapi-stdext-date
  470. xapi-stdext-encodings
  471. xapi-stdext-std >= "4.16.0"
  472. yaml < "3.2.0"
  473. yaml-sexp
  474. yocaml
  475. yocaml_yaml
  476. yojson >= "1.6.0"
  477. yuscii >= "0.3.0"
  478. zar
  479. zed >= "3.2.2"
  480. zlist < "0.4.0"

Conflicts

None