package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

Authors

Maintainers

Sources

alcotest-mirage-1.4.0.tbz
sha256=b1aaccfb2d651c902592c04953e2619169c91f797cf4f04a7dda2cab09b93ec1
sha512=8a13d5d4c8c77f115903e6b8e58160c6e6ec27870440bd38a674e9406f57f1eff299e65f006fd77728015d1a8f0ae30a714fe47e035824950a71ebfdff2cf3c9

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 16 Apr 2021

README

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run. See the manpage for details.

For information on contributing to Alcotest, see CONTRIBUTING.md.

Examples

A simple example (taken from examples/simple.ml):

Generated by the following test suite specification:

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use dune exec examples/simple.exe -- --help to see the runtime options.

Here's an example of a of failing test suite:

By default, only the first failing test log is printed to the console (and all test logs are captured on disk). Pass --show-errors to print all error messages.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)

  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (9)

  1. uutf >= "1.0.0"
  2. stdlib-shims
  3. re >= "1.7.2"
  4. uuidm
  5. cmdliner >= "1.0.3"
  6. astring
  7. fmt >= "0.8.7"
  8. ocaml >= "4.03.0"
  9. dune >= "2.2"

Dev Dependencies (1)

  1. cmdliner with-test & < "1.1.0"

  1. ahrocksdb
  2. albatross >= "1.5.0"
  3. alcotest-async < "1.0.0" | = "1.4.0"
  4. alcotest-lwt < "1.0.0" | = "1.4.0"
  5. alcotest-mirage = "1.4.0"
  6. alg_structs_qcheck
  7. ambient-context
  8. ambient-context-eio
  9. ambient-context-lwt
  10. angstrom >= "0.7.0"
  11. ansi >= "0.6.0"
  12. anycache >= "0.7.4"
  13. anycache-async
  14. anycache-lwt
  15. archetype >= "1.4.2"
  16. archi
  17. arp != "2.3.1"
  18. arp-mirage < "2.0.0"
  19. arrakis
  20. art
  21. asak >= "0.2"
  22. asli >= "0.2.0"
  23. asn1-combinators >= "0.2.2"
  24. atd >= "2.3.3"
  25. atdgen >= "2.10.0"
  26. atdpy
  27. atdts
  28. base32
  29. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  30. bastet
  31. bastet_async
  32. bastet_lwt
  33. bech32
  34. bechamel >= "0.5.0"
  35. bigarray-overlap
  36. bigstringaf
  37. bitlib
  38. blake2
  39. bloomf
  40. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  41. bls12-381-hash
  42. bls12-381-js >= "0.4.2"
  43. bls12-381-js-gen >= "0.4.2"
  44. bls12-381-legacy
  45. bls12-381-signature
  46. bls12-381-unix
  47. blurhash
  48. builder-web
  49. bulletml
  50. bytebuffer
  51. ca-certs
  52. ca-certs-nss
  53. cactus
  54. caldav
  55. calendar >= "3.0.0"
  56. callipyge
  57. camlix
  58. camlkit
  59. camlkit-base
  60. capnp-rpc < "1.2.3"
  61. capnp-rpc-lwt < "0.3"
  62. capnp-rpc-mirage >= "0.9.0"
  63. capnp-rpc-unix >= "0.9.0" & < "1.2.3"
  64. carray
  65. carton
  66. carton-git
  67. carton-lwt >= "0.4.1"
  68. cborl
  69. ccss >= "1.6"
  70. cf-lwt
  71. chacha
  72. channel
  73. charrua-client
  74. charrua-client-lwt
  75. charrua-client-mirage < "0.11.0"
  76. charrua-server >= "1.4.1"
  77. checked_oint < "0.1.1"
  78. checkseum >= "0.0.3"
  79. cid
  80. clarity-lang
  81. class_group_vdf
  82. cohttp >= "0.17.0"
  83. cohttp-curl-async
  84. cohttp-curl-lwt
  85. cohttp-eio >= "6.0.0~beta2"
  86. colombe >= "0.2.0"
  87. color
  88. conan
  89. conan-cli
  90. conan-database
  91. conan-lwt
  92. conan-unix
  93. conduit = "3.0.0"
  94. conex < "0.10.0"
  95. conex-mirage-crypto
  96. conex-nocrypto
  97. conformist
  98. cookie
  99. cow >= "2.2.0"
  100. css
  101. css-parser
  102. cstruct >= "3.3.0"
  103. cstruct-sexp
  104. ctypes-zarith
  105. cuid
  106. curly
  107. current >= "0.4"
  108. current-albatross-deployer
  109. current_git >= "0.6.4"
  110. current_incr
  111. cwe_checker
  112. data-encoding
  113. datakit >= "0.12.0"
  114. datakit-bridge-github >= "0.12.0"
  115. datakit-ci
  116. datakit-client-git >= "0.12.0"
  117. decompress >= "0.8" & < "1.5.3"
  118. depyt
  119. digestif >= "0.8.1"
  120. dirsp-exchange-kbb2017
  121. dirsp-proscript-mirage
  122. dirsp-ps2ocaml
  123. dispatch >= "0.4.1"
  124. dkim
  125. dkim-bin
  126. dkim-mirage
  127. dkml-install
  128. dkml-install-installer
  129. dkml-install-runner
  130. dkml-package-console
  131. dns >= "4.0.0"
  132. dns-cli
  133. dns-client >= "4.6.0"
  134. dns-forward < "0.9.0"
  135. dns-forward-lwt-unix
  136. dns-resolver
  137. dns-server
  138. dns-tsig
  139. dnssd
  140. dnssec
  141. docfd >= "2.2.0"
  142. dog < "0.2.1"
  143. domain-name
  144. dream
  145. dream-pure
  146. duff
  147. dune-release >= "1.0.0"
  148. duration >= "0.1.1"
  149. eio < "0.12"
  150. eio_linux < "0.12"
  151. eio_windows < "0.12"
  152. emile
  153. encore
  154. eqaf >= "0.5"
  155. equinoxe
  156. equinoxe-cohttp
  157. equinoxe-hlc
  158. eris
  159. eris-lwt
  160. ezgzip
  161. ezjsonm >= "0.4.2" & < "1.3.0"
  162. ezjsonm-lwt
  163. FPauth
  164. FPauth-core
  165. FPauth-responses
  166. FPauth-strategies
  167. faraday != "0.2.0"
  168. farfadet
  169. fat-filesystem >= "0.12.0"
  170. ff
  171. ff-pbt
  172. flex-array
  173. fsevents-lwt
  174. functoria >= "2.2.0"
  175. functoria-runtime >= "2.2.0" & < "3.0.1" | = "3.1.2"
  176. geojson
  177. geoml >= "0.1.1"
  178. git = "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0"
  179. git-cohttp
  180. git-cohttp-mirage
  181. git-cohttp-unix
  182. git-mirage
  183. git-unix >= "1.10.0" & != "2.1.0"
  184. gitlab-unix
  185. glicko2
  186. gmap >= "0.3.0"
  187. gobba
  188. gpt
  189. graphql
  190. graphql-async
  191. graphql-cohttp >= "0.13.0"
  192. graphql-lwt
  193. graphql_parser != "0.11.0"
  194. graphql_ppx >= "0.7.1"
  195. h1_parser
  196. h2
  197. hacl
  198. hacl-star >= "0.6.0"
  199. hacl_func
  200. hacl_x25519 >= "0.2.0"
  201. highlexer
  202. hkdf
  203. hockmd
  204. html_of_jsx
  205. http
  206. http-multipart-formdata < "2.0.0"
  207. httpaf >= "0.2.0"
  208. hvsock
  209. icalendar >= "0.1.4"
  210. imagelib >= "20200929"
  211. index
  212. inferno >= "20220603"
  213. influxdb-async
  214. influxdb-lwt
  215. inquire < "0.2.0"
  216. interval-map
  217. iomux
  218. irmin < "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & != "2.3.0"
  219. irmin-bench >= "2.7.0"
  220. irmin-chunk < "1.3.0" | >= "2.3.0"
  221. irmin-cli
  222. irmin-containers
  223. irmin-fs < "1.3.0" | >= "2.3.0"
  224. irmin-git < "2.0.0" | >= "2.3.0"
  225. irmin-graphql >= "2.3.0"
  226. irmin-http < "2.0.0"
  227. irmin-mem < "1.3.0" | >= "2.3.0"
  228. irmin-pack >= "2.4.0" & != "2.6.1"
  229. irmin-pack-tools
  230. irmin-test >= "2.2.0" & < "3.0.0"
  231. irmin-tezos
  232. irmin-tezos-utils
  233. irmin-unix >= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
  234. irmin-watcher != "0.3.0"
  235. jekyll-format
  236. jerboa
  237. jitsu
  238. jose
  239. json-data-encoding >= "0.9"
  240. json_decoder
  241. jsonxt
  242. junit_alcotest
  243. jwto
  244. ke >= "0.2"
  245. kkmarkdown
  246. lambda-runtime
  247. lambda_streams
  248. lambda_streams_async
  249. lambdapi >= "2.0.0"
  250. lambdoc >= "1.0-beta4"
  251. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  252. letters
  253. lmdb >= "1.0"
  254. logical
  255. logtk >= "1.6"
  256. lp
  257. lp-glpk
  258. lp-glpk-js
  259. lp-gurobi
  260. lru
  261. lt-code
  262. luv
  263. mbr-format >= "1.0.0"
  264. mdx >= "1.6.0"
  265. mec
  266. mechaml = "1.0.0" | >= "1.2.1"
  267. merge-queues >= "0.2.0"
  268. merge-ropes >= "0.2.0"
  269. metrics
  270. middleware
  271. mimic
  272. minicaml = "0.3.1" | >= "0.4"
  273. mirage >= "4.0.0~beta1"
  274. mirage-block-partition
  275. mirage-block-ramdisk = "0.3"
  276. mirage-channel >= "4.0.0"
  277. mirage-channel-lwt < "3.1.0"
  278. mirage-crypto-ec != "0.9.2"
  279. mirage-flow >= "1.0.2" & < "1.2.0"
  280. mirage-flow-unix != "1.3.0" & < "1.5.0" | = "2.0.0" | >= "3.0.0"
  281. mirage-fs-mem
  282. mirage-fs-unix >= "1.2.0" & < "1.4.1"
  283. mirage-kv >= "2.0.0"
  284. mirage-kv-mem
  285. mirage-kv-unix >= "3.0.0"
  286. mirage-logs >= "0.3.0"
  287. mirage-nat
  288. mirage-net-unix >= "2.3.0"
  289. mirage-runtime >= "4.0.0~beta1" & < "4.5.0"
  290. mirage-tc
  291. mjson
  292. mmdb < "0.3.0"
  293. mnd
  294. monocypher
  295. mrmime >= "0.2.0"
  296. mrt-format
  297. msgpck >= "1.6"
  298. mssql >= "2.0.3"
  299. multibase
  300. multihash
  301. multihash-digestif
  302. multipart-form-data
  303. multipart_form
  304. multipart_form-eio
  305. multipart_form-lwt
  306. named-pipe
  307. nanoid
  308. nbd >= "4.0.3"
  309. nbd-tool
  310. nloge
  311. nocoiner
  312. non_empty_list
  313. OCADml >= "0.6.0"
  314. ocaml-r >= "0.4.0"
  315. ocaml-version >= "3.1.0"
  316. ocamlformat >= "0.13.0" & != "0.19.0~4.13preview" & < "0.25.1"
  317. ocamlformat-lib
  318. ocamlformat-rpc < "removed"
  319. ocamline
  320. ocluster < "0.3.0"
  321. odoc >= "1.4.0" & < "2.1.0"
  322. ohex
  323. oidc
  324. opam-0install
  325. opam-compiler
  326. opam-file-format >= "2.1.1"
  327. opentelemetry >= "0.6"
  328. opentelemetry-client-cohttp-lwt >= "0.6"
  329. opentelemetry-client-ocurl >= "0.6"
  330. opentelemetry-cohttp-lwt >= "0.6"
  331. opentelemetry-lwt >= "0.6"
  332. opium >= "0.15.0"
  333. opium-graphql
  334. opium-testing
  335. opium_kernel
  336. orewa
  337. orgeat
  338. ortac-core
  339. osnap < "0.3.0"
  340. osx-acl
  341. osx-attr
  342. osx-cf
  343. osx-fsevents
  344. osx-membership
  345. osx-mount
  346. osx-xattr
  347. otoggl
  348. owl >= "0.6.0" & != "0.9.0" & != "1.0.0"
  349. owl-base < "0.5.0"
  350. owl-ode >= "0.1.0" & != "0.2.0"
  351. owl-symbolic
  352. passmaker
  353. patch
  354. pbkdf
  355. pecu >= "0.2"
  356. pf-qubes
  357. pg_query >= "0.9.6"
  358. pgx >= "1.0"
  359. pgx_unix >= "1.0"
  360. pgx_value_core
  361. pgx_value_ptime
  362. phylogenetics
  363. piaf
  364. polyglot
  365. polynomial
  366. ppx_blob >= "0.3.0"
  367. ppx_deriving_cmdliner
  368. ppx_deriving_qcheck
  369. ppx_deriving_rpc
  370. ppx_deriving_yaml
  371. ppx_graphql >= "0.2.0"
  372. ppx_inline_alcotest
  373. ppx_parser
  374. ppx_protocol_conv >= "5.0.0"
  375. ppx_protocol_conv_json >= "5.0.0"
  376. ppx_protocol_conv_jsonm >= "5.0.0"
  377. ppx_protocol_conv_msgpack >= "5.0.0"
  378. ppx_protocol_conv_xml_light >= "5.0.0"
  379. ppx_protocol_conv_xmlm
  380. ppx_protocol_conv_yaml >= "5.0.0"
  381. ppx_repr
  382. ppx_subliner
  383. ppx_units
  384. ppx_yojson >= "1.1.0"
  385. pratter
  386. prbnmcn-ucb1 >= "0.0.2"
  387. prc
  388. preface
  389. pretty_expressive
  390. prettym
  391. proc-smaps
  392. producer < "0.2.0"
  393. progress
  394. prom
  395. prometheus < "1.2"
  396. prometheus-app
  397. protocell
  398. protocol-9p >= "0.3" & < "0.11.0" | >= "0.11.2"
  399. protocol-9p-unix
  400. psq
  401. pyast
  402. qcheck >= "0.18"
  403. qcheck-alcotest
  404. qcheck-core >= "0.18"
  405. quickjs
  406. radis
  407. randii
  408. reason-standard
  409. reparse >= "2.0.0" & < "3.0.0"
  410. reparse-unix < "2.1.0"
  411. resp
  412. resp-unix >= "0.10.0"
  413. rfc1951 < "1.0.0"
  414. routes < "2.0.0"
  415. rpc >= "7.1.0"
  416. rpclib >= "7.1.0"
  417. rpclib-async
  418. rpclib-lwt >= "7.1.0"
  419. rubytt
  420. SZXX >= "4.0.0"
  421. salsa20
  422. salsa20-core
  423. sanddb >= "0.2"
  424. scaml >= "1.5.0"
  425. scrypt-kdf
  426. secp256k1 >= "0.4.1"
  427. secp256k1-internal
  428. semver >= "0.2.1"
  429. sendmail
  430. sendmail-lwt
  431. sendmsg
  432. server-reason-react
  433. session-cookie
  434. session-cookie-async
  435. session-cookie-lwt
  436. sherlodoc
  437. sihl < "0.2.0"
  438. sihl-type
  439. slug
  440. smol
  441. smol-helpers
  442. sodium-fmt
  443. solidity-alcotest
  444. spdx_licenses
  445. spectrum
  446. spin >= "0.7.0"
  447. squirrel
  448. ssh-agent
  449. ssl >= "0.6.0"
  450. stramon-lib
  451. styled-ppx
  452. syslog-rfc5424
  453. tcpip >= "2.4.2" & < "3.4.2" | >= "6.2.0" & < "7.0.0"
  454. tdigest < "2.1.0"
  455. terminal
  456. terminal_size >= "0.1.1"
  457. terminus
  458. terminus-cohttp
  459. terminus-hlc
  460. terml
  461. textmate-language >= "0.3.0"
  462. textrazor
  463. tezos-base-test-helpers < "13.0"
  464. tezos-bls12-381-polynomial
  465. tezos-client-base < "12.0"
  466. tezos-crypto >= "8.0" & < "9.0"
  467. tezos-lmdb
  468. tezos-plompiler = "0.1.3"
  469. tezos-plonk = "0.1.3"
  470. tezos-signer-backends >= "8.0" & < "13.0"
  471. tezos-stdlib >= "8.0" & < "12.0"
  472. tezos-test-helpers < "12.0"
  473. tftp
  474. timedesc
  475. timere
  476. timmy
  477. timmy-jsoo
  478. timmy-unix
  479. tls >= "0.12.0"
  480. toc
  481. topojson
  482. topojsone
  483. transept
  484. twostep
  485. type_eq
  486. type_id
  487. typebeat
  488. typeid >= "1.0.1"
  489. tyre >= "0.4"
  490. tyxml >= "4.0.0"
  491. tyxml-jsx
  492. tyxml-ppx >= "4.3.0"
  493. tyxml-syntax
  494. uecc
  495. ulid
  496. universal-portal
  497. unix-dirent
  498. unix-errno >= "0.3.0"
  499. unix-fcntl >= "0.3.0"
  500. unix-sys-resource
  501. unix-sys-stat
  502. unix-time
  503. unstrctrd
  504. uring < "0.4"
  505. user-agent-parser
  506. uspf
  507. uspf-lwt
  508. uspf-unix
  509. utop >= "2.13.0"
  510. validate
  511. validator
  512. vercel
  513. vpnkit
  514. wayland >= "2.0"
  515. wcwidth
  516. websocketaf
  517. x509 >= "0.7.0"
  518. xapi-rrd >= "1.8.2"
  519. xapi-stdext-date
  520. xapi-stdext-encodings
  521. xapi-stdext-std >= "4.16.0"
  522. yaml < "3.2.0"
  523. yaml-sexp
  524. yocaml
  525. yocaml_yaml
  526. yojson >= "1.6.0"
  527. yuscii >= "0.3.0"
  528. yuujinchou = "1.0.0"
  529. zar
  530. zed >= "3.2.2"
  531. zlist < "0.4.0"

Conflicts

None

OCaml

Innovation. Community. Security.