package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

github.com Readme Changelog ISC License Edit opam file Versions (41)

Authors

  1. T
    Thomas Gazagnaire

Maintainers

  1. T
    thomas@gazagnaire.org

Sources

alcotest-lwt-1.1.0.tbz
sha256=212827a49abf4008581c0da53e7ec78a9d639b415380dcb1fdeeb23f3ff083e2
sha512=c47d04b3c7100af703b470b93ff9fe9fe22790415370b6d5972736f46a5c83901717d67caf0c4115d01020d3078dc7f3063838578174921cab352546dad00148

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 03 Apr 2020

README

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Examples

A simple example (taken from examples/simple.ml):

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use ./simple.byte --help to see the runtime options.

$ ./simple.native
Testing Utils.
[OK]       string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
Test Successful in 0.001s. 4 tests run.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead. Test names may contain only alphanumeric characters, spaces, hyphens and underscores.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Screenshots

The following screenshots demonstrate the HTML testing output from the odoc project.

All tests passed Some tests failed Failed test with custom diffing
ok err diff

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)

  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (8)

  1. stdlib-shims
  2. re >= "1.7.2"
  3. uuidm
  4. cmdliner >= "1.0.3" & < "1.1.0"
  5. astring
  6. fmt >= "0.8.6"
  7. ocaml >= "4.03.0"
  8. dune >= "2.0"

Dev Dependencies (1)

  1. odoc with-doc

  1. ahrocksdb
  2. albatross >= "1.5.0"
  3. alcotest-async < "1.0.0" | = "1.1.0"
  4. alcotest-lwt < "1.0.0" | = "1.1.0"
  5. alg_structs_qcheck
  6. ambient-context
  7. ambient-context-eio
  8. angstrom >= "0.7.0"
  9. ansi >= "0.6.0"
  10. anycache >= "0.7.4"
  11. anycache-async
  12. anycache-lwt
  13. archetype >= "1.4.2"
  14. archi
  15. arp
  16. arp-mirage
  17. arrakis
  18. art
  19. asak >= "0.2"
  20. asli >= "0.2.0"
  21. asn1-combinators >= "0.2.2"
  22. atd >= "2.3.3"
  23. atdgen >= "2.10.0"
  24. atdpy
  25. atdts
  26. base32
  27. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  28. bastet
  29. bastet_async
  30. bastet_lwt
  31. bech32
  32. bechamel >= "0.5.0"
  33. bigarray-overlap
  34. bigstring >= "0.3"
  35. bigstring-unix >= "0.3"
  36. bigstringaf
  37. bitlib
  38. blake2
  39. bloomf
  40. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  41. bls12-381-hash
  42. bls12-381-js >= "0.4.2"
  43. bls12-381-js-gen >= "0.4.2"
  44. bls12-381-legacy
  45. bls12-381-signature
  46. bls12-381-unix
  47. blurhash
  48. builder-web
  49. bulletml
  50. bytebuffer
  51. ca-certs
  52. ca-certs-nss
  53. cactus
  54. caldav
  55. calendar >= "3.0.0"
  56. callipyge
  57. camlix
  58. camlkit
  59. camlkit-base
  60. capnp-rpc < "1.2.3"
  61. capnp-rpc-lwt < "0.3"
  62. capnp-rpc-mirage >= "0.9.0"
  63. capnp-rpc-unix >= "0.9.0" & < "1.2.3"
  64. carray
  65. carton
  66. cborl
  67. ccss >= "1.6"
  68. cf-lwt
  69. chacha
  70. channel
  71. charrua-client
  72. charrua-client-lwt
  73. charrua-client-mirage < "0.11.0"
  74. checked_oint < "0.1.1"
  75. checkseum >= "0.0.3"
  76. cid
  77. clarity-lang
  78. class_group_vdf
  79. cohttp >= "0.17.0"
  80. cohttp-curl-async
  81. cohttp-curl-lwt
  82. cohttp-eio >= "6.0.0~beta2"
  83. colombe >= "0.2.0"
  84. color
  85. conan
  86. conan-cli
  87. conan-database
  88. conan-lwt
  89. conan-unix
  90. conduit = "3.0.0"
  91. conex < "0.10.0"
  92. conex-mirage-crypto
  93. conex-nocrypto
  94. cookie
  95. cow >= "2.2.0"
  96. css
  97. css-parser
  98. cstruct >= "3.3.0"
  99. cstruct-sexp
  100. ctypes-zarith
  101. cuid
  102. curly
  103. current_incr
  104. cwe_checker
  105. data-encoding
  106. datakit >= "0.12.0"
  107. datakit-bridge-github >= "0.12.0"
  108. datakit-ci
  109. datakit-client-git >= "0.12.0"
  110. decompress >= "0.8" & < "1.5.3"
  111. depyt
  112. digestif >= "0.8.1"
  113. dispatch >= "0.4.1"
  114. dkim
  115. dkim-bin
  116. dkim-mirage
  117. dns >= "4.0.0"
  118. dns-cli
  119. dns-client >= "4.6.0"
  120. dns-forward < "0.9.0"
  121. dns-forward-lwt-unix
  122. dns-resolver
  123. dns-server
  124. dns-tsig
  125. dnssd
  126. dnssec
  127. docfd >= "2.2.0"
  128. dog < "0.2.1"
  129. domain-name
  130. dream
  131. dream-pure
  132. duff
  133. dune-release >= "1.0.0"
  134. duration >= "0.1.1"
  135. emile
  136. encore
  137. eqaf >= "0.5"
  138. equinoxe
  139. equinoxe-cohttp
  140. equinoxe-hlc
  141. eris
  142. eris-lwt
  143. ezgzip
  144. ezjsonm >= "0.4.2" & < "1.3.0"
  145. ezjsonm-lwt < "1.3.0"
  146. FPauth
  147. FPauth-core
  148. FPauth-responses
  149. FPauth-strategies
  150. faraday != "0.2.0"
  151. farfadet
  152. fat-filesystem >= "0.12.0"
  153. ff
  154. ff-pbt
  155. fiat-p256
  156. flex-array
  157. fsevents-lwt
  158. functoria >= "2.2.0"
  159. functoria-runtime >= "2.2.0" & != "3.0.1" & < "4.0.0~beta1"
  160. geojson
  161. geoml >= "0.1.1"
  162. git = "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0"
  163. git-mirage < "3.0.0"
  164. git-unix >= "1.10.0" & != "2.1.0"
  165. git_split
  166. gitlab-unix
  167. glicko2
  168. gmap >= "0.3.0"
  169. gobba
  170. gpt
  171. graphql
  172. graphql-async
  173. graphql-cohttp >= "0.13.0"
  174. graphql-lwt
  175. graphql_parser != "0.11.0"
  176. graphql_ppx >= "0.7.1"
  177. h1_parser
  178. h2
  179. hacl
  180. hacl-star >= "0.6.0"
  181. hacl_func
  182. hacl_x25519 >= "0.2.0"
  183. highlexer
  184. hkdf
  185. hockmd
  186. html_of_jsx
  187. http
  188. http-multipart-formdata < "2.0.0"
  189. httpaf >= "0.2.0"
  190. httpun
  191. httpun-ws
  192. hvsock
  193. icalendar >= "0.1.4"
  194. imagelib >= "20200929"
  195. index
  196. inferno >= "20220603"
  197. influxdb-async
  198. influxdb-lwt
  199. inquire < "0.2.0"
  200. interval-map
  201. iomux
  202. irmin < "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & != "2.3.0"
  203. irmin-bench >= "2.7.0"
  204. irmin-chunk < "1.3.0" | >= "2.3.0"
  205. irmin-cli
  206. irmin-containers
  207. irmin-fs < "1.3.0" | >= "2.3.0"
  208. irmin-git < "2.0.0" | >= "2.3.0"
  209. irmin-http < "2.0.0"
  210. irmin-mem < "1.3.0"
  211. irmin-pack >= "2.4.0" & != "2.6.1"
  212. irmin-pack-tools
  213. irmin-test >= "2.2.0" & < "3.0.0"
  214. irmin-tezos
  215. irmin-tezos-utils
  216. irmin-unix >= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
  217. irmin-watcher
  218. jekyll-format
  219. jerboa
  220. jitsu
  221. jose
  222. json-data-encoding >= "0.9"
  223. json_decoder
  224. jsonxt
  225. junit_alcotest
  226. jwto
  227. ke >= "0.2"
  228. kkmarkdown
  229. lambda-runtime
  230. lambda_streams
  231. lambda_streams_async
  232. lambdapi >= "2.0.0"
  233. lambdoc >= "1.0-beta4"
  234. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  235. letters
  236. lmdb >= "1.0"
  237. logical
  238. logtk >= "1.6"
  239. lp
  240. lp-glpk
  241. lp-glpk-js
  242. lp-gurobi
  243. lru
  244. lt-code
  245. luv
  246. mbr-format >= "1.0.0"
  247. mdx >= "1.6.0"
  248. mec
  249. mechaml >= "1.0.0"
  250. merge-queues >= "0.2.0"
  251. merge-ropes >= "0.2.0"
  252. metrics
  253. minicaml = "0.3.1" | >= "0.4"
  254. mirage >= "4.0.0~beta1"
  255. mirage-block-partition
  256. mirage-block-ramdisk >= "0.3"
  257. mirage-channel >= "4.0.0"
  258. mirage-channel-lwt
  259. mirage-crypto-ec
  260. mirage-flow >= "1.0.2" & < "1.2.0"
  261. mirage-flow-unix
  262. mirage-fs-mem
  263. mirage-fs-unix >= "1.2.0"
  264. mirage-kv >= "2.0.0"
  265. mirage-kv-mem
  266. mirage-kv-unix
  267. mirage-logs >= "0.3.0"
  268. mirage-nat
  269. mirage-net-unix >= "2.3.0"
  270. mirage-runtime >= "4.0.0~beta1" & < "4.5.0"
  271. mirage-tc
  272. mjson
  273. mmdb
  274. mnd
  275. monocypher
  276. mrmime >= "0.2.0"
  277. mrt-format
  278. msgpck >= "1.6"
  279. mssql >= "2.0.3"
  280. multibase
  281. multihash
  282. multihash-digestif
  283. multipart-form-data
  284. multipart_form
  285. multipart_form-eio
  286. multipart_form-lwt
  287. named-pipe
  288. nanoid
  289. nbd >= "4.0.3"
  290. nbd-tool
  291. nloge
  292. nocoiner
  293. non_empty_list
  294. OCADml >= "0.6.0"
  295. ocaml-r >= "0.5.0"
  296. ocaml-version >= "3.1.0"
  297. ocamlformat >= "0.13.0" & != "0.19.0~4.13preview" & < "0.25.1"
  298. ocamlformat-rpc < "removed"
  299. ocamline
  300. ocluster < "0.3.0"
  301. odoc >= "1.4.0" & < "2.1.0"
  302. ohex
  303. oidc
  304. opam-0install
  305. opam-file-format >= "2.1.1"
  306. opentelemetry >= "0.6"
  307. opentelemetry-client-cohttp-lwt >= "0.6"
  308. opentelemetry-client-ocurl >= "0.6"
  309. opentelemetry-cohttp-lwt >= "0.6"
  310. opentelemetry-lwt >= "0.6"
  311. opium >= "0.15.0"
  312. opium-graphql
  313. opium-testing
  314. opium_kernel
  315. orewa
  316. ortac-core
  317. osx-acl
  318. osx-attr
  319. osx-cf
  320. osx-fsevents
  321. osx-membership
  322. osx-mount
  323. osx-xattr
  324. otoggl
  325. owl >= "0.6.0" & != "0.9.0" & != "1.0.0"
  326. owl-base < "0.5.0"
  327. owl-ode >= "0.1.0" & != "0.2.0"
  328. owl-symbolic
  329. passmaker
  330. patch
  331. pbkdf
  332. pecu >= "0.2"
  333. pf-qubes
  334. pg_query >= "0.9.6"
  335. pgx >= "1.0"
  336. pgx_unix >= "1.0"
  337. pgx_value_core
  338. pgx_value_ptime < "2.2"
  339. phylogenetics
  340. piaf
  341. polyglot
  342. polynomial
  343. ppx_blob >= "0.3.0"
  344. ppx_deriving_cmdliner
  345. ppx_deriving_rpc
  346. ppx_deriving_yaml
  347. ppx_graphql >= "0.2.0"
  348. ppx_inline_alcotest
  349. ppx_protocol_conv >= "5.0.0"
  350. ppx_protocol_conv_json >= "5.0.0"
  351. ppx_protocol_conv_jsonm >= "5.0.0"
  352. ppx_protocol_conv_msgpack >= "5.0.0"
  353. ppx_protocol_conv_xml_light >= "5.0.0"
  354. ppx_protocol_conv_xmlm
  355. ppx_protocol_conv_yaml >= "5.0.0"
  356. ppx_repr < "0.4.0"
  357. ppx_subliner
  358. ppx_units
  359. ppx_yojson >= "1.1.0"
  360. pratter
  361. prc
  362. preface
  363. pretty_expressive
  364. prettym
  365. proc-smaps
  366. producer < "0.2.0"
  367. progress < "0.2.0"
  368. prom
  369. prometheus < "1.2"
  370. prometheus-app
  371. protocell
  372. protocol-9p >= "0.3" & < "0.11.0" | >= "0.11.2"
  373. protocol-9p-unix
  374. psq
  375. qcheck >= "0.18" & < "0.22"
  376. qcheck-alcotest < "0.22"
  377. quickjs
  378. radis
  379. randii
  380. reason-standard
  381. reparse >= "2.0.0" & < "3.0.0"
  382. reparse-unix < "2.1.0"
  383. resp
  384. resp-unix >= "0.10.0"
  385. rfc1951 < "1.0.0"
  386. routes < "2.0.0"
  387. rpc >= "7.1.0"
  388. rpclib >= "7.1.0"
  389. rpclib-async
  390. rpclib-lwt >= "7.1.0"
  391. rpmfile
  392. rubytt
  393. SZXX >= "4.0.0"
  394. salsa20
  395. salsa20-core
  396. sanddb >= "0.2"
  397. scaml >= "1.5.0"
  398. scrypt-kdf
  399. secp256k1 >= "0.4.1"
  400. secp256k1-internal
  401. semver >= "0.2.1"
  402. sendmail
  403. sendmail-lwt
  404. sendmsg
  405. server-reason-react
  406. session-cookie
  407. session-cookie-async
  408. session-cookie-lwt
  409. sherlodoc
  410. slug
  411. sodium-fmt
  412. spin >= "0.6.0"
  413. squirrel
  414. ssh-agent
  415. ssl >= "0.6.0"
  416. stramon-lib
  417. styled-ppx
  418. syslog-rfc5424
  419. tcpip >= "2.4.2" & < "4.0.0" | >= "5.0.1" & < "7.0.0"
  420. tdigest < "2.1.0"
  421. term-indexing
  422. terminal_size >= "0.1.1"
  423. terminus
  424. terminus-cohttp
  425. terminus-hlc
  426. terml
  427. textrazor
  428. tezos-base-test-helpers < "13.0"
  429. tezos-bls12-381-polynomial
  430. tezos-client-base < "12.0"
  431. tezos-crypto >= "8.0" & < "9.0"
  432. tezos-lmdb
  433. tezos-plompiler = "0.1.3"
  434. tezos-plonk = "0.1.3"
  435. tezos-signer-backends >= "8.0" & < "13.0"
  436. tezos-stdlib >= "8.0" & < "12.0"
  437. tezos-test-helpers < "12.0"
  438. tftp
  439. timedesc
  440. timere
  441. tls >= "0.12.0"
  442. toc
  443. topojson
  444. topojsone
  445. transept
  446. twostep
  447. type_eq
  448. type_id
  449. typebeat
  450. typeid >= "1.0.1"
  451. tyre >= "0.4"
  452. tyxml >= "4.0.0"
  453. tyxml-jsx
  454. tyxml-ppx >= "4.3.0"
  455. tyxml-syntax
  456. uecc
  457. ulid
  458. universal-portal
  459. unix-dirent
  460. unix-errno >= "0.3.0"
  461. unix-fcntl >= "0.3.0"
  462. unix-sys-resource
  463. unix-sys-stat
  464. unix-time
  465. unstrctrd
  466. user-agent-parser
  467. uspf
  468. uspf-lwt
  469. uspf-unix
  470. utop >= "2.13.0"
  471. validate
  472. validator
  473. vercel
  474. vpnkit
  475. wcwidth
  476. websocketaf
  477. x509 >= "0.7.0"
  478. xapi-rrd >= "1.8.2"
  479. xapi-stdext-date
  480. xapi-stdext-encodings
  481. xapi-stdext-std >= "4.16.0"
  482. yaml < "3.2.0"
  483. yaml-sexp
  484. yocaml
  485. yocaml_yaml
  486. yojson >= "1.6.0"
  487. yojson-five
  488. yuscii >= "0.3.0"
  489. zar
  490. zed >= "3.2.2"
  491. zlist < "0.4.0"

Conflicts

None

OCaml

Innovation. Community. Security.

GitHub Discord Twitter Peertube RSS
About
Changelog Releases Industrial Users Academic Users Why OCaml
Ecosystem
Packages Community Events OCaml Planet Jobs
Resources
Install OCaml Get Started Platform Tools Language Manual Standard Library API Books Exercises Papers OCaml Playground Logo
Policies
Carbon Footprint Governance Privacy Code of Conduct