package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

github.com Readme Changelog ISC License Edit opam file Versions (40)

Authors

  1. T
    Thomas Gazagnaire

Maintainers

  1. T
    thomas@gazagnaire.org

Sources

alcotest-mirage-1.2.0.tbz
sha256=d82b7236081362320bd6dd968b192096bb68e3c688beab825fb0ebd3994daa87
sha512=67ec5355bb4f8cb806b8b0cecf4831c4ad20aa53ab161c8991bf46642c6d828a4e1de0aa2a16f880ebe528dbf7800e2a99778bf8397a9b7cb921301d0c0f4b70

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 13 Jul 2020

README

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run. See the manpage for details.

For information on contributing to Alcotest, see CONTRIBUTING.md.

Examples

A simple example (taken from examples/simple.ml):

Generated by the following test suite specification:

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use dune exec examples/simple.exe -- --help to see the runtime options.

Here's an example of a of failing test suite:

By default, only the first failing test log is printed to the console (and all test logs are captured on disk). Pass --show-errors to print all error messages.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)

  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (9)

  1. uutf >= "1.0.0"
  2. stdlib-shims
  3. re >= "1.7.2"
  4. uuidm
  5. cmdliner >= "1.0.3" & < "1.1.0"
  6. astring
  7. fmt >= "0.8.7"
  8. ocaml >= "4.03.0"
  9. dune >= "2.0"

Dev Dependencies

None

  1. ahrocksdb
  2. albatross >= "1.5.0"
  3. alcotest-async < "1.0.0" | = "1.2.0"
  4. alcotest-lwt < "1.0.0" | = "1.2.0"
  5. alcotest-mirage < "1.2.1"
  6. alg_structs_qcheck
  7. ambient-context
  8. ambient-context-eio
  9. ambient-context-lwt
  10. angstrom >= "0.7.0"
  11. ansi >= "0.6.0"
  12. anycache >= "0.7.4"
  13. anycache-async
  14. anycache-lwt
  15. archetype >= "1.4.2"
  16. archi
  17. arp
  18. arp-mirage
  19. arrakis
  20. art
  21. asak >= "0.2"
  22. asli >= "0.2.0"
  23. asn1-combinators >= "0.2.2"
  24. atd >= "2.3.3"
  25. atdgen >= "2.10.0"
  26. atdpy
  27. atdts
  28. base32
  29. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  30. bastet
  31. bastet_async
  32. bastet_lwt
  33. bech32
  34. bechamel >= "0.5.0"
  35. bigarray-overlap
  36. bigstringaf
  37. bitlib
  38. blake2
  39. bloomf
  40. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  41. bls12-381-hash
  42. bls12-381-js >= "0.4.2"
  43. bls12-381-js-gen >= "0.4.2"
  44. bls12-381-legacy
  45. bls12-381-signature
  46. bls12-381-unix
  47. blurhash
  48. builder-web
  49. bulletml
  50. bytebuffer
  51. ca-certs
  52. ca-certs-nss
  53. cactus
  54. caldav
  55. calendar >= "3.0.0"
  56. callipyge
  57. camlix
  58. camlkit
  59. camlkit-base
  60. capnp-rpc < "1.2.3"
  61. capnp-rpc-lwt < "0.3"
  62. capnp-rpc-mirage >= "0.9.0"
  63. capnp-rpc-unix >= "0.9.0" & < "1.2.3"
  64. carray
  65. carton
  66. cborl
  67. ccss >= "1.6"
  68. cf-lwt
  69. chacha
  70. channel
  71. charrua-client
  72. charrua-client-lwt
  73. charrua-client-mirage < "0.11.0"
  74. checked_oint < "0.1.1"
  75. checkseum >= "0.0.3"
  76. cid
  77. clarity-lang
  78. class_group_vdf
  79. cohttp >= "0.17.0"
  80. cohttp-curl-async
  81. cohttp-curl-lwt
  82. cohttp-eio >= "6.0.0~beta2"
  83. colombe >= "0.2.0"
  84. color
  85. conan
  86. conan-cli
  87. conan-database
  88. conan-lwt
  89. conan-unix
  90. conduit = "3.0.0"
  91. conex < "0.10.0"
  92. conex-mirage-crypto
  93. conex-nocrypto
  94. cookie
  95. cow >= "2.2.0"
  96. css
  97. css-parser
  98. cstruct >= "3.3.0"
  99. cstruct-sexp
  100. ctypes-zarith
  101. cuid
  102. curly
  103. current >= "0.4"
  104. current_git >= "0.6.4"
  105. current_incr
  106. cwe_checker
  107. data-encoding
  108. datakit >= "0.12.0"
  109. datakit-bridge-github >= "0.12.0"
  110. datakit-ci
  111. datakit-client-git >= "0.12.0"
  112. decompress >= "0.8" & < "1.5.3"
  113. depyt
  114. digestif >= "0.8.1"
  115. dispatch >= "0.4.1"
  116. dkim
  117. dkim-bin
  118. dkim-mirage
  119. dns >= "4.0.0"
  120. dns-cli
  121. dns-client >= "4.6.0"
  122. dns-forward < "0.9.0"
  123. dns-forward-lwt-unix
  124. dns-resolver
  125. dns-server
  126. dns-tsig
  127. dnssd
  128. dnssec
  129. docfd >= "2.2.0"
  130. dog < "0.2.1"
  131. domain-name
  132. dream
  133. dream-pure
  134. duff
  135. dune-release >= "1.0.0"
  136. duration >= "0.1.1"
  137. emile
  138. encore
  139. eqaf >= "0.5"
  140. equinoxe
  141. equinoxe-cohttp
  142. equinoxe-hlc
  143. eris
  144. eris-lwt
  145. ezgzip
  146. ezjsonm >= "0.4.2" & < "1.3.0"
  147. ezjsonm-lwt
  148. FPauth
  149. FPauth-core
  150. FPauth-responses
  151. FPauth-strategies
  152. faraday != "0.2.0"
  153. farfadet
  154. fat-filesystem >= "0.12.0"
  155. ff
  156. ff-pbt
  157. fiat-p256
  158. flex-array
  159. fsevents-lwt
  160. functoria >= "2.2.0"
  161. functoria-runtime >= "2.2.0" & != "3.0.1" & < "4.0.0~beta1"
  162. geojson
  163. geoml >= "0.1.1"
  164. git = "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0"
  165. git-mirage < "3.0.0"
  166. git-unix >= "1.10.0" & != "2.1.0"
  167. gitlab-unix
  168. glicko2
  169. gmap >= "0.3.0"
  170. gobba
  171. gpt
  172. graphql
  173. graphql-async
  174. graphql-cohttp >= "0.13.0"
  175. graphql-lwt
  176. graphql_parser != "0.11.0"
  177. graphql_ppx >= "0.7.1"
  178. h1_parser
  179. h2
  180. hacl
  181. hacl-star >= "0.6.0"
  182. hacl_func
  183. hacl_x25519 >= "0.2.0"
  184. highlexer
  185. hkdf
  186. hockmd
  187. html_of_jsx
  188. http
  189. http-multipart-formdata < "2.0.0"
  190. httpaf >= "0.2.0"
  191. httpun
  192. hvsock
  193. icalendar >= "0.1.4"
  194. imagelib >= "20200929"
  195. index
  196. inferno >= "20220603"
  197. influxdb-async
  198. influxdb-lwt
  199. inquire < "0.2.0"
  200. interval-map
  201. iomux
  202. irmin < "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & != "2.3.0"
  203. irmin-bench >= "2.7.0"
  204. irmin-chunk < "1.3.0" | >= "2.3.0"
  205. irmin-cli
  206. irmin-containers
  207. irmin-fs < "1.3.0" | >= "2.3.0"
  208. irmin-git < "2.0.0" | >= "2.3.0"
  209. irmin-http < "2.0.0"
  210. irmin-mem < "1.3.0"
  211. irmin-pack >= "2.4.0" & != "2.6.1"
  212. irmin-pack-tools
  213. irmin-test >= "2.2.0" & < "3.0.0"
  214. irmin-tezos
  215. irmin-tezos-utils
  216. irmin-unix >= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
  217. irmin-watcher
  218. jekyll-format
  219. jerboa
  220. jitsu
  221. jose
  222. json-data-encoding >= "0.9"
  223. json_decoder
  224. jsonxt
  225. junit_alcotest
  226. jwto
  227. ke >= "0.2"
  228. kkmarkdown
  229. lambda-runtime
  230. lambda_streams
  231. lambda_streams_async
  232. lambdapi >= "2.0.0"
  233. lambdoc >= "1.0-beta4"
  234. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  235. letters
  236. lmdb >= "1.0"
  237. logical
  238. logtk >= "1.6"
  239. lp
  240. lp-glpk
  241. lp-glpk-js
  242. lp-gurobi
  243. lru
  244. lt-code
  245. luv
  246. mbr-format >= "1.0.0"
  247. mdx >= "1.6.0"
  248. mec
  249. mechaml >= "1.0.0"
  250. merge-queues >= "0.2.0"
  251. merge-ropes >= "0.2.0"
  252. metrics
  253. middleware
  254. minicaml = "0.3.1" | >= "0.4"
  255. mirage >= "4.0.0~beta1"
  256. mirage-block-partition
  257. mirage-block-ramdisk >= "0.3"
  258. mirage-channel >= "4.0.0"
  259. mirage-channel-lwt
  260. mirage-crypto-ec
  261. mirage-flow >= "1.0.2" & < "1.2.0"
  262. mirage-flow-unix
  263. mirage-fs-mem
  264. mirage-fs-unix >= "1.2.0"
  265. mirage-kv >= "2.0.0"
  266. mirage-kv-mem
  267. mirage-kv-unix
  268. mirage-logs >= "0.3.0"
  269. mirage-nat
  270. mirage-net-unix >= "2.3.0"
  271. mirage-runtime >= "4.0.0~beta1" & < "4.5.0"
  272. mirage-tc
  273. mjson
  274. mmdb
  275. mnd
  276. monocypher
  277. mrmime >= "0.2.0"
  278. mrt-format
  279. msgpck >= "1.6"
  280. mssql >= "2.0.3"
  281. multibase
  282. multihash
  283. multihash-digestif
  284. multipart-form-data
  285. multipart_form
  286. multipart_form-eio
  287. multipart_form-lwt
  288. named-pipe
  289. nanoid
  290. nbd >= "4.0.3"
  291. nbd-tool
  292. nloge
  293. nocoiner
  294. non_empty_list
  295. OCADml >= "0.6.0"
  296. ocaml-r >= "0.4.0"
  297. ocaml-version >= "3.1.0"
  298. ocamlformat >= "0.13.0" & != "0.19.0~4.13preview" & < "0.25.1"
  299. ocamlformat-rpc < "removed"
  300. ocamline
  301. ocluster < "0.3.0"
  302. odoc >= "1.4.0" & < "2.1.0"
  303. ohex
  304. oidc
  305. opam-0install
  306. opam-compiler
  307. opam-file-format >= "2.1.1"
  308. opentelemetry >= "0.6"
  309. opentelemetry-client-cohttp-lwt >= "0.6"
  310. opentelemetry-client-ocurl >= "0.6"
  311. opentelemetry-cohttp-lwt >= "0.6"
  312. opentelemetry-lwt >= "0.6"
  313. opium >= "0.15.0"
  314. opium-graphql
  315. opium-testing
  316. opium_kernel
  317. orewa
  318. ortac-core
  319. osx-acl
  320. osx-attr
  321. osx-cf
  322. osx-fsevents
  323. osx-membership
  324. osx-mount
  325. osx-xattr
  326. otoggl
  327. owl >= "0.6.0" & != "0.9.0" & != "1.0.0"
  328. owl-base < "0.5.0"
  329. owl-ode >= "0.1.0" & != "0.2.0"
  330. owl-symbolic
  331. passmaker
  332. patch
  333. pbkdf
  334. pecu >= "0.2"
  335. pf-qubes
  336. pg_query >= "0.9.6"
  337. pgx >= "1.0"
  338. pgx_unix >= "1.0"
  339. pgx_value_core
  340. pgx_value_ptime < "2.2"
  341. phylogenetics
  342. piaf
  343. polyglot
  344. polynomial
  345. ppx_blob >= "0.3.0"
  346. ppx_deriving_cmdliner
  347. ppx_deriving_rpc
  348. ppx_deriving_yaml
  349. ppx_graphql >= "0.2.0"
  350. ppx_inline_alcotest
  351. ppx_parser
  352. ppx_protocol_conv >= "5.0.0"
  353. ppx_protocol_conv_json >= "5.0.0"
  354. ppx_protocol_conv_jsonm >= "5.0.0"
  355. ppx_protocol_conv_msgpack >= "5.0.0"
  356. ppx_protocol_conv_xml_light >= "5.0.0"
  357. ppx_protocol_conv_xmlm
  358. ppx_protocol_conv_yaml >= "5.0.0"
  359. ppx_repr < "0.4.0"
  360. ppx_subliner
  361. ppx_units
  362. ppx_yojson >= "1.1.0"
  363. pratter
  364. prc
  365. preface
  366. pretty_expressive
  367. prettym
  368. proc-smaps
  369. producer < "0.2.0"
  370. progress < "0.2.0"
  371. prom
  372. prometheus < "1.2"
  373. prometheus-app
  374. protocell
  375. protocol-9p >= "0.3" & < "0.11.0" | >= "0.11.2"
  376. protocol-9p-unix
  377. psq
  378. qcheck >= "0.18"
  379. qcheck-alcotest
  380. qcheck-core >= "0.18"
  381. quickjs
  382. radis
  383. randii
  384. reason-standard
  385. reparse >= "2.0.0" & < "3.0.0"
  386. reparse-unix < "2.1.0"
  387. resp
  388. resp-unix >= "0.10.0"
  389. rfc1951 < "1.0.0"
  390. routes < "2.0.0"
  391. rpc >= "7.1.0"
  392. rpclib >= "7.1.0"
  393. rpclib-async
  394. rpclib-lwt >= "7.1.0"
  395. rubytt
  396. SZXX >= "4.0.0"
  397. salsa20
  398. salsa20-core
  399. sanddb >= "0.2"
  400. scaml >= "1.5.0"
  401. scrypt-kdf
  402. secp256k1 >= "0.4.1"
  403. secp256k1-internal
  404. semver >= "0.2.1"
  405. sendmail
  406. sendmail-lwt
  407. sendmsg
  408. server-reason-react
  409. session-cookie
  410. session-cookie-async
  411. session-cookie-lwt
  412. sherlodoc
  413. sihl < "0.2.0"
  414. sihl-type
  415. slug
  416. sodium-fmt
  417. solidity-alcotest
  418. spin >= "0.7.0"
  419. squirrel
  420. ssh-agent
  421. ssl >= "0.6.0"
  422. stramon-lib
  423. styled-ppx
  424. syslog-rfc5424
  425. tcpip >= "2.4.2" & < "4.0.0" | >= "5.0.1" & < "7.0.0"
  426. tdigest < "2.1.0"
  427. terminal_size >= "0.1.1"
  428. terminus
  429. terminus-cohttp
  430. terminus-hlc
  431. terml
  432. textrazor
  433. tezos-base-test-helpers < "13.0"
  434. tezos-bls12-381-polynomial
  435. tezos-client-base < "12.0"
  436. tezos-crypto >= "8.0" & < "9.0"
  437. tezos-lmdb
  438. tezos-plompiler = "0.1.3"
  439. tezos-plonk = "0.1.3"
  440. tezos-signer-backends >= "8.0" & < "13.0"
  441. tezos-stdlib >= "8.0" & < "12.0"
  442. tezos-test-helpers < "12.0"
  443. tftp
  444. timedesc
  445. timere
  446. tls >= "0.12.0"
  447. toc
  448. topojson
  449. topojsone
  450. transept
  451. twostep
  452. type_eq
  453. type_id
  454. typebeat
  455. typeid >= "1.0.1"
  456. tyre >= "0.4"
  457. tyxml >= "4.0.0"
  458. tyxml-jsx
  459. tyxml-ppx >= "4.3.0"
  460. tyxml-syntax
  461. uecc
  462. ulid
  463. universal-portal
  464. unix-dirent
  465. unix-errno >= "0.3.0"
  466. unix-fcntl >= "0.3.0"
  467. unix-sys-resource
  468. unix-sys-stat
  469. unix-time
  470. unstrctrd
  471. user-agent-parser
  472. uspf
  473. uspf-lwt
  474. uspf-unix
  475. utop >= "2.13.0"
  476. validate
  477. validator
  478. vercel
  479. vpnkit
  480. wcwidth
  481. websocketaf
  482. x509 >= "0.7.0"
  483. xapi-rrd >= "1.8.2"
  484. xapi-stdext-date
  485. xapi-stdext-encodings
  486. xapi-stdext-std >= "4.16.0"
  487. yaml < "3.2.0"
  488. yaml-sexp
  489. yocaml
  490. yocaml_yaml
  491. yojson >= "1.6.0"
  492. yojson-five
  493. yuscii >= "0.3.0"
  494. zar
  495. zed >= "3.2.2"
  496. zlist < "0.4.0"

Conflicts

None

OCaml

Innovation. Community. Security.

GitHub Discord Twitter Peertube RSS
About
Changelog Releases Industrial Users Academic Users Why OCaml
Ecosystem
Packages Community Events OCaml Planet Jobs
Resources
Install OCaml Get Started Platform Tools Language Manual Standard Library API Books Exercises Papers OCaml Playground Logo
Policies
Carbon Footprint Governance Privacy Code of Conduct