package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

github.com Readme Changelog ISC License Edit opam file Versions (40)

Authors

  1. T
    Thomas Gazagnaire

Maintainers

  1. T
    thomas@gazagnaire.org

Sources

alcotest-1.0.1.tbz
sha256=0c8748838a89df6dee4850aa7ef5e46c573265a9bf1589dec255bd8156a793f6
sha512=f5f52dea5bb143e7001b8d0ac6131f8851389b080f46b9ad1ccacb95cc31a38143dd7122ccba59bb190abe559dbf81f33cc4dc3401ed95772d59be75fa566f19

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 13 Feb 2020

README

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Examples

A simple example (taken from examples/simple.ml):

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use ./simple.byte --help to see the runtime options.

$ ./simple.native
Testing Utils.
[OK]       string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
Test Successful in 0.001s. 4 tests run.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead. Test names may contain only alphanumeric characters, spaces, hyphens and underscores.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Screenshots

The following screenshots demonstrate the HTML testing output from the odoc project.

All tests passed Some tests failed Failed test with custom diffing
ok err diff

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)

  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (8)

  1. stdlib-shims
  2. re >= "1.7.2"
  3. uuidm
  4. cmdliner >= "1.0.3"
  5. astring
  6. fmt >= "0.8.6"
  7. ocaml >= "4.03.0"
  8. dune >= "1.11"

Dev Dependencies

None

  1. ahrocksdb
  2. albatross >= "1.5.0"
  3. alcotest-async != "1.0.0" & < "1.1.0"
  4. alcotest-lwt != "1.0.0" & < "1.1.0"
  5. alg_structs_qcheck
  6. ambient-context
  7. ambient-context-eio
  8. ambient-context-lwt
  9. angstrom >= "0.7.0"
  10. ansi >= "0.6.0"
  11. anycache >= "0.7.4"
  12. anycache-async
  13. anycache-lwt
  14. archetype >= "1.4.2"
  15. archi
  16. arp
  17. arp-mirage
  18. arrakis
  19. art
  20. asak >= "0.2"
  21. asli >= "0.2.0"
  22. asn1-combinators >= "0.2.2"
  23. atd >= "2.3.3"
  24. atdgen >= "2.10.0"
  25. atdpy
  26. atdts
  27. base32
  28. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  29. bastet
  30. bastet_async
  31. bastet_lwt
  32. bechamel >= "0.5.0"
  33. bigarray-overlap
  34. bigstring >= "0.3"
  35. bigstring-unix >= "0.3"
  36. bigstringaf
  37. bitlib
  38. blake2
  39. bloomf
  40. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  41. bls12-381-hash
  42. bls12-381-js >= "0.4.2"
  43. bls12-381-js-gen >= "0.4.2"
  44. bls12-381-legacy
  45. bls12-381-signature
  46. bls12-381-unix
  47. blurhash
  48. builder-web
  49. bulletml
  50. bytebuffer
  51. ca-certs
  52. ca-certs-nss
  53. cactus
  54. caldav
  55. calendar >= "3.0.0"
  56. callipyge
  57. camlix
  58. camlkit
  59. camlkit-base
  60. capnp-rpc < "1.2.3"
  61. capnp-rpc-lwt < "0.3"
  62. capnp-rpc-mirage >= "0.9.0"
  63. capnp-rpc-unix >= "0.9.0" & < "1.2.3"
  64. carray
  65. carton
  66. cborl
  67. ccss >= "1.6"
  68. cf-lwt
  69. chacha
  70. channel
  71. charrua-client
  72. charrua-client-lwt
  73. charrua-client-mirage < "0.11.0"
  74. checked_oint < "0.1.1"
  75. checkseum >= "0.0.3"
  76. cid
  77. clarity-lang
  78. class_group_vdf
  79. cohttp >= "0.17.0"
  80. cohttp-curl-async
  81. cohttp-curl-lwt
  82. cohttp-eio >= "6.0.0~beta2"
  83. colombe >= "0.2.0"
  84. color
  85. conan
  86. conan-cli
  87. conan-database
  88. conan-lwt
  89. conan-unix
  90. conduit = "3.0.0"
  91. conex < "0.10.0"
  92. conex-mirage-crypto
  93. conex-nocrypto
  94. cookie
  95. cow >= "2.2.0"
  96. css
  97. css-parser
  98. cstruct >= "3.3.0"
  99. cstruct-sexp
  100. ctypes-zarith
  101. cuid
  102. curly
  103. current_incr
  104. cwe_checker
  105. data-encoding
  106. datakit >= "0.12.0"
  107. datakit-bridge-github >= "0.12.0"
  108. datakit-ci
  109. datakit-client-git >= "0.12.0"
  110. decompress >= "0.8" & < "1.5.3"
  111. depyt
  112. digestif >= "0.8.1"
  113. dispatch >= "0.4.1"
  114. dkim
  115. dkim-bin
  116. dkim-mirage
  117. dns >= "4.0.0"
  118. dns-cli
  119. dns-client >= "4.6.0"
  120. dns-forward < "0.9.0"
  121. dns-forward-lwt-unix
  122. dns-resolver
  123. dns-server
  124. dns-tsig
  125. dnssd
  126. dnssec
  127. docfd >= "2.2.0"
  128. dog < "0.2.1"
  129. domain-name
  130. dream
  131. dream-pure
  132. duff
  133. dune-release >= "1.0.0"
  134. duration >= "0.1.1"
  135. emile
  136. encore
  137. eqaf >= "0.5"
  138. equinoxe
  139. equinoxe-cohttp
  140. equinoxe-hlc
  141. eris
  142. eris-lwt
  143. ezgzip
  144. ezjsonm >= "0.4.2" & < "1.3.0"
  145. ezjsonm-lwt < "1.3.0"
  146. FPauth
  147. FPauth-core
  148. FPauth-responses
  149. FPauth-strategies
  150. faraday != "0.2.0"
  151. farfadet
  152. fat-filesystem >= "0.12.0"
  153. ff
  154. ff-pbt
  155. fiat-p256
  156. flex-array
  157. fsevents-lwt
  158. functoria >= "2.2.0"
  159. functoria-runtime >= "2.2.0" & != "3.0.1" & < "4.0.0~beta1"
  160. geojson
  161. geoml >= "0.1.1"
  162. git = "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0" & < "3.0.0"
  163. git-mirage < "3.0.0"
  164. git-unix >= "1.10.0" & != "2.1.0" & < "3.0.0"
  165. gitlab-unix
  166. glicko2
  167. gmap >= "0.3.0"
  168. gobba
  169. gpt
  170. graphql
  171. graphql-async
  172. graphql-cohttp >= "0.13.0"
  173. graphql-lwt
  174. graphql_parser != "0.11.0"
  175. graphql_ppx >= "0.7.1"
  176. h1_parser
  177. h2
  178. hacl
  179. hacl_func
  180. hacl_x25519 >= "0.2.0"
  181. highlexer
  182. hkdf
  183. hockmd
  184. html_of_jsx
  185. http
  186. http-multipart-formdata < "2.0.0"
  187. httpaf >= "0.2.0"
  188. httpun
  189. hvsock
  190. icalendar >= "0.1.4"
  191. imagelib >= "20200929"
  192. index
  193. inferno >= "20220603"
  194. influxdb-async
  195. influxdb-lwt
  196. inquire < "0.2.0"
  197. interval-map
  198. iomux
  199. irmin < "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & < "2.3.0"
  200. irmin-bench >= "2.7.0"
  201. irmin-chunk < "1.3.0" | >= "2.3.0"
  202. irmin-cli
  203. irmin-containers
  204. irmin-fs < "1.3.0" | >= "2.3.0"
  205. irmin-git < "2.0.0" | >= "2.3.0"
  206. irmin-http < "2.0.0"
  207. irmin-mem < "1.3.0"
  208. irmin-pack >= "2.4.0" & != "2.6.1"
  209. irmin-pack-tools
  210. irmin-test >= "2.2.0" & < "3.0.0"
  211. irmin-tezos
  212. irmin-tezos-utils
  213. irmin-unix >= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
  214. irmin-watcher
  215. jekyll-format
  216. jerboa
  217. jitsu
  218. jose
  219. json-data-encoding >= "0.9"
  220. json_decoder
  221. jsonxt
  222. junit_alcotest
  223. jwto
  224. ke >= "0.2"
  225. kkmarkdown
  226. lambda-runtime
  227. lambda_streams
  228. lambda_streams_async
  229. lambda_streams_lwt
  230. lambdapi >= "2.0.0"
  231. lambdoc >= "1.0-beta4"
  232. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  233. lmdb >= "1.0"
  234. logical
  235. logtk >= "1.6"
  236. lp
  237. lp-glpk
  238. lp-glpk-js
  239. lp-gurobi
  240. lru
  241. lt-code
  242. luv
  243. mbr-format >= "1.0.0"
  244. mdx >= "1.6.0"
  245. mec
  246. mechaml >= "1.0.0"
  247. merge-queues >= "0.2.0"
  248. merge-ropes >= "0.2.0"
  249. metrics
  250. minicaml = "0.3.1" | >= "0.4"
  251. mirage >= "4.0.0~beta1"
  252. mirage-block-partition
  253. mirage-block-ramdisk >= "0.3"
  254. mirage-channel >= "4.0.0"
  255. mirage-channel-lwt
  256. mirage-crypto-ec
  257. mirage-flow >= "1.0.2" & < "1.2.0"
  258. mirage-flow-unix
  259. mirage-fs-mem
  260. mirage-fs-unix >= "1.2.0"
  261. mirage-kv >= "2.0.0"
  262. mirage-kv-mem
  263. mirage-kv-unix
  264. mirage-logs >= "0.3.0"
  265. mirage-nat
  266. mirage-net-unix >= "2.3.0"
  267. mirage-runtime >= "4.0.0~beta1" & < "4.5.0"
  268. mirage-tc
  269. mjson
  270. mmdb
  271. mnd
  272. monocypher
  273. mrmime >= "0.2.0"
  274. mrt-format
  275. msgpck >= "1.6"
  276. mssql >= "2.0.3"
  277. multibase
  278. multihash
  279. multihash-digestif
  280. multipart-form-data
  281. multipart_form
  282. multipart_form-eio
  283. multipart_form-lwt
  284. named-pipe
  285. nanoid
  286. nbd >= "4.0.3"
  287. nbd-tool
  288. nloge
  289. nocoiner
  290. non_empty_list
  291. OCADml >= "0.6.0"
  292. ocaml-r >= "0.5.0"
  293. ocaml-version >= "3.1.0"
  294. ocamlformat >= "0.13.0" & != "0.19.0~4.13preview" & < "0.25.1"
  295. ocamlformat-rpc < "removed"
  296. ocamline
  297. ocluster < "0.3.0"
  298. odoc >= "1.4.0" & < "2.1.0"
  299. ohex
  300. oidc
  301. opam-0install
  302. opam-file-format >= "2.1.1"
  303. opentelemetry >= "0.6"
  304. opentelemetry-client-cohttp-lwt >= "0.6"
  305. opentelemetry-client-ocurl >= "0.6"
  306. opentelemetry-cohttp-lwt >= "0.6"
  307. opentelemetry-lwt >= "0.6"
  308. opium >= "0.15.0"
  309. opium-graphql
  310. opium-testing
  311. opium_kernel
  312. orewa
  313. ortac-core
  314. osx-acl
  315. osx-attr
  316. osx-cf
  317. osx-fsevents
  318. osx-membership
  319. osx-mount
  320. osx-xattr
  321. otoggl
  322. owl >= "0.6.0" & != "0.9.0" & != "1.0.0"
  323. owl-base < "0.5.0"
  324. owl-ode >= "0.1.0" & != "0.2.0"
  325. owl-symbolic
  326. passmaker
  327. patch
  328. pbkdf
  329. pecu >= "0.2"
  330. pf-qubes
  331. pg_query >= "0.9.6"
  332. pgx >= "1.0"
  333. pgx_unix >= "1.0"
  334. pgx_value_core
  335. pgx_value_ptime < "2.2"
  336. phylogenetics
  337. piaf
  338. polyglot
  339. polynomial
  340. ppx_blob >= "0.3.0"
  341. ppx_deriving_cmdliner
  342. ppx_deriving_rpc
  343. ppx_deriving_yaml
  344. ppx_graphql >= "0.2.0"
  345. ppx_inline_alcotest
  346. ppx_parser
  347. ppx_protocol_conv >= "5.0.0"
  348. ppx_protocol_conv_json >= "5.0.0"
  349. ppx_protocol_conv_jsonm >= "5.0.0"
  350. ppx_protocol_conv_msgpack >= "5.0.0"
  351. ppx_protocol_conv_xml_light >= "5.0.0"
  352. ppx_protocol_conv_xmlm
  353. ppx_protocol_conv_yaml >= "5.0.0"
  354. ppx_subliner
  355. ppx_units
  356. ppx_yojson >= "1.1.0"
  357. pratter
  358. prc
  359. preface
  360. pretty_expressive
  361. prettym
  362. proc-smaps
  363. producer < "0.2.0"
  364. prom
  365. prometheus < "1.2"
  366. prometheus-app
  367. protocell
  368. protocol-9p >= "0.3" & < "0.11.0" | >= "0.11.2"
  369. protocol-9p-unix
  370. psq
  371. qcheck >= "0.18"
  372. qcheck-alcotest
  373. qcheck-core >= "0.18"
  374. quickjs
  375. radis
  376. randii
  377. reason-standard
  378. reparse >= "2.0.0" & < "3.0.0"
  379. reparse-unix < "2.1.0"
  380. resp
  381. resp-unix >= "0.10.0"
  382. rfc1951 < "1.0.0"
  383. routes < "2.0.0"
  384. rpc >= "7.1.0"
  385. rpclib >= "7.1.0"
  386. rpclib-async
  387. rpclib-lwt >= "7.1.0"
  388. rubytt
  389. SZXX >= "4.0.0"
  390. salsa20
  391. salsa20-core
  392. sanddb >= "0.2"
  393. scaml >= "1.5.0"
  394. scrypt-kdf
  395. secp256k1 >= "0.4.1"
  396. secp256k1-internal
  397. semver >= "0.2.1"
  398. sendmail
  399. sendmail-lwt
  400. sendmsg
  401. server-reason-react
  402. session-cookie
  403. session-cookie-async
  404. session-cookie-lwt
  405. sherlodoc
  406. slug
  407. sodium-fmt
  408. spin >= "0.6.0"
  409. squirrel
  410. ssh-agent
  411. ssl >= "0.6.0"
  412. stramon-lib
  413. styled-ppx
  414. syslog-rfc5424
  415. tcpip >= "2.4.2" & < "4.0.0" | >= "5.0.1" & < "7.0.0"
  416. tdigest < "2.1.0"
  417. terminal_size >= "0.1.1"
  418. terminus
  419. terminus-cohttp
  420. terminus-hlc
  421. terml
  422. textrazor
  423. tezos-base-test-helpers < "13.0"
  424. tezos-client-base < "12.0"
  425. tezos-lmdb
  426. tezos-test-helpers < "11.0"
  427. tftp
  428. timedesc
  429. timere
  430. tls >= "0.12.0"
  431. toc
  432. topojson
  433. topojsone
  434. transept
  435. twostep
  436. type_eq
  437. type_id
  438. typebeat
  439. typeid >= "1.0.1"
  440. tyre >= "0.4"
  441. tyxml >= "4.0.0"
  442. tyxml-jsx
  443. tyxml-ppx >= "4.3.0"
  444. tyxml-syntax
  445. uecc
  446. ulid
  447. universal-portal
  448. unix-dirent
  449. unix-errno >= "0.3.0"
  450. unix-fcntl >= "0.3.0"
  451. unix-sys-resource
  452. unix-sys-stat
  453. unix-time
  454. unstrctrd
  455. user-agent-parser
  456. uspf
  457. uspf-lwt
  458. uspf-unix
  459. utop >= "2.13.0"
  460. validate
  461. validator
  462. vercel
  463. vpnkit
  464. wcwidth
  465. websocketaf
  466. x509 >= "0.7.0"
  467. xapi-rrd >= "1.8.2"
  468. xapi-stdext-date
  469. xapi-stdext-encodings
  470. xapi-stdext-std >= "4.16.0"
  471. yaml < "3.2.0"
  472. yaml-sexp
  473. yocaml
  474. yocaml_yaml
  475. yojson >= "1.6.0"
  476. yojson-five
  477. yuscii >= "0.3.0"
  478. zar
  479. zed >= "3.2.2"
  480. zlist < "0.4.0"

Conflicts

None

OCaml

Innovation. Community. Security.

GitHub Discord Twitter Peertube RSS
About
Changelog Releases Industrial Users Academic Users Why OCaml
Ecosystem
Packages Community Events OCaml Planet Jobs
Resources
Install OCaml Get Started Platform Tools Language Manual Standard Library API Books Exercises Papers OCaml Playground Logo
Policies
Carbon Footprint Governance Privacy Code of Conduct