package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

github.com Readme Changelog ISC License Edit opam file Versions (40)

Authors

  1. T
    Thomas Gazagnaire

Maintainers

  1. T
    thomas@gazagnaire.org

Sources

alcotest-1.0.0.tbz
sha256=ddabff1722ddef4a521c89b9572b9d06f2440d89169db806bea848cb218d83a8
sha512=3c9dffbb5064cf3e9995110628c7fdf466651e9e022addc8eb1c79585863112a195c254994eb8f8384e183c9e2d9c946e28dcd4b1cac7ca48114a478de2362c0

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 11 Feb 2020

README

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Examples

A simple example (taken from examples/simple.ml):

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use ./simple.byte --help to see the runtime options.

$ ./simple.native
Testing Utils.
[OK]       string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
Test Successful in 0.001s. 4 tests run.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead. Test names may contain only alphanumeric characters, spaces, hyphens and underscores.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Screenshots

The following screenshots demonstrate the HTML testing output from the odoc project.

All tests passed Some tests failed Failed test with custom diffing
ok err diff

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)

  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (8)

  1. stdlib-shims
  2. re >= "1.7.2"
  3. uuidm
  4. cmdliner >= "1.0.3"
  5. astring
  6. fmt >= "0.8.6"
  7. ocaml >= "4.03.0"
  8. dune >= "1.11"

Dev Dependencies

None

  1. ahrocksdb
  2. albatross >= "1.5.0"
  3. alcotest-async < "1.0.1"
  4. alcotest-lwt < "1.0.1"
  5. alg_structs_qcheck
  6. ambient-context
  7. ambient-context-eio
  8. ambient-context-lwt
  9. angstrom >= "0.7.0"
  10. ansi >= "0.6.0"
  11. anycache >= "0.7.4"
  12. anycache-async
  13. anycache-lwt
  14. archetype >= "1.4.2"
  15. archi
  16. arp
  17. arp-mirage
  18. arrakis
  19. art
  20. asak >= "0.2"
  21. asli >= "0.2.0"
  22. asn1-combinators >= "0.2.2"
  23. atd >= "2.3.3"
  24. atdgen >= "2.10.0"
  25. atdpy
  26. atdts
  27. base32
  28. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  29. bechamel >= "0.5.0"
  30. bigarray-overlap
  31. bigstring >= "0.3"
  32. bigstring-unix >= "0.3"
  33. bigstringaf
  34. bitlib
  35. blake2
  36. bloomf
  37. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  38. bls12-381-hash
  39. bls12-381-js >= "0.4.2"
  40. bls12-381-js-gen >= "0.4.2"
  41. bls12-381-legacy
  42. bls12-381-signature
  43. bls12-381-unix
  44. blurhash
  45. builder-web
  46. bulletml
  47. bytebuffer
  48. ca-certs
  49. ca-certs-nss
  50. cactus
  51. caldav
  52. calendar >= "3.0.0"
  53. callipyge
  54. camlix
  55. camlkit
  56. camlkit-base
  57. capnp-rpc < "0.6.0"
  58. capnp-rpc-lwt < "0.3"
  59. carray
  60. carton
  61. cborl
  62. ccss >= "1.6"
  63. cf-lwt
  64. chacha
  65. channel
  66. charrua-client
  67. charrua-client-lwt
  68. charrua-client-mirage < "0.11.0"
  69. checked_oint < "0.1.1"
  70. checkseum >= "0.0.3"
  71. cid
  72. clarity-lang
  73. class_group_vdf
  74. cohttp >= "0.17.0"
  75. cohttp-curl-async
  76. cohttp-curl-lwt
  77. cohttp-eio >= "6.0.0~beta2"
  78. colombe >= "0.2.0"
  79. color
  80. conan
  81. conan-cli
  82. conan-database
  83. conan-lwt
  84. conan-unix
  85. conduit = "3.0.0"
  86. conex < "0.10.0"
  87. conex-mirage-crypto
  88. conex-nocrypto
  89. cookie
  90. cow >= "2.2.0"
  91. css
  92. css-parser
  93. cstruct >= "3.3.0"
  94. cstruct-sexp
  95. ctypes-zarith
  96. cuid
  97. curly
  98. current_incr
  99. cwe_checker
  100. data-encoding
  101. datakit >= "0.12.0"
  102. datakit-bridge-github >= "0.12.0"
  103. datakit-ci
  104. datakit-client-git >= "0.12.0"
  105. decompress >= "0.8" & < "1.5.3"
  106. depyt
  107. digestif >= "0.8.1"
  108. dispatch >= "0.4.1"
  109. dkim
  110. dkim-bin
  111. dkim-mirage
  112. dns >= "4.0.0"
  113. dns-cli
  114. dns-client >= "4.6.0"
  115. dns-forward < "0.9.0"
  116. dns-forward-lwt-unix
  117. dns-resolver
  118. dns-server
  119. dns-tsig
  120. dnssd
  121. dnssec
  122. docfd >= "2.2.0"
  123. dog < "0.2.1"
  124. domain-name
  125. dream
  126. dream-pure
  127. duff
  128. dune-release >= "1.0.0"
  129. duration >= "0.1.1"
  130. emile
  131. encore
  132. eqaf >= "0.5"
  133. equinoxe
  134. equinoxe-cohttp
  135. equinoxe-hlc
  136. eris
  137. eris-lwt
  138. ezgzip
  139. ezjsonm >= "0.4.2" & < "1.3.0"
  140. ezjsonm-lwt < "1.3.0"
  141. FPauth
  142. FPauth-core
  143. FPauth-responses
  144. FPauth-strategies
  145. faraday != "0.2.0"
  146. farfadet
  147. fat-filesystem >= "0.12.0"
  148. ff
  149. ff-pbt
  150. fiat-p256
  151. flex-array
  152. fsevents-lwt
  153. functoria >= "2.2.0"
  154. functoria-runtime >= "2.2.0" & != "3.0.1" & < "4.0.0~beta1"
  155. geojson
  156. geoml >= "0.1.1"
  157. git = "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0" & < "3.0.0"
  158. git-mirage < "3.0.0"
  159. git-unix >= "1.10.0" & != "2.1.0" & < "3.0.0"
  160. gitlab-unix
  161. glicko2
  162. gmap >= "0.3.0"
  163. gobba
  164. gpt
  165. graphql
  166. graphql-async
  167. graphql-cohttp >= "0.13.0"
  168. graphql-lwt
  169. graphql_parser != "0.11.0"
  170. graphql_ppx >= "0.7.1"
  171. h1_parser
  172. h2
  173. hacl
  174. hacl_func
  175. hacl_x25519 >= "0.2.0"
  176. highlexer
  177. hkdf
  178. hockmd
  179. html_of_jsx
  180. http
  181. http-multipart-formdata < "2.0.0"
  182. httpaf >= "0.2.0"
  183. hvsock
  184. icalendar >= "0.1.4"
  185. imagelib >= "20200929"
  186. index
  187. inferno >= "20220603"
  188. influxdb-async
  189. influxdb-lwt
  190. inquire < "0.2.0"
  191. interval-map
  192. iomux
  193. irmin < "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & < "2.3.0"
  194. irmin-bench >= "2.7.0"
  195. irmin-chunk < "1.3.0" | >= "2.3.0"
  196. irmin-cli
  197. irmin-containers
  198. irmin-fs < "1.3.0" | >= "2.3.0"
  199. irmin-git < "2.0.0" | >= "2.3.0"
  200. irmin-http < "2.0.0"
  201. irmin-mem < "1.3.0"
  202. irmin-pack >= "2.4.0" & != "2.6.1"
  203. irmin-pack-tools
  204. irmin-tezos
  205. irmin-tezos-utils
  206. irmin-unix >= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
  207. irmin-watcher
  208. jekyll-format
  209. jerboa
  210. jitsu
  211. jose
  212. json-data-encoding >= "0.9"
  213. json_decoder
  214. jsonxt
  215. junit_alcotest
  216. jwto
  217. ke >= "0.2"
  218. kkmarkdown
  219. lambda-runtime
  220. lambdapi >= "2.0.0"
  221. lambdoc >= "1.0-beta4"
  222. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  223. lmdb >= "1.0"
  224. logical
  225. logtk >= "1.6"
  226. lp
  227. lp-glpk
  228. lp-glpk-js
  229. lp-gurobi
  230. lru
  231. lt-code
  232. luv
  233. mbr-format >= "1.0.0"
  234. mdx >= "1.6.0"
  235. mec
  236. mechaml >= "1.0.0"
  237. merge-queues >= "0.2.0"
  238. merge-ropes >= "0.2.0"
  239. metrics
  240. minicaml = "0.3.1" | >= "0.4"
  241. mirage >= "4.0.0~beta1"
  242. mirage-block-partition
  243. mirage-block-ramdisk >= "0.3"
  244. mirage-channel >= "4.0.0"
  245. mirage-channel-lwt
  246. mirage-crypto-ec
  247. mirage-flow >= "1.0.2" & < "1.2.0"
  248. mirage-flow-unix
  249. mirage-fs-mem
  250. mirage-fs-unix >= "1.2.0"
  251. mirage-kv >= "2.0.0"
  252. mirage-kv-mem
  253. mirage-kv-unix
  254. mirage-logs >= "0.3.0"
  255. mirage-nat
  256. mirage-net-unix >= "2.3.0"
  257. mirage-runtime >= "4.0.0~beta1" & < "4.5.0"
  258. mirage-tc
  259. mjson
  260. mmdb
  261. mnd
  262. monocypher
  263. mrmime >= "0.2.0"
  264. mrt-format
  265. msgpck >= "1.6"
  266. multibase
  267. multihash
  268. multihash-digestif
  269. multipart-form-data
  270. multipart_form
  271. multipart_form-eio
  272. multipart_form-lwt
  273. named-pipe
  274. nanoid
  275. nbd >= "4.0.3"
  276. nbd-tool
  277. nloge
  278. nocoiner
  279. non_empty_list
  280. OCADml >= "0.6.0"
  281. ocaml-r >= "0.5.0"
  282. ocaml-version >= "3.1.0"
  283. ocamlformat >= "0.13.0" & != "0.19.0~4.13preview" & < "0.25.1"
  284. ocamlformat-rpc < "removed"
  285. ocamline
  286. ocluster < "0.3.0"
  287. odoc >= "1.4.0" & < "2.1.0"
  288. ohex
  289. oidc
  290. opam-0install
  291. opam-file-format >= "2.1.1"
  292. opentelemetry >= "0.6"
  293. opentelemetry-client-cohttp-lwt >= "0.6"
  294. opentelemetry-client-ocurl >= "0.6"
  295. opentelemetry-cohttp-lwt >= "0.6"
  296. opentelemetry-lwt >= "0.6"
  297. opium >= "0.15.0"
  298. opium-graphql
  299. opium-testing
  300. opium_kernel
  301. orewa
  302. ortac-core
  303. osx-acl
  304. osx-attr
  305. osx-cf
  306. osx-fsevents
  307. osx-membership
  308. osx-mount
  309. osx-xattr
  310. otoggl
  311. owl >= "0.6.0" & != "0.9.0" & != "1.0.0"
  312. owl-base < "0.5.0"
  313. owl-ode >= "0.1.0" & != "0.2.0"
  314. owl-symbolic
  315. passmaker
  316. patch
  317. pbkdf
  318. pecu >= "0.2"
  319. pf-qubes
  320. pg_query >= "0.9.6"
  321. pgx >= "1.0"
  322. pgx_unix >= "1.0"
  323. pgx_value_core
  324. pgx_value_ptime < "2.2"
  325. phylogenetics
  326. piaf
  327. polyglot
  328. polynomial
  329. ppx_blob >= "0.3.0"
  330. ppx_deriving_cmdliner
  331. ppx_deriving_rpc
  332. ppx_deriving_yaml
  333. ppx_graphql >= "0.2.0"
  334. ppx_inline_alcotest
  335. ppx_parser
  336. ppx_protocol_conv >= "5.0.0"
  337. ppx_protocol_conv_json >= "5.0.0"
  338. ppx_protocol_conv_jsonm >= "5.0.0"
  339. ppx_protocol_conv_msgpack >= "5.0.0"
  340. ppx_protocol_conv_xml_light >= "5.0.0"
  341. ppx_protocol_conv_xmlm
  342. ppx_protocol_conv_yaml >= "5.0.0"
  343. ppx_subliner
  344. ppx_units
  345. ppx_yojson >= "1.1.0"
  346. pratter
  347. prc
  348. preface
  349. pretty_expressive
  350. prettym
  351. proc-smaps
  352. producer < "0.2.0"
  353. prom < "0.2"
  354. prometheus < "1.2"
  355. prometheus-app
  356. protocell
  357. protocol-9p >= "0.3" & < "0.11.0" | >= "0.11.2"
  358. protocol-9p-unix
  359. psq
  360. qcheck >= "0.18"
  361. qcheck-alcotest
  362. qcheck-core >= "0.18"
  363. quickjs
  364. radis
  365. randii
  366. reason-standard
  367. reparse >= "2.0.0" & < "3.0.0"
  368. reparse-unix < "2.1.0"
  369. resp < "0.10.0"
  370. resp-unix >= "0.10.0"
  371. rfc1951 < "1.0.0"
  372. routes < "2.0.0"
  373. rpc >= "7.1.0"
  374. rpclib >= "7.1.0"
  375. rpclib-async
  376. rpclib-lwt >= "7.1.0"
  377. rubytt
  378. SZXX >= "4.0.0"
  379. salsa20
  380. salsa20-core
  381. sanddb >= "0.2"
  382. scaml >= "1.5.0"
  383. scrypt-kdf
  384. secp256k1 >= "0.4.1"
  385. secp256k1-internal
  386. semver >= "0.2.1"
  387. sendmail
  388. sendmail-lwt
  389. sendmsg
  390. server-reason-react
  391. session-cookie
  392. session-cookie-async
  393. session-cookie-lwt
  394. sherlodoc
  395. slug
  396. sodium-fmt
  397. spin >= "0.6.0"
  398. squirrel
  399. ssh-agent
  400. ssl >= "0.6.0"
  401. stramon-lib
  402. styled-ppx
  403. syslog-rfc5424
  404. tcpip >= "2.4.2" & < "4.0.0" | >= "5.0.1" & < "7.0.0"
  405. tdigest < "2.1.0"
  406. terminal_size >= "0.1.1"
  407. terminus
  408. terminus-cohttp
  409. terminus-hlc
  410. terml
  411. textrazor
  412. tezos-base-test-helpers < "13.0"
  413. tezos-client-base < "12.0"
  414. tezos-lmdb
  415. tezos-test-helpers < "11.0"
  416. tftp
  417. timedesc
  418. timere
  419. tls >= "0.12.0"
  420. toc
  421. topojson
  422. topojsone
  423. transept
  424. twostep
  425. type_eq
  426. type_id
  427. typebeat
  428. typeid >= "1.0.1"
  429. tyre >= "0.4"
  430. tyxml >= "4.0.0"
  431. tyxml-jsx
  432. tyxml-ppx >= "4.3.0"
  433. tyxml-syntax
  434. uecc
  435. ulid
  436. universal-portal
  437. unix-dirent
  438. unix-errno >= "0.3.0"
  439. unix-fcntl >= "0.3.0"
  440. unix-sys-resource
  441. unix-sys-stat
  442. unix-time
  443. unstrctrd
  444. user-agent-parser
  445. uspf
  446. uspf-lwt
  447. uspf-unix
  448. utop >= "2.13.0"
  449. validate
  450. validator
  451. vercel
  452. vpnkit
  453. wcwidth
  454. websocketaf
  455. x509 >= "0.7.0"
  456. xapi-rrd >= "1.8.2"
  457. xapi-stdext-date
  458. xapi-stdext-encodings
  459. xapi-stdext-std >= "4.16.0"
  460. yaml < "3.2.0"
  461. yaml-sexp
  462. yocaml
  463. yocaml_yaml
  464. yojson >= "1.6.0"
  465. yuscii >= "0.3.0"
  466. zar
  467. zed >= "3.2.2"
  468. zlist < "0.4.0"

Conflicts

None

OCaml

Innovation. Community. Security.

GitHub Discord Twitter Peertube RSS
About
Changelog Releases Industrial Users Academic Users Why OCaml
Ecosystem
Packages Community Events OCaml Planet Jobs
Resources
Install OCaml Get Started Platform Tools Language Manual Standard Library API Books Exercises Papers OCaml Playground Logo
Policies
Carbon Footprint Governance Privacy Code of Conduct