alcotest

Alcotest is a lightweight and colourful test framework
README

Alcotest exposes a simple interface to perform unit tests. It exposes
a simple TESTABLE module type, a check function to assert test
predicates and a run function to perform a list of unit -> unit
test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully
displayed at the end of the run (with the full logs ready to inspect), with a
simple (yet expressive) query language to select the tests to run. See the
manpage
for details.

The API documentation can be found [here][docs]. For information on contributing to Alcotest, see
CONTRIBUTING.md.


Examples

A simple example (taken from examples/simple.ml):

Generated by the following test suite specification:

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use dune exec examples/simple.exe -- --help to see the runtime options.

Here's an example of a of failing test suite:

By default, only the first failing test log is printed to the console (and all
test logs are captured on disk). Pass --show-errors to print all error
messages.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names
of the tests to execute, or by passing a regular expression and a comma-separated list
of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must
filter by test name & number instead.

See the examples
folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any
invocations of the test suite. You should only use `Slow tests for stress
tests that are ran only on occasion (typically before a release or after a major
change). These slow tests can be suppressed by passing the -q flag on the
command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However,
it is also possible to pass an extra option to all the test functions
by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is
read on the command-line, by providing a Cmdliner term for
command-line
arguments

which explains how to parse and serialize values of type 'a (note: do not
use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap
Lwt test cases. The basic idea is that instead of providing a test
function in the form unit -> unit, you provide one with the type
unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you
    and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case
    finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit
but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what
    succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and
    comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random
    generation and property testing (e.g. Quick Check)

  • crowbar
    and bun
    are similar to qcheck, but use compiler-directed randomness,
    e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests
    allows to write tests in the same file as your source-code; they
    will be run only in a special mode of compilation.

Install
Sources
alcotest-js-1.5.0.tbz
sha256=54281907e02d78995df246dc2e10ed182828294ad2059347a1e3a13354848f6c
sha512=1aea91de40795ec4f6603d510107e4b663c1a94bd223f162ad231316d8595e9e098cabbe28a46bdcb588942f3d103d8377373d533bcc7413ba3868a577469b45
Dependencies
odoc
with-doc
uutf
>= "1.0.1"
re
>= "1.7.2"
cmdliner
with-test & < "1.1.0"
cmdliner
>= "1.0.0"
fmt
>= "0.8.7"
ocaml
>= "4.03.0"
dune
>= "2.8"
Reverse Dependencies
alcotest-async
< "1.0.0" | >= "1.5.0"
alcotest-lwt
< "1.0.0" | >= "1.5.0"
angstrom
>= "0.7.0"
anycache
>= "0.7.4"
arp
!= "2.3.1"
arp-mirage
< "2.0.0"
asak
>= "0.2"
asli
>= "0.2.0"
atd
>= "2.3.3"
base64
>= "2.1.2" & < "3.2.0" | >= "3.4.0"
bls12-381
< "0.4.1" | >= "3.0.0"
bls12-381-js
>= "0.4.2"
capnp-rpc-unix
>= "0.9.0"
caqti
>= "1.7.0"
caqti-async
>= "1.7.0"
caqti-lwt
>= "1.7.0"
carton-lwt
>= "0.4.1"
catala
>= "0.6.0"
ccss
>= "1.6"
charrua-server
>= "1.4.1"
checkseum
>= "0.0.3"
cohttp
>= "0.17.0"
colombe
>= "0.2.0"
conduit
= "3.0.0"
conex
< "0.10.0"
cow
>= "2.2.0"
cstruct
>= "3.3.0"
current
>= "0.4"
datakit
>= "0.12.0"
decimal
>= "0.3.0"
decompress
>= "0.8"
digestif
>= "0.8.1"
dispatch
>= "0.4.1"
dns
>= "4.0.0"
dns-client
>= "4.6.0"
dns-forward
< "0.9.0"
dog
< "0.2.1"
dune-release
>= "1.0.0"
duration
>= "0.1.1"
eqaf
>= "0.5"
ezjsonm
>= "0.4.2"
faraday
!= "0.2.0"
fat-filesystem
>= "0.12.0"
functoria
>= "2.2.0"
functoria-runtime
>= "2.2.0" & < "3.0.1" | = "3.1.2"
git
= "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0"
git-unix
>= "1.10.0" & != "2.1.0"
gmap
>= "0.3.0"
graphql-cohttp
>= "0.13.0"
graphql_parser
!= "0.11.0"
graphql_ppx
>= "0.7.1"
hacl_x25519
>= "0.2.0"
httpaf
>= "0.2.0"
icalendar
>= "0.1.4"
imagelib
>= "20200929"
inquire
< "0.2.0"
irmin
< "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & != "2.3.0"
irmin-bench
>= "2.7.0"
irmin-chunk
< "1.3.0" | >= "2.3.0"
irmin-fs
< "1.3.0" | >= "2.3.0"
irmin-git
< "2.0.0" | >= "2.3.0"
irmin-graphql
>= "2.3.0"
irmin-http
< "2.0.0"
irmin-mem
< "1.3.0" | >= "2.3.0"
irmin-pack
>= "2.4.0" & != "2.6.1"
irmin-test
>= "2.2.0"
irmin-unix
>= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
irmin-watcher
!= "0.3.0"
ke
>= "0.2"
lambdapi
>= "2.0.0"
lambdoc
>= "1.0-beta4"
lmdb
>= "1.0"
logtk
>= "1.6"
mdx
>= "1.6.0"
mechaml
= "1.0.0" | >= "1.2.1"
merge-queues
>= "0.2.0"
merge-ropes
>= "0.2.0"
minicaml
= "0.3.1" | >= "0.4"
mirage
>= "4.0.0~beta1"
mirage-channel
>= "4.0.0"
mirage-flow
>= "1.0.2" & < "1.2.0"
mirage-flow-unix
!= "1.3.0" & < "1.5.0" | = "2.0.0" | >= "3.0.0"
mirage-fs-unix
>= "1.2.0" & < "1.4.1"
mirage-kv
>= "2.0.0"
mirage-logs
>= "0.3.0"
mirage-runtime
>= "4.0.0~beta1"
mmdb
< "0.3.0"
mrmime
>= "0.2.0"
msgpck
>= "1.6"
mssql
>= "2.0.3"
nbd
>= "4.0.3"
ocaml-r
>= "0.4.0"
ocaml-version
>= "3.1.0"
ocamlformat
>= "0.13.0" & != "0.19.0~4.13preview"
odoc
>= "1.4.0" & < "2.1.0"
opium
>= "0.15.0"
osnap
< "0.3.0"
owl
>= "0.6.0" & != "0.9.0" & != "1.0.0"
owl-base
< "0.5.0"
owl-ode
>= "0.1.0" & != "0.2.0"
pecu
>= "0.2"
pg_query
>= "0.9.6"
pgx
>= "1.0"
pgx_unix
>= "1.0"
ppx_blob
>= "0.3.0"
ppx_graphql
>= "0.2.0"
ppx_yojson
>= "1.1.0"
prbnmcn-ucb1
>= "0.0.2"
protocol-9p
>= "0.3" & < "0.11.0" | >= "0.11.2"
qcheck
>= "0.18"
qcheck-core
>= "0.18"
reparse
>= "2.0.0" & < "3.0.0"
reparse-unix
< "2.1.0"
resp-unix
>= "0.10.0"
rfc1951
< "1.0.0"
rpc
>= "7.1.0"
rpclib
>= "7.1.0"
rpclib-lwt
>= "7.1.0"
sanddb
>= "0.2"
secp256k1
>= "0.4.1"
sihl
< "0.2.0"
spectrum
>= "0.2.0"
spin
>= "0.7.0"
tcpip
>= "2.4.2" & < "3.4.2" | >= "6.2.0"
terminal_size
>= "0.1.1"
tezos-crypto
>= "8.0" & < "9.0" | >= "11.0" & < "12.0"
tezos-stdlib
>= "8.0" & < "12.0"
tls
>= "0.12.0"
tyre
>= "0.4"
tyxml
>= "4.0.0"
tyxml-ppx
>= "4.3.0"
unix-errno
>= "0.3.0"
unix-fcntl
>= "0.3.0"
x509
>= "0.7.0"
xapi-rrd
>= "1.8.2"
xapi-stdext-std
>= "4.16.0"
yojson
>= "1.6.0"
yuscii
>= "0.3.0"
yuujinchou
>= "1.0.0"
zlist
< "0.4.0"