package windtrap

Windtrap — One library for all your OCaml tests

Windtrap runs unit, property, stateful and expect tests from one flat interface, with coverage and mutation testing in the companion ppx. A suite is an ordinary executable that dune runtest builds and runs.

A first suite

open Windtrap

let basics =
  group "basics"
    [
      test "evicts the oldest entry when full" (fun () ->
          let c = Lru.create 2 in
          List.iter (fun k -> Lru.add c k k) [ 1; 2; 3 ];
          equal (option int) None (Lru.find c 1));
      test "lists its keys, most recent first" (fun () ->
          let c = Lru.create 3 in
          List.iter (fun k -> Lru.add c k k) [ 1; 2; 3 ];
          ignore (Lru.find c 1);
          expect (Lru.to_string c) @@ __POS_OF__ {|1 3 2|});
    ]

let bounded =
  prop "never holds more than its capacity" Gen.(list int) (fun keys ->
      let c = Lru.create 3 in
      List.iter (fun k -> Lru.add c k k) keys;
      at_most int ~than:3 (Lru.size c))

let () = exit (run "lru" [ basics; bounded ])

A run with nothing to report prints one line:

$ dune runtest
mylib: 3 passed in 1.0ms (seed s1:b02192cebcec40d2).

Features

Unit tests

test and group declare a suite, and run runs it. Assertions such as equal, less, contains, raises and require_some take a witness, for example int or list string, and a failure prints the values it compared.

test "splits on commas" (fun () ->
    equal (list string) [ "a"; "b" ] (String.split_on_char ',' "a,b"))

Property tests

prop checks a law over values generated with Gen. A failing input is reduced to a smaller one that still fails. Law states textbook laws, such as an involution or a round trip, and a failure names each term.

prop "rev is an involution" Gen.(list int) (Law.involutive (list int) List.rev)

Stateful tests

stateful runs generated programs of calls on a system and on a reference, such as a model of its state or another implementation, and compares what each call returns or raises. A failing program is reduced to a shorter one and printed as the calls that ran. With ~domains:2, the middle of each program runs on two domains at once, and the test fails when no order of the calls explains the results.

command "pop" (queue ^-> returns int) Model.pop Bounded_queue.pop

Expect tests

expect compares a string with a literal in the test's source, and expect_file with a file. ppx_windtrap provides let%expect_test and [%expect]. dune promote accepts a change.

expect (Printf.sprintf "%d items" (List.length cart)) @@ __POS_OF__ {|3 items|}

Coverage

ppx_windtrap.coverage instruments a library, and windtrap coverage reports the expressions that the tests did not run.

dune runtest --instrument-with ppx_windtrap.coverage
dune exec windtrap -- coverage

Mutation testing

ppx_windtrap.mutate compiles mutants of a library, small changes such as >= into >, into its test executables. --mutate runs the tests on each mutant and reports the mutants that no test fails on.

dune exec --instrument-with ppx_windtrap.mutate test/test_calc.exe -- --mutate

Installation

    opam install windtrap
opam install ppx_windtrap   # expect tests, coverage, mutation testing

Documentation

The manual, doc/manual/, has one page per need:

• Tutorial: Getting started, the suite above and its first failure.

• How-to:

  • Assertions: values, bounds, strings, results and exceptions.
  • Property testing: laws over generated values.
  • Stateful testing: a system against a reference.
  • Baselines and expect tests: expect, expect_file, let%expect_test.
  • Resources and structure: a suite's layout and resources.
  • Running tests: selection, reruns, dune runtest and CI.
  • Coverage: the code no test runs.
  • Mutation testing: the changes no test notices.
  • Migrating from 0.1: each 0.1 spelling and its replacement.
• Explanation: Design notes, why windtrap is shaped as it is.
• Reference: lib/windtrap.mli, and ppx/ppx_windtrap.mli for the inline test forms.

Questions are welcome on the OCaml forum.

Examples

examples/ holds the project of each manual page, run by dune runtest; its README lists them.

Acknowledgments

Windtrap builds on ideas and code from several OCaml projects:

• Alcotest by Thomas Gazagnaire: test structure and runner design.
• Craig Ferguson's Alcotest PRs (#294, #247): API design, subcomponent diffing, and Levenshtein distance (ISC).
• QCheck2 by Simon Cruanes et al.: generator design and integrated shrinking (BSD 2-Clause).
• ppx_expect and ppx_inline_test by Jane Street: expect test paradigm and dune integration.
• Bisect_ppx by Anton Bachin et al.: coverage instrumentation and runtime (MIT).
• mtime by Daniel Bünzli: the monotonic clock (ISC).

THIRD_PARTY_LICENSES.md holds the notices of the code derived from them.