package qcheck

  1. Overview
  2. Docs
QuickCheck inspired property-based testing for OCaml


Dune Dependency






This module allows to check invariants (properties of some types) over randomly generated instances of the type. It provides combinators for generating instances and printing them.


test property quickcheck

Published: 21 Sep 2018



= QCheck
:toc: macro
:toclevels: 4
:source-highlighter: pygments

QuickCheck inspired property-based testing for OCaml, and combinators to
generate random values to run tests on.

The documentation can be found[here].
This library spent some time in[qtest], but is now
standalone again!
Note that @gasche's[generator library]
can be useful too, for generating random values.


image::[alt="Build Status", link=""]

== Use

See the documentation. I also wrote[a blog post] that explains
how to use it and some design choices; however, be warned that the API
changed in lots of small ways (in the right direction, I hope) so the code
will not work any more.
<<examples>> is an updated version of the blog post's examples.

== Build

    $ make

You can use opam:

    $ opam install qcheck

== License

The code is now released under the BSD license.

== An Introduction to the Library

First, let's see a few tests. Let's open a toplevel (e.g. utop)
and type the following to load QCheck:

#require "qcheck";;

NOTE: alternatively, it is now possible to locally do: `dune utop src`
to load `qcheck`.

=== List Reverse is Involutive

We write a random test for checking that `List.rev (List.rev l) = l` for
any list `l`:

let test =
  QCheck.Test.make ~count:1000 ~name:"list_rev_is_involutive"
   QCheck.(list small_nat)
   (fun l -> List.rev (List.rev l) = l);;

(* we can check right now the property... *)
QCheck.Test.check_exn test;;

In the above example, we applied the combinator `list` to
the random generator `small_nat` (ints between 0 and 100), to create a
new generator of lists of random integers. These builtin generators
come with printers and shrinkers which are handy for outputting and
minimizing a counterexample when a test fails.

Consider the buggy property `List.rev l = l`:

let test =
  QCheck.Test.make ~count:1000 ~name:"my_buggy_test"
   QCheck.(list small_nat)
   (fun l -> List.rev l = l);;

When we run this test we are presented with a counterexample:

# QCheck.Test.check_exn test;;
QCheck.Test.Test_fail ("my_buggy_test", ["[0; 1] (after 23 shrink steps)"]).

In this case QCheck found the minimal counterexample `[0;1]` to the property
`List.rev l = l` and it spent 23 steps shrinking it.

Now, let's run the buggy test with a decent runner that will print the results
nicely (the exact output will change at each run, because of the random seed):

# QCheck_runner.run_tests [test];;

--- Failure --------------------------------------------------------------------

Test my_buggy_test failed (10 shrink steps):

[0; 1]
failure (1 tests failed, 0 tests errored, ran 1 tests)
- : int = 1

For an even nicer output `QCheck_runner.run_tests` also accepts an optional
parameter `~verbose:true`.

=== Mirrors and Trees

`QCheck` provides many useful combinators to write
generators, especially for recursive types, algebraic types,
and tuples.

Let's see how to generate random trees:

type tree = Leaf of int | Node of tree * tree

let leaf x = Leaf x
let node x y = Node (x,y)

let tree_gen = QCheck.Gen.(sized @@ fix
  (fun self n -> match n with
    | 0 -> map leaf nat
    | n ->
        [1, map leaf nat;
         2, map2 node (self (n/2)) (self (n/2))]

(* generate a few trees, just to check what they look like: *)
QCheck.Gen.generate ~n:20 tree_gen;;

let arbitrary_tree =
  let open QCheck.Iter in
  let rec print_tree = function
    | Leaf i -> "Leaf " ^ (string_of_int i)
    | Node (a,b) -> "Node (" ^ (print_tree a) ^ "," ^ (print_tree b) ^ ")"
  let rec shrink_tree = function
    | Leaf i -> i >|= leaf
    | Node (a,b) ->
      of_list [a;b]
      (shrink_tree a >|= fun a' -> node a' b)
      (shrink_tree b >|= fun b' -> node a b')
  QCheck.make tree_gen ~print:print_tree ~shrink:shrink_tree;;


Here we write a generator of random trees, `tree_gen`, using
the `fix` combinator. `fix` is *sized* (it is a function from `int` to
a random generator; in particular for size 0 it returns only leaves).
The `sized` combinator first generates a random size, and then applies
its argument to this size.

Other combinators include monadic abstraction, lifting functions,
generation of lists, arrays, and a choice function.

Then, we define `arbitrary_tree`, a `tree QCheck.arbitrary` value, which
contains everything needed for testing on trees:

- a random generator (mandatory), weighted with `frequency` to
  increase the chance of generating deep trees
- a printer (optional), very useful for printing counterexamples
- a *shrinker* (optional), very useful for trying to reduce big
  counterexamples to small counterexamples that are usually
  more easy to understand. 

The above shrinker strategy is to

- reduce the integer leaves, and
- substitute an internal `Node` with either of its subtrees or
  by splicing in a recursively shrunk subtree.

A range of combinators in `QCheck.Shrink` and `QCheck.Iter` are available
for building shrinking functions.

We can write a failing test using this generator to see the
printer and shrinker in action:

let rec mirror_tree (t:tree) : tree = match t with
  | Leaf _ -> t
  | Node (a,b) -> node (mirror_tree b) (mirror_tree a);;

let test_buggy =
  QCheck.Test.make ~name:"buggy_mirror" ~count:200
    arbitrary_tree (fun t -> t = mirror_tree t);;

QCheck_runner.run_tests [test_buggy];;

This test fails with:


--- Failure --------------------------------------------------------------------

Test mirror_buggy failed (6 shrink steps):

Node (Leaf 0,Leaf 1)
failure (1 tests failed, 0 tests errored, ran 1 tests)
- : int = 1

With the (new found) understanding that mirroring a tree
changes its structure, we can formulate another property
that involves sequentializing its elements in a traversal:

let tree_infix (t:tree): int list =
  let rec aux acc t = match t with
    | Leaf i -> i :: acc
    | Node (a,b) ->
      aux (aux acc b) a
  aux [] t;;

let test_mirror =
  QCheck.Test.make ~name:"mirror_tree" ~count:200
    (fun t -> List.rev (tree_infix t) = tree_infix (mirror_tree t));;

QCheck_runner.run_tests [test_mirror];;


=== Preconditions

The functions `QCheck.assume` and `QCheck.(==>)` can be used for
tests with preconditions.
For instance, `List.hd l :: l = l` only holds for non-empty lists.
Without the precondition, the property is false and will even raise
an exception in some cases.

let test_hd_tl =
    (list int) (fun l ->
      assume (l <> []);
      l = List.hd l :: l));;

QCheck_runner.run_tests [test_hd_tl];;

=== Long tests

It is often useful to have two version of a testsuite: a short one that runs
reasonably fast (so that it is effectively run each time a projet is built),
and a long one that might be more exhaustive (but whose running time makes it
impossible to run at each build). To that end, each test has a 'long' version.
In the long version of a test, the number of tests to run is multiplied by
the `~long_factor` argument of `QCheck.Test.make`.

=== Runners

The module `QCheck_runner` defines several functions to run tests, including
compatibility with `OUnit`.
The easiest one is probably `run_tests`, but if you write your tests in
a separate executable you can also use `run_tests_main` which parses
command line arguments and exits with `0` in case of success,
or an error number otherwise.

=== Integration within OUnit[OUnit] is a popular unit-testing framework
for OCaml.
QCheck provides a sub-library `qcheck-ounit` with some helpers, in `QCheck_ounit`,
to convert its random tests into OUnit tests that can be part of a wider

let passing =
  QCheck.Test.make ~count:1000
    QCheck.(list small_nat)
    (fun l -> List.rev (List.rev l) = l);;

let failing =
  QCheck.Test.make ~count:10
    QCheck.(list small_nat)
    (fun l -> l = List.sort compare l);;

let _ =
  let open OUnit in
    ("tests" >::: QCheck_ounit.to_ounit_test [passing; failing])


NOTE: the package `qcheck` contains the module `QCheck_runner`
which contains both custom runners and OUnit-based runners.

=== Integration within alcotest[Alcotest] is a simple and colorful test framework for
OCaml. QCheck now provides a sub-library `qcheck-alcotest` to
easily integrate into an alcotest test suite:


let passing =
  QCheck.Test.make ~count:1000
    QCheck.(list small_int)
    (fun l -> List.rev (List.rev l) = l);;

let failing =
  QCheck.Test.make ~count:10
    QCheck.(list small_int)
    (fun l -> l = List.sort compare l);;

let () =
  let suite = QCheck_alcotest.to_alcotest
      [ passing; failing]
  in "my test" [
    "suite", suite


=== Compatibility notes

Starting with 0.9, the library is split into several components:

- `qcheck-core` depends only on unix and bytes. It contains the module
  `QCheck` and a `QCheck_base_runner` module with our custom runners.
- `qcheck-ounit` provides an integration layer for `OUnit`
- `qcheck` provides a compatibility API with older versions of qcheck,
  using both `qcheck-core` and `qcheck-ounit`.
  It provides `QCheck_runner` which is similar to older versions and contains
  both custom and Ounit-based runners.
- `qcheck-alcotest` provides an integration layer with `alcotest`

Normally, for contributors,
`opam pin` will pin all these packages.

Dependencies (6)

  1. qcheck-ounit = version
  2. qcheck-core = version
  3. base-unix
  4. base-bytes
  5. ocaml >= "4.03.0"
  6. dune

Dev Dependencies (1)

  1. odoc with-doc

Conflicts (1)

  1. ounit < "2.0"

Innovation. Community. Security.