Module HegelSource

Introduction

Property-based testing for OCaml, powered by the native Hegel engine based on Hypothesis.

Hegel runs the test function on many generated inputs. You generate data inline, drawing values with draw as the test runs, rather than generating the data then running the property body. Each draw returns an ordinary OCaml value that you bind with let, compute with, and branch on, so a later draw can depend on an earlier generated value or a value from the system under test.

Because Hegel uses integrated shrinking, shrinking comes for free.

Getting started

Install Hegel

To install Hegel for OCaml:

  opam install hegel

The version of Hegel in OPAM sometimes lags behind the version in Github. To pin the version in Github:

  opam pin add hegel "git+https://github.com/hegeldev/hegel-ocaml.git"

Hegel for OCaml supports Linux (amd64/arm64) and macOS (Apple Silicon). macOS amd64 (Intel) has no published libhegel artifact, so on that platform point HEGEL_LIBHEGEL_PATH at a locally built libhegel.dylib.

Hegel works with whatever test framework your project already uses. The examples below use Alcotest.

Add hegel and alcotest to your dune test stanza. The examples in this documentation also use ppx_sexp_conv's [%sexp_of: t] to write value printers.

  (test
   (name my_tests)
   (libraries hegel alcotest)
   (preprocess (pps ppx_hegel_test ppx_sexp_conv)))

Write your first test

Write a property test using let%hegel_test:

  open Hegel

  let%hegel_test commutative_addition tc =
    let a = draw tc (integers ~min_value:(-1000) ~max_value:1000 ()) in
    let b = draw tc (integers ~min_value:(-1000) ~max_value:1000 ()) in
    require_equal tc [%sexp_of: int] (a + b) (b + a)

  let () =
    Alcotest.run
      "my_tests"
      [ "properties", [ Alcotest.test_case "commutative addition" `Quick commutative_addition ] ]

We check the property with require_equal rather than assert (a + b = b + a). It takes a printer for the values and, when they differ, shows a structural diff of the two sides in the failure report instead of a bare "assertion failed". Use require for a boolean check with a custom message. An assert can be used as well, but it does not provide as much information as require_equal and require.

Run dune runtest. You should see Alcotest report the test as passing. Hegel generates up to 100 random input pairs and reports the minimal counterexample if it finds one. When a test fails, Hegel prints each value you drew from the failing case, named after the let binding it was bound to (a = …, b = …).

The rest of the examples in the documentation assume you have open Hegel at the top of the test file like in the example above.

Next, let's try a test that fails.

  let%hegel_test every_int_is_small tc =
    let n = draw tc (integers ()) in
    assert (n < 50)

This test asserts that any integer is less than 50, which is obviously incorrect. Hegel finds a test case that makes the assertion fail, then shrinks it to the smallest counterexample (n = 50). The final replay prints the drawn values, the exception, and a rerun with: line that replays the exact case:

  --- Failure: every_int_is_small (my_tests.ml:3) ------------------
  Falsified after 1 test case (0 discarded):

    n = 50

  Exception: File "my_tests.ml", line 5, characters 2-8: Assertion failed
  rerun with: [@@failure_blobs [ "AAEAAAAACgEAAAAy" ]]

To fix this test, you can constrain the integers you generate with min_value and max_value:

  let%hegel_test every_int_is_small tc =
    let n = draw tc (integers ~min_value:0 ~max_value:49 ()) in
    assert (n < 50)

Use generators

Hegel provides a rich library of generators that you can use out of the box. See Generators for the full reference.

For instance, you can use lists to construct a list of integers:

  let%hegel_test append_increases_length tc =
    let xs = draw tc (lists (integers ()) ()) in
    let initial_length = List.length xs in
    let xs = draw tc (integers ()) :: xs in
    require tc ~msg:"prepending an element must grow the list"
      (List.length xs > initial_length)

Custom generators are also supported. Suppose you have a person record that requires generation. Build a generator for it with composite, drawing each field in sequence:

  type person =
    { age : int
    ; name : string
    }

  let person =
    composite (fun tc ->
      let age = draw_silent tc (integers ()) in
      let name = draw_silent tc (text ()) in
      { age; name })

You can chain drawing operations together, so a later draw depends on an earlier one. For instance, extending person with a driving_license field that can only be true once age is at least 18:

  type person =
    { age : int
    ; name : string
    ; driving_license : bool
    }

  let person =
    composite (fun tc ->
      let age = draw_silent tc (integers ()) in
      let name = draw_silent tc (text ()) in
      let driving_license =
        if age >= 18 then draw_silent tc (booleans ()) else false
      in
      { age; name; driving_license })

Derive generators

Annotate the type with [@@deriving hegel_generator] and add ppx_hegel_generator to your preprocess stanza:

  (test
   (name my_tests)
   (libraries hegel alcotest)
   (preprocess (pps ppx_hegel_test ppx_hegel_generator)))

open Hegel is required before the first @@deriving hegel_generator, unless you want to derive generators for Core types (see Hegel_jane.Derive).

  open Hegel

  type point =
    { x : int
    ; y : int
    }
  [@@deriving hegel_generator]

  let%hegel_test point_roundtrip tc =
    let p = draw tc hegel_generator_point in
    assert ({ x = p.x; y = p.y } = p)

The type t derives a value named hegel_generator. Any other type foo derives hegel_generator_foo. Derived generators print drawn values as s-expressions.

Deriving generators also works on types in modules:

  module Temperature = struct
    type t = { celsius : float } [@@deriving hegel_generator]
  end

  type reading =
    { sensor_id : int
    ; temp : Temperature.t
    }
  [@@deriving hegel_generator]

The reading generator draws its temp field through Temperature.hegel_generator.

[@hegel.generator expr] sets the generator for a type instead of deriving it:

  type ranked =
    { name : string
    ; level :
        (int[@hegel.generator integers ~min_value:1 ~max_value:5 ()])
    }
  [@@deriving hegel_generator]

[@hegel.do_not_generate] excludes a variant constructor from being generated.

  type response =
    | Ok_response of int
    | Errored of exn [@hegel.do_not_generate]
  [@@deriving hegel_generator]

If a field's type has no sexp_of_* representation, mark the field [@sexp.opaque]:

  type connection = { send : bytes -> unit }

  type session =
    { id : int
    ; conn : (connection [@sexp.opaque])
    }
  [@@deriving hegel_generator]

The conn field prints as <opaque>.

Changing test settings

To override the default settings, attach a [@@settings ...] attribute:

  let%hegel_test commutative_addition tc =
    let a = draw tc (integers ()) in
    let b = draw tc (integers ()) in
    require_equal tc [%sexp_of: int] (a + b) (b + a)
  [@@settings Settings.create ~test_cases:500 ()]

This increases the number of test cases run from 100 to 500.

Settings can be changed with record update syntax.

  let%hegel_test commutative_addition tc =
    let a = draw tc (integers ()) in
    let b = draw tc (integers ()) in
    require_equal tc [%sexp_of: int] (a + b) (b + a)
  [@@settings { (Settings.create ~test_cases:500 ()) with verbosity = Settings.Verbose }]

Debugging failing test cases

Use note to attach debug information:

  let%hegel_test every_int_is_small tc =
    let n = draw tc (integers ()) in
    note tc (Printf.sprintf "n is %d" n);
    assert (n < 50)

A failing run prints a framed report: the shrunk counterexample's draws and notes, the exception, and a copy-pasteable rerun with: line whose base64 blob encodes the choice sequence that caused the failure (disable it with print_blob = false). On a terminal the report headers (and require_equal diffs) print in color; set HEGEL_COLOR to 1 or 0 to force colors on or off.

For an equality property, prefer require_equal over assert (x = y): it adds a structural diff of the two values to this report, so you see exactly how they differ. require is the message-carrying boolean variant.

  let%hegel_test every_int_is_small tc =
    let n = draw tc (integers ()) in
    assert (n < 50)

  --- Failure: every_int_is_small (my_tests.ml:3) ------------------
  Falsified after 2 test cases (0 discarded):

    n = 50

  Exception: File "my_tests.ml", line 5, characters 2-8: Assertion failed
  rerun with: [@@failure_blobs [ "AAEAAAAACgEAAAAy" ]]

The blob can then be used to replay the failing test case:

  let%hegel_test every_int_is_small tc =
    let n = draw tc (integers ()) in
    assert (n < 50)
  [@@failure_blobs [ "AAEAAAAACgEAAAAy" ]]

The blob is only meant to reproduce the failure within a specific version of Hegel, since the choice sequence leading to a failure can change from version to version.

Jane Street Core support

Projects that use Jane Street's Core can also use the optional Hegel_jane library to generate Core values. See Hegel_jane.

Learning more

See Generators for the generators and Stateful for state-machine testing.

Hegel module documentation

Submodules

Settings

Running tests

  let my_settings = Settings.create ~test_cases:50 ~seed:5 () in
  let () =
    run_hegel_test ~settings:my_settings (fun tc ->
      let n = draw tc (integers ~min_value:0 ~max_value:9 ()) in
      assert (n >= 0 && n <= 9))

Drawing values

  let%hegel_test draw_example tc =
    let n = draw tc (integers ~min_value:0 ~max_value:100 ()) in
    assert (n >= 0)

  let%hegel_test draw_silent_example tc =
    let n = draw_silent tc (map (fun x -> x * 2) (integers ~min_value:0 ~max_value:9 ())) in
    assert (n >= 0)

Guiding generation

  let%hegel_test head_cons_tail_reconstructs tc =
    let xs = draw tc (lists (integers ()) ()) in
    (* The property is only meaningful for non-empty lists. *)
    assume tc (xs <> []);
    assert (List.hd xs :: List.tl xs = xs)

  let%hegel_test grow_size tc =
    let v = draw tc (integers ~min_value:0 ~max_value:1000 ()) in
    target tc ~label:"size" ~value:(float_of_int v);
    assert (v <= 1000)

Collecting statistics

With show_statistics enabled in Settings.t, every run prints test statistics.

  let%hegel_test list_statistics tc =
    let xs = draw_silent tc (lists (integers ()) ()) in
    (match xs with
     | [] -> event tc ~label:"empty input"
     | _ -> ());
    event_value tc ~label:"length" ~value:(float_of_int (List.length xs))

prints:

  Statistics (over 100 test cases):
    * empty input: 5.0% of test cases
    * length: count 100, min 0, median 4, mean 5.12, p90 9, max 15

  let%hegel_test observe_emptiness tc =
    let xs = draw tc (lists (integers ()) ()) in
    if List.is_empty xs then event tc ~label:"empty input";
    assert (List.length (List.sort compare xs) = List.length xs)

  let%hegel_test observe_length tc =
    let xs = draw tc (lists (integers ()) ()) in
    event_value tc ~label:"length" ~value:(float_of_int (List.length xs));
    assert (List.length (List.sort compare xs) = List.length xs)

Debugging tests

  let%hegel_test note_value tc =
    let n = draw tc (integers ~min_value:0 ~max_value:99 ()) in
    note tc (Printf.sprintf "n is %d" n);
    assert (n < 100)

  let%hegel_test balanced tc =
    let l = draw tc (lists (integers ()) ()) in
    require tc ~msg:"sum must stay non-negative" (running_sum l >= 0)

  let%hegel_test sort_is_stable tc =
    let l = draw tc (lists (integers ()) ()) in
    require_equal
      tc
      [%sexp_of: int list]
      (List.sort compare l)
      (stable_sort l)

  let%hegel_test with_printer_example tc =
    let doubled = map (fun x -> x * 2) (integers ~min_value:0 ~max_value:9 ()) in
    let n = draw tc (with_printer [%sexp_of: int] doubled) in
    assert (n >= 0)

Concurrency and parallelism

Hegel can drive generation from more than one thread or domain within a single test. Two rules govern it.

First, test-case handles may not be shared. A single handle must be drawn from by one thread at a time, so give each thread its own clone using clone. A clone has its own choice sequence. Drawing from one shared handle on multiple threads throws a concurrent-use error. Concurrently driving one shared collection, pool, or state machine will likely produce flaky results, so always make a new one per unit of concurrency/parallelism.

Second, a draw is a synchronous engine call that holds its domain's runtime lock and never yields, so it cannot cooperate with an event loop or overlap another draw on the same domain.

As long as you follow these two rules and your code is deterministic, you will be able to replay failures.

Some advice for common concurrency/parallelism libraries:

Use Threads for interleaving of concurrent operations and overlapping blocking work, not parallel generation, since draws serialize under the runtime lock. You should use spawn / join rather than Thread.create. Thread.join drops a worker's exception, whereas join re-raises it into the runner.

  let%hegel_test concurrent_workers tc =
    let w = spawn tc (fun worker -> draw_silent worker gen) in
    let mine = draw_silent tc gen in
    ignore (mine, join w)

Use Domainslib or any domain pool for when you need true parallelism, such as higher generation throughput. We strongly recommend that you do not use domains directly, as they are expensive to create and destruct. Set up the pool once and reuse it. Clone up front then Task.async each clone and Task.await it.

  (* the pool is created once and reused across cases *)
  let pool = Domainslib.Task.setup_pool ~num_domains:2 ()

  let%hegel_test parallel_generation tc =
    Domainslib.Task.run pool (fun () ->
      let worker = clone tc in
      let p = Domainslib.Task.async pool (fun () -> draw_silent worker gen) in
      let mine = draw_silent tc gen in
      ignore (mine, Domainslib.Task.await pool p))

Use Eio for concurrent generation with structured concurrency or if your code already uses Eio. Each fiber should draw its own data from its own clone. Since a draw does not yield, only separate domains make draws truly parallel. Here two workers race increments onto a shared atomic. The property is that no update is lost.

  Eio_main.run @@ fun env ->
  let dmgr = Eio.Stdenv.domain_mgr env in
  run_hegel_test (fun tc ->
    let counter = Atomic.make 0 in
    let amounts = integers ~min_value:0 ~max_value:100 () in
    let worker g () =
      Eio.Domain_manager.run dmgr (fun () ->
        let n = draw_silent g amounts in
        ignore (Atomic.fetch_and_add counter n : int);
        n)
    in
    let worker_b = clone tc in
    let sum_a, sum_b = Eio.Fiber.pair (worker tc) (worker worker_b) in
    require_equal tc [%sexp_of: int] (sum_a + sum_b) (Atomic.get counter))

  let%hegel_test two_hands_two_dice_manual tc =
    let die = integers ~min_value:1 ~max_value:6 () in
    let other_hand = clone tc in
    let out = ref (Error (Failure "unset")) in
    let rolling =
      Thread.create
        (fun () -> out := (try Ok (draw_silent other_hand die) with e -> Error e))
        ()
    in
    let right_hand = draw_silent tc die in
    Thread.join rolling;
    match !out with
    | Ok left_hand -> assert (right_hand + left_hand >= 2)
    | Error e -> raise e

  let%hegel_test two_hands_two_dice tc =
    let die = integers ~min_value:1 ~max_value:6 () in
    let other_hand = spawn tc (fun worker -> draw_silent worker die) in
    let this_hand = draw_silent tc die in
    assert (this_hand + join other_hand >= 2)