package tezt-bam

You can search for identifiers within the package.

in-package search v0.2.0

tezt-bam
- Tezt_bam
  - Pbt
  - Test

Legend:
Library
Module
Module type
Parameter
Class
Class type

This module declares a unique function to register a new test within Tezt. A typical way of using this function is as follows:

type t = A | B | C

let my_test =
let gen =
let open Std.Syntax in
  let gen =
    let* i = Std.int ~max:3 () in
    (if i = 0 then A else if i = 1 then B else C) |> return
  in
  Std.list ~size:(Std.int ~max:10 ()) gen
in
let property l =
  List.length l < 5 && List.mem A l
in
Pbt.register ~__FILE__ ~title:"Simple test" ~tags:["pbt";"simple"] ~gen ~property ()

Running this example with Tezt would give:

[18:06:14.776] [pbt] Please run the test again with option --shrink to get a minimal example.
[18:06:14.776] [pbt] Counter example found:
[18:06:14.776] [pbt] C B B C A C
[18:06:14.776] [pbt] Error:
[18:06:14.776] [pbt] Counter-example was found
[18:06:14.776] [pbt] Runtime statistics (before shrinking):
[18:06:14.776] [pbt] Execution time: 1.28ms
[18:06:14.776] [pbt] Minimum execution time: 737ns
[18:06:14.776] [pbt] Maximum execution time: 1.25ms
[18:06:14.776] [pbt] Average execution time: 255μs
[18:06:14.776] [pbt] Number of executions (without shrinking): 5
[18:06:14.776] [pbt] Number of distinct samples (not producing 'Bad_value' error): 5
[18:06:14.776] [pbt] sampling ratio: 1.000000
[18:06:14.776] [pbt] Running property on the smaller counter example found...
[18:06:14.776] [pbt] Total execution time: 1.35ms
[18:06:14.776] [error] Test failed with error:
[18:06:14.776] [error] Counter-example was found
[18:06:14.776] [FAILURE] (1/1, 1 failed) Simple test
[18:06:14.776] Try again with: _build/default/test/main.exe --verbose --file test/pbt.ml --title 'Simple test' --seed 227649358

Following the suggestion provided, if we run the same test with option --shrink and option --seed 227649358 to get the same counter-example we get this time:

[18:06:37.025] Starting test: Simple test
[18:06:37.025] [pbt] Start searching for a counter example...
[18:06:37.026] [pbt] Counter example found:
[18:06:37.026] [pbt] C B B C A C
[18:06:37.026] [pbt] Error:
[18:06:37.026] [pbt] Counter-example was found
[18:06:37.026] [pbt] Runtime statistics (before shrinking):
[18:06:37.026] [pbt] Execution time: 1.97ms
[18:06:37.026] [pbt] Minimum execution time: 1.75μs
[18:06:37.026] [pbt] Maximum execution time: 1.93ms
[18:06:37.026] [pbt] Average execution time: 394μs
[18:06:37.026] [pbt] Number of executions (without shrinking): 5
[18:06:37.026] [pbt] Number of distinct samples (not producing 'Bad_value' error): 5
[18:06:37.026] [pbt] sampling ratio: 1.000000
[18:06:37.026] [pbt] Start shrinking...
[18:06:37.027] [pbt] Smaller counter example found:
[18:06:37.027] [pbt] B B
[18:06:37.027] [pbt] Running property on the smaller counter example found...
[18:06:37.027] [pbt] Total execution time: 2.13ms
[18:06:37.027] [error] Test failed with error:
[18:06:37.027] [error] Counter-example was found
[18:06:37.027] [FAILURE] (1/1, 1 failed) Simple test
[18:06:37.027] Try again with: _build/default/test/main.exe --verbose --file test/pbt.ml --title 'Simple test' --seed 227649358

The framework did not find the smalletest counter-example. We can make the default shrinking strategy more aggressive. For example by changing the generator as dollows:

let gen =
  let open Std.Syntax in
  let gen =
    let* i = Std.int ~max:3 () in
    (if i = 0 then A else if i = 1 then B else C) |> return
  in
  Std.list ~shrinker:(Skip `Auto) ~size:(Std.int ~max:10 ()) gen

Please, refers to Bam.Std.Shrinker.t.Skip for more detailed explanations.

val register : 
  ?hash:('a -> int) ->
  ?pp:(Format.formatter -> 'a -> unit) ->
  ?compute_execution_statistics:bool ->
  ?expected_sampling_ratio:float ->
  ?minimum_number_of_samples:int ->
  ?log_statistics_frequency:int ->
  ?regression:'a list ->
  ?stop_after:[ `Timeout of float | `Count of int | `Loop ] ->
  ?on_sample:('a -> unit) ->
  __FILE__:string ->
  title:string ->
  tags:string list ->
  gen:'a Gen.t ->
  property:('a -> ('b, [ `Fail of string | `Bad_value ]) Result.t) ->
  unit ->
  unit

register ?hash ?pp ?expected_sampling_ratio ?minimum_number_of_samples ?regression ?stop_after ?on_sample gen property runs the property on random values generated with gen.

The property is run on as many samples as possible depending on stop_after. gen is called any time a new simple must be provided.

The seed used to generate samples is controlled by Tezt (see Test.register). This function ensures that if the test is run twice with the same seed, then the same values will be generated.

The property will be called on each generated sample produced by gen. It does so in an environment where the output on both stdout and stderr is captured by default. Except for the counter-example found for which, property is run again without capturing anything. This behaviour should ease the debugging (especially hello debugging).

The test can fail in the following cases:

1) A counter example was found

2) The last run with the counter example succeeded

3) Not enough samples were generated

The first case is expected. The reason for the second case, is likely related to non-determinism of the property since the test was run twice with the same generated value. For the third case, this is to ensure the property is run on many samples as detailed below.

One issue with property-based testing is that it is not easy to ensure that the generator and the property are run an a large range of values. The default behaviour of this register function is to prevent such a case to occur. To do so, two parameters are used: expected_sampling_ratio andminimum_number_of_samples.

compute_execution_statistics enables to compute statistics on the test. It will compute the number of distinct values generated, or statistics about the execution time of each sample. Its default value is true unless shrinking is activated or stop_after is `Loop. The reason is that this may slow down the execution of the test which is superfluous when we shrink or we try to find a counter-example.

expected_sampling_ratio ensures that the generator generates in general different sample. Its default value is 0.10 if compute_execution_statistics is true, 0.0 otherwise.

minimum_number_of_samples ensures that the property is run at least that numbero f times. Its default value is 50 if compute_execution-statistics is true, 0 otherwise.

log_statistics_frequency controls the frequency (in seconds) at which a log shows some execution statistics. Don't forget to use --no-capture if you want to see the log on stdout, see the corresponding documentation for this option. A negative value for this parameter combined with compute_execution_statistics sets to false ensures no statistics are computed for this test. Its default value is 2.

The default values should ensure that we can capture obvious erroneous cases where the test is not executed on enough samples without being too restrictive.

stop_after allows to determine how many samples should be drawn to run the property on. By default it is set to `Timeout 0.10, meaning that the test will stop after 100ms. It can be set to `Count n to decide a precise number of times the test should be run. Do note that the default value depends on the CLI arguments provided by the user. If --loop was set, then the default value is `Loop. While if `--loop-count n was set with n > 1, the default value will be `Count n.

on_sample is called any time a new value is generated.

If pp is provided, it can be used to print the counter-example found.

If hash is provided, it will be used to hash values. This function is used to count the number of distinct samples. Default value is Stdlib.Hashtbl.hash. This function may generate a lot of collisions on complex values. You can redefine a better hash function via ppx-hash for example.

regression can be instantiated to be sure that some fixed deterministic examples is always run.

This function can be influenced with two options from the command-line:

--shrink triggers the shrinker to find a smaller counter-example. An explanation about shrinking can be found in the documentation of the package Bam.

--no-capture so that output is not captured. Its default value is true unless stop_after is set to `Loop.