package core_bench

  1. Overview
  2. Docs

Core_bench is a micro-benchmarking library for OCaml that can measure execution costs of operations that take 1ns to about 100ms. Core_bench tries to measure execution costs of such short-lived computations precisely while trying to account for delayed GC costs and noise introduced by other activity on the system.

The easiest way to get started is using an example:

open! Core
open Core_bench

let () =
  Random.self_init ();
  let x = Random.float 10.0 in
  let y = Random.float 10.0 in
  Command.run (Bench.make_command [
    Bench.Test.create ~name:"Float add" (fun () ->
      ignore (x +. y));
    Bench.Test.create ~name:"Float mul" (fun () ->
      ignore (x *. y));
    Bench.Test.create ~name:"Float div" (fun () ->
      ignore (x /. y));
  ])

When compiled this gives you an executable:

$ ./z.exe -ascii
Estimated testing time 30s (3 benchmarks x 10s). Change using -quota SECS.

  Name        Time/Run   mWd/Run   Percentage
 ----------- ---------- --------- ------------
  Float add     2.50ns     2.00w       41.70%
  Float mul     2.55ns     2.00w       42.52%
  Float div     5.99ns     2.00w      100.00%

If any of the functions resulted in allocation on the major heap (mjWd) or promotions (Prom), columns corresponding to those would be automatically displayed. Columns that do not have significant values are not displayed by default. The most common options one would want to change are the `-q` flag which controls the time quota for testing and enabling/disabling specific columns. For example:

$ ./z.exe -ascii -q 1 cycles
Estimated testing time 3s (3 benchmarks x 1s). Change using -quota SECS.

  Name        Time/Run   Cycls/Run   mWd/Run   Percentage
 ----------- ---------- ----------- --------- ------------
  Float add     2.50ns       8.49c     2.00w       41.78%
  Float mul     2.77ns       9.40c     2.00w       46.29%
  Float div     5.99ns      20.31c     2.00w      100.00%

If you drop the `-ascii` flag, the output table uses extended Ascii characters. These display well on most modern terminals, but not on ocamldoc.

The simplest benchmark specification is just a unit -> unit thunk and a name:

Bench.Test.create ~name:"Float add" (fun () -> ignore (x +. y));

One can also create indexed benchmarks, which can be helpful in understanding non-linearities in the execution profiles of functions. For example:

open! Core
open Core_bench

let () =
 Command.run (Bench.make_command [
   Bench.Test.create_indexed
     ~name:"Array.create"
     ~args:[1; 10; 100; 200; 300; 400]
     (fun len ->
        Staged.stage (fun () -> ignore(Array.create ~len 0)));
 ])

this produces:

$ ./z.exe -ascii -q 3
Estimated testing time 18s (6 benchmarks x 3s). Change using -quota SECS.

  Name                 Time/Run   mWd/Run   mjWd/Run   Percentage
 ------------------ ------------ --------- ---------- ------------
  Array.create:1        27.23ns     2.00w                   1.08%
  Array.create:10       38.79ns    11.00w                   1.53%
  Array.create:100     124.05ns   101.00w                   4.91%
  Array.create:200     188.13ns   201.00w                   7.44%
  Array.create:300   1_887.20ns              301.00w       74.64%
  Array.create:400   2_528.43ns              401.00w      100.00%

Executables produced using Bench.make_command are self documenting (use the `-?` flag). The documentation in the executable also closely corresponds to the functionality exposed through the .mli and is a great way to interactively explore what the various options do.

module Test : sig ... end

Test.t are benchmarked by calls to bench.

module Variable : sig ... end

Variable.ts represent variables than can be used as predictors or the responder when specifying a regression.

module Quota : sig ... end

A quota can be specified as an amount of wall time, or a number of times to run the function.

module Run_config : sig ... end

Run_config.t specifies how a benchmark should be run.

module Display_config : sig ... end

Display_config.t specifies how the output tables should be formatted.

module Analysis_config : sig ... end

Each Analysis_config.t specifies a regression run by Core_bench. This module also provides several typical regressions that one might want to run.

module Analysis_result : sig ... end

Results of a benchmark analysis, including all the regressions.

module Measurement : sig ... end

A Measurement.t represents the result of measuring execution of a Test.t. It is used as input for subsequent analysis.

val make_command : Test.t list -> Core.Command.t

make_command tests is the easiest way to generate a command-line program that runs a list of benchmarks. Here tests : Test.t list are the benchmarks that should be run. This returns a Command.t which provides a command-line interface for running the benchmarks. See notes above for an example.

val bench : ?run_config:Run_config.t -> ?analysis_configs:Analysis_config.t list -> ?display_config:Display_config.t -> ?save_to_file:(Measurement.t -> string) -> ?libname:string -> Test.t list -> unit

bench tests will run, analyze and display the specified tests. Use this when one needs more control over the execution parameters that what is exposed through make_command. bench can also save the measurements of each test to the filename returned by save_to_file.

val measure : ?run_config:Run_config.t -> Test.t list -> Measurement.t list

measure is a fragment of the functionality of bench. measure tests will run the specified tests and return the resulting measurement results.

val analyze : ?analysis_configs:Analysis_config.t list -> Measurement.t -> Analysis_result.t Core.Or_error.t

analyze is a fragment of the functionality of bench. analyze ~analysis_configs m will analyze the measurement m using the regressions specified.

val display : ?libname:string -> ?display_config:Display_config.t -> Analysis_result.t list -> unit

display is a fragment of the functionality of bench. display results will display a tabular summary of results on the terminal.

val make_command_ext : summary:string -> ((Analysis_config.t list * Display_config.t * [ `From_file of string list | `Run of (Measurement.t -> string) option * Run_config.t ]) -> unit) Core.Command.Param.t -> Core.Command.t

make_command_ext is useful for creating Command.ts that have command line flags in addition to those provided by make_command.

OCaml

Innovation. Community. Security.