package bisect_ppx

Bisect_ppx  

Bisect_ppx is a code coverage tool for OCaml and Reason. It helps you test thoroughly by showing what's not tested.

You can browse the report seen above online here. The details of how it is generated are in the worked example.

Table of contents

• Usage

  • Dune   (starter repo, report)

  • esy   (starter repo, report)

  • BuckleScript   (starter repo, report)

  • Js_of_ocaml   (starter repo, report)

  • Ocamlfind, Ocamlbuild, and OASIS

• Sending to Coveralls

• Controlling coverage with [@coverage off]

• Real-world example examined   (repo, report, coveralls)

• Other topics

• Bisect_ppx users

• Contributing

Usage

Dune

Refer to aantron/bisect-starter-dune, which produces this report.

1. Depend on Bisect_ppx in your opam file:

   depends: [
     "bisect_ppx" {dev & >= "2.0.0"}
   ]
   

2. Mark the code under test for preprocessing by bisect_ppx (but don't preprocess the tester itself):

   (library
    (public_name my_lib)
    (preprocess (pps bisect_ppx --conditional)))
   

3. Build and run your test binary. In addition to testing your code, when exiting, it will write one or more files with names like bisect0123456789.coverage.

   BISECT_ENABLE=yes dune runtest --force
   

   BISECT_ENABLE=yes turns on instrumentation. If you run dune runtest (or any other build) without it, Bisect_ppx will pass the code through unchanged (i.e., without adding instrumentation).

   At the moment, if you switch between building with and without instrumentation, you have to run dune clean in between the builds.

4. Generate the coverage report in _coverage/index.html:

   bisect-ppx-report html
   

5. For releasing, you have two options:

   • If you don't want a dependency on package bisect_ppx, you have to manually remove (preprocess (pps bisect_ppx)) from your dune files.

   • If a dependency on Bisect_ppx is okay, you don't have to do anything. A regular build command, without BISECT_ENABLE=yes will produce uninstrumented code:

     ["dune" "build" "-p" name "-j" $jobs]
     

   This choice is due to a limitation of Dune that we hope to address in ocaml/dune#57. After that, BISECT_ENABLE=yes won't be necessary anymore, and you will be able to release without a dependency on Bisect_ppx, without having to edit any files.

esy

Refer to aantron/bisect-starter-esy, which produces this report.

The instructions are the same as for regular Dune usage, but...

1. Depend on Bisect_ppx in package.json, instead of in an opam file:

   "devDependencies": {
     "@opam/bisect_ppx": "^2.0.0",
   }
   

2. Use the esy command for the build and for running binaries:

   esy install
   BISECT_ENABLE=yes esy dune runtest --force
   esy dune exec bisect-ppx-report -- html
   

BuckleScript

Refer to aantron/bisect-starter-bsb, which produces this report.

1. Depend on Bisect_ppx in package.json, and install it:

   "devDependencies": {
     "bisect_ppx": "^2.0.0"
   },
   "dependencies": {
     "bs-platform": "*"
   }
   

   npm install
   

   This points bisect_ppx to the binaries branch, so the installation should be quite fast.

   If pre-built binaries aren't available for your system, the build will automatically fall back to building Bisect_ppx from source using esy, which will take a few minutes the first time. If this happens, you may need to install esy, if it is not already installed:

   npm install -g esy
   npm install
   

2. Add Bisect_ppx to your bsconfig.json:

   "bs-dependencies": [
     "bisect_ppx"
   ],
   "ppx-flags": [
     "bisect_ppx/ppx"
   ]
   

3. If your tests will be running on Node, call this function somewhere in your tester, which will have Node write a file like bisect0123456789.coverage when the tester exits:

   Bisect.Runtime.write_coverage_data_on_exit();
   

   If the tests will be running in the browser, at the end of testing, call

   Bisect.Runtime.get_coverage_data();
   

   This returns binary coverage data in a string option, which you should upload or otherwise get out of the browser, and write into an .coverage file yourself.

4. Build in development with BISECT_ENABLE=yes, run tests, and generate the coverage report in _coverage/index.html:

   BISECT_ENABLE=yes npm run build
   npm run test
   npx bisect-ppx-report.exe html
   

Js_of_ocaml

Refer to aantron/bisect-starter-jsoo, which produces this report.

1. Follow the Dune instructions above, except that the final test script must be linked with bisect_ppx.runtime (but not instrumented):

   (executable
    (name my_tester)
    (libraries bisect_ppx.runtime))
   

2. If the tests will run on Node, call this function at the end of testing to write bisect0123456789.coverage:

   Bisect.Runtime.write_coverage_data ()
   

   If the tests will run in the browser, call

   Bisect.Runtime.get_coverage_data ()
   

   to get binary coverage data in a string option. Upload this string or otherwise extract it from the browser to create an .coverage file.

3. Build the usual Js_of_ocaml target, including the instrumented code under test, then run the reporter to generate the coverage report in _coverage/index.html:

   BISECT_ENABLE=yes dune build my_tester.bc.js
   bisect-ppx-report html
   

Ocamlfind, Ocamlbuild, and OASIS

• Ocamlbuild and OASIS instructions can be found at aantron/bisect_ppx-ocamlbuild.

• With Ocamlfind, you must have your build script issue the right commands, to instrument the code under test, but not the tester:

  ocamlfind opt -package bisect_ppx -c src/source.ml
  ocamlfind opt -c test/test.ml
  ocamlfind opt -linkpkg -package bisect_ppx src/source.cmx test/test.cmx
  

  Running the tester will then produce bisect0123456789.coverage files, which you can process with bisect-ppx-report.

Sending to Coveralls

bisect-ppx-report can send reports to Coveralls and Codecov directly from Travis, CircleCI, and GitHub Actions. To do this, run

bisect-ppx-report send-to Coveralls

or

bisect-ppx-report send-to Codecov

When sending specifically from GitHub Actions to Coveralls, use

- run: bisect-ppx-report send-to Coveralls
  env:
    COVERALLS_REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    PULL_REQUEST_NUMBER: ${{ github.event.number }}

Put these commands in your CI script in the same place you would run bisect-ppx-report html locally. See bisect-ci-integration-megatest for example CI scripts and current status of these integrations.

If you'd like Bisect_ppx to support other CI and/or coverage services, please open an issue or send a pull request!

As a workaround for missing CI/coverage integrations, and for development, bisect-ppx-report can also generate a JSON report in Coveralls format, which can be uploaded to a service of your choice using a separate command. For example, to send manually from Travis to Coveralls:

bisect-ppx-report \
  coveralls coverage.json \
  --service-name travis-ci \
  --service-job-id $TRAVIS_JOB_ID
curl -L -F json_file=@./coverage.json https://coveralls.io/api/v1/jobs

For other CI services, replace --service-name and --service-job-id as follows:

Note that Coveralls-style reports are less precise than the HTML reports generated by Bisect_ppx, because Coveralls considers entire lines as visited or not visited. Bisect_ppx instead considers individual expressions. There can be many expressions on a single line, and the HTML report separately considers each expression as visited or not visited.

Controlling coverage with [@coverage off]

You can tag expressions with [@coverage off], and neither they, nor their subexpressions, will be instrumented by Bisect_ppx.

Likewise, you can tag module-level let-declarations with [@@coverage off], and they won't be instrumented.

You can also turn off instrumentation for blocks of declarations inside a module with [@@@coverage off] and [@@@coverage on].

Finally, you can exclude an entire file by putting [@@@coverage exclude_file] into its top-level module. However, whenever possible, it is recommended to exclude files by not preprocessing with Bisect_ppx to begin with.

Real-world example examined

Refer to:

• aantron/markup.ml, which produces this local report, and this report on Coveralls.

• The Dune instructions and Coveralls instructions above.

The details:

1. The project depeds on package bisect_ppx, so that Bisect_ppx is installed by opam pin --dev-repo markup and opam install .

2. There are three libraries in src/, each set to have its sources preprocessed by Bisect_ppx:

   • markup

   • markup-lwt

   • markup-lwt.unix

   Because of the --conditional flag, preprocessing is enabled only when BISECT_ENABLE=yes is set in the environment, so it is off by default.

3. A coverage build is triggered by running make coverage. This target...

   • Depends on make clean. This is a workaround until ocaml/dune#57 is solved. The problem is that doing a coverage build, after normal builds, should force all sources to be recompiled, so that they can be instrumented by the Bisect_ppx preprocessor. However, Dune doesn't know about this — it doesn't know that the behavior of the preprocessor depends on the BISECT_ENABLE environment variable.

     Indeed, the preprocessor shouldn't read this environment variable. The preprocessor should just be turned off by Dune when not building for coverage. However, Dune does not currently have the ability to conditionally turn off a preprocessor.

     In any case, to deal with this problem, the project always does a clean build when building for coverage.

   • Does a fresh build with BISECT_ENABLE=yes, causing the sources of the three libraries mentioned above to be instrumented.

   • Runs the test suite. bisect*.coverage files with coverage data are produced as a side effect.

   • Runs bisect-ppx-report to generate both the typical HTML report in _coverage/index.html, and also a textual summary in the terminal for very fast iteration.

4. make coverage is also used in Travis to submit coverage reports to Coveralls. At the end of make coverage, the bisect*.coverage files are still present, so .travis.yml runs bisect-ppx-report again to generate the Coveralls report. This follows the Coveralls instructions exactly.

   Coveralls can be configured to leave comments about changes in coverage. It is usually configured to at least add an additional check to branches and PRs — see the "3 checks passed" in the hidden Details of the linked PR.

5. During release, (preprocess (pps bisect_ppx)) is removed from all libraries that are being released. This is typically in a one-commit release branch off master, which is what ends up being tagged.

   This won't be necessary after ocaml/dune#57 is addressed.

Other topics

See advanced usage for:

• Exhaustiveness checking.

• Excluding generated files from coverage.

• Environment variables.

Bisect_ppx users

A small sample of projects using Bisect_ppx:

• Core tools

  • Lwt   (report)

  • Odoc

  • ocamlformat

  • OCaml

  • Reason

  • ctypes

• Libraries

  • Markup.ml   (report)

  • Lambda Soup   (report)

  • Trie   (report)

  • ocaml-ooxml   (report)

• Applications

  • XAPI   (1, 2, 3, 4, 5)

  • Scilla   (report)

  • Coda

  • snarky

  • comby   (report)

  • ocaml-irc-client   (report)

Contributing

Bug reports and pull requests are warmly welcome. Bisect_ppx is developed on GitHub, so please open an issue.

Bisect_ppx is developed mainly using opam. To get the latest development version, run

opam source --dev-repo --pin bisect_ppx

You will now have a bisect_ppx subdirectory to work in. Try these Makefile targets:

• make test for unit tests.

• make usage for build system integration tests, except BuckleScript.

• make -C test/bucklescript full-test for BuckleScript. This requires NPM and esy.