package eio-chaos

Build not yet checked
  1. Overview
  2. Docs
Chaos tests of Eio programs on simulated devices that fail together

Install

dune-project
 Dependency

Authors

Maintainers

Sources

eio-chaos-0.1.0.tbz
sha256=f383da15add3f6311b0725d15bd9144ab3c674b049c20364a760cc4165e307d3
sha512=8e348df4c72b86010c229b9f04f28ab08d752ccedba1a58d143e8768414522a87729350cb273ec3e4bf739013258bf6fc2a751f997411b1b320e2fb007e84b3d

doc/README.html

eio-chaos

Chaos tests of Eio programs, on simulated devices.

An Eio program receives the outside world as values, such as env#fs and env#net. eio-chaos builds that env from simulated devices and runs the program on it many times. In each run a device fails as a real one can, for instance with a full disk or a connection reset. After each fault, the program starts again on what the fault left, as a machine restarts a service, and must still work.

The devices come from other packages. eio-fs gives a disk and eio-net a network. The examples below use a toy DNS resolver, whose source is also an example of a device; it is not installed.

Installation

opam install eio-chaos

Testing a program

Eio_chaos.run devices main runs main with an env built from devices. main reads env as it does under Eio_main.run, so the program does not change. This one looks up its database and gives up when the lookup fails:

# let main (env : Eio_chaos.env) =
    match Eio.Net.getaddrinfo_stream (Eio.Stdenv.net env) "db" with
    | _ -> ()
    | exception Eio.Io _ -> failwith "no database";;
val main : Eio_chaos.env -> unit = <fun>
# match Eio_chaos.run [ Eio_chaos_toy.resolver () ] main with
  | () -> print_endline "passed"
  | exception Failure report -> print_endline report;;
Eio_chaos.run: the program failed under a fault
  faults:   #1 lookup db times out
  error:    Failure("no database")
  trace:
    #1 lookup db  <- times out
  replay:   Eio_chaos.run ~replay:"dns/355f1c0e/lookup:1.timeout" devices app
- : unit = ()

The report says which fault was injected, what the program did, and gives one line that replays that case alone. Left uncaught in a program that uses console, it prints in colour on a terminal, and as the plain text above anywhere else:

The report on a terminal, its title in red and the faulted lookup highlighted

A run meets at most two faults, so a program that tries three times passes. The pause between tries takes no time, as the run's clock jumps to the next timer:

# let rec lookup (env : Eio_chaos.env) n =
    match Eio.Net.getaddrinfo_stream env#net "db" with
    | addrs -> addrs
    | exception Eio.Io _ when n > 0 ->
        Eio.Time.sleep env#clock 1.;
        lookup env (n - 1);;
val lookup : Eio_chaos.env -> int -> Eio.Net.Sockaddr.stream list = <fun>
# Eio_chaos.run [ Eio_chaos_toy.resolver () ] (fun env -> ignore (lookup env 2));;
- : unit = ()

A program that uses several devices is run on all of them, as Eio_chaos.run [ Eio_fs_chaos.disk (); Eio_net_chaos.network () ] main, and its runs fail the disk, the network, or both.

Faults and restarts

The first run has no fault, and records the operations the program makes on each device. Each later run starts from the same state and injects one or two faults at those operations, or cuts the power. Each device proposes only the faults the real one can produce at that operation, such as an error, a read that stops half-way or a disk that fills up. After each run, the program restarts with no fault on what the devices kept.

The check fails if a run hangs, if the program raises an exception the device cannot explain, if a device lost something the program had promised to keep, or if the restart fails. An Eio.Io error that escapes a faulty run is not a failure in itself, since the device did fail: the restart decides.

Everything in env is simulated, including the clocks and the random seed, so a check is deterministic and its replay line always reproduces the case. A direct call to Unix or open_in escapes the simulation; hermetic makes the compiler reject such calls.

A pass means that no fault tried broke the program, not that none can. The faults fall only on the operations of the first run, at most two at a time, and a long program gets a sample of its pairs rather than all of them. eio_chaos.mli says how the faults are chosen, how the run budget is spent, and what env holds.

Writing a device

A device is a module of type Eio_chaos.DEVICE, given to Eio_chaos.Device. At each operation, the device asks Eio_chaos.decide whether it fails, listing the faults the kernel allows there, and acts out the answer. In production it gets Eio_chaos.none, which never fails anything, so the same device serves both. The rest of the module says how the device starts, what a power cut leaves, and how it restarts. The toy resolver is a whole device in about 60 lines.

FoundationDB, TigerBeetle, turmoil and madsim run disk and network in one deterministic simulation with one source of choices; Eio_chaos.decide is that source here. Enumerating the states a power cut leaves follows ALICE and CrashMonkey.

Licence

ISC, see LICENSE.md.