package irmin

Irmin public API.

Irmin is a library to design and use persistent stores with built-in snapshot, branching and reverting mechanisms. Irmin uses concepts similar to Git but it exposes them as a high level library instead of a complex command-line frontend. It features a bidirectional Git backend, where an application can read and persist its state using the Git format, fully-compatible with the usual Git tools and workflows.

Irmin is designed to use a large variety of backends. It is written in pure OCaml and does not depend on external C stubs; it is thus very portable and aims to run everywhere, from Linux to browser and MirageOS unikernels.

Consult the basics and Examples of use for a quick start. See also the documentation for the unix backends.

Release 1.1.0 - %%HOMEPAGE%%

Preliminaries

Stores

An Irmin store is automatically built from a number of lower-level stores, implementing fewer operations, such as append-only and read-write stores. These low-level stores are provided by various backends.

User-Defined Contents

High-level Stores

An Irmin store is a branch-consistent store where keys are lists of steps.

An example is a Git repository where keys are filenames, i.e. list of '/'-separated strings. More complex examples are structured values, where steps might contain first-class field accessors and array offsets.

Irmin provides the following features:

• Support for fast clones, branches and merges, in a fashion very similar to Git.
• Efficient taging areas for fast, transient, in-memory operations.
• Fast synchronization primitives between remote stores, using native backend protocols (as the Git protocol) when available.

Synchronization

Examples

These examples are in the examples directory of the distribution.

Synchronization

A simple synchronization example, using the Git backend and the Sync helpers. The code clones a fresh repository if the repository does not exist locally, otherwise it performs a fetch: in this case, only the missing contents is downloaded.

open Lwt.Infix

module S = Irmin_unix.Git.FS.KV(Irmin.Contents.String)
module Sync = Irmin.Sync(S)
let config = Irmin_git.config ~root:"/tmp/test" ()

let upstream =
  if Array.length Sys.argv = 2 then (Irmin.remote_uri Sys.argv.(1))
  else (Printf.eprintf "Usage: sync [uri]\n%!"; exit 1)

let test () =
  S.Repo.v config >>= S.master
  >>= fun t  -> Sync.pull_exn t upstream `Set
  >>= fun () -> S.get t ["README.md"]
  >|= fun r  -> Printf.printf "%s\n%!" r

let () = Lwt_main.run (test ())

Mergeable logs

We will demonstrate the use of custom merge operators by defining mergeable debug log files. We first define a log entry as a pair of a timestamp and a message, using the combinator exposed by mirage-tc:

module Entry = sig
  include Irmin.Contents.Conv
  val v: string -> t
  val compare: t -> t -> int
  val timestamp: t -> int
end = struct

  type t = { timestamp: int; message : string; }

  let compare x y = compare x.timestamp y.timestamp

  let v message =
    incr time;
    { timestamp = !time; message }

  let t =
    let open Irmin.Type in
    record "entry" (fun timestamp message -> { timestamp; message })
    |+ field "timestamp" int    (fun t -> t.timestamp)
    |+ field "message"   string (fun t -> t.message)
    |> sealr

  let timestamp t = t.timestamp

  let pp ppf { timestamp; message } =
    Fmt.pf ppf  "%04d: %s\n" timestamp message

  let of_string str =
    match String.cut ~sep:": " str with
    | None -> Error (`Msg ("invalid entry: " ^ str))
    | Some (x, message) ->
      try Ok { timestamp = int_of_string x; message }
      with Failure e -> Error (`Msg e)
end

A log file is a list of entries (one per line), ordered by decreasing order of timestamps. The 3-way merge operator for log files concatenates and sorts the new entries and prepend them to the common ancestor's ones.

(* A log file *)
module Log: sig
  include Irmin.Contents.S
  val add: t -> Entry.t -> t
  val empty: t
end = struct

  type t = Entry.t list
  let t = Irmin.Type.(list Entry.t)

  let empty = []

  let pp ppf l = List.iter (Fmt.pf ppf "%a\n" Entry.pp ) (List.rev l)

  let of_string str =
    let lines = String.cuts ~sep:"\n" str in
    try
      List.fold_left (fun acc l ->
          match Entry.of_string l with
          | Ok x           -> x :: acc
          | Error (`Msg e) -> failwith e
        ) [] lines
      |> fun l -> Ok l
    with Failure e ->
      Error (`Msg e)

  let timestamp = function
    | [] -> 0
    | e :: _ -> Entry.timestamp e

  let newer_than timestamp file =
    let rec aux acc = function
      | [] -> List.rev acc
      | h:: _ when Entry.timestamp h <= timestamp -> List.rev acc
      | h::t -> aux (h::acc) t
    in
    aux [] file

  let merge ~old t1 t2 =
    let open Irmin.Merge.Infix in
    old () >>=* fun old ->
    let old = match old with None -> [] | Some o -> o in
    let ts = timestamp old in
    let t1 = newer_than ts t1 in
    let t2 = newer_than ts t2 in
    let t3 = List.sort Entry.compare (List.rev_append t1 t2) in
    Irmin.Merge.ok (List.rev_append t3 old)

  let merge = Irmin.Merge.(option (v t merge))

  let add t e = e :: t

end 

Note: The serialisation primitives used in that example are not very efficient in this case as they parse the file every-time. For real usage, you would write buffered versions of Log.pp and Log.of_string.

To persist the log file on disk, we need to choose a backend. We show here how to use the on-disk Git backend on Unix.

(* Build an Irmin store containing log files. *)
module S = Irmin_unix.Git.FS.KV(Log)

(* Set-up the local configuration of the Git repository. *)
let config = Irmin_git.config ~root:"/tmp/irmin/test" ~bare:true ()

(* Set-up the commit info function *)
let info fmt = Irmin_unix.info ~author:"logger" fmt

We can now define a toy example to use our mergeable log files.

open Lwt.Infix

(* Name of the log file. *)
let file = [ "local"; "debug" ]

(* Read the entire log file. *)
let read_file t =
  S.find t file >|= function
  | None   -> []
  | Some l -> l

(* Persist a new entry in the log. *)
let log t fmt =
  Fmt.kstrf (fun message ->
      read_file t >>= fun logs ->
      let logs = Log.add logs (Entry.v message) in
      S.set t (info "Adding a new entry") file logs
    ) fmt

let () =
  Lwt_main.run begin
    S.Repo.v config >>= S.master
    >>= fun t  -> log t "Adding a new log entry"
    >>= fun () -> Irmin.clone_force ~src:t ~dst:"x"
    >>= fun x  -> log x "Adding new stuff to x"
    >>= fun () -> log x "Adding more stuff to x"
    >>= fun () -> log x "More. Stuff. To x."
    >>= fun () -> log t "I can add stuff on t also"
    >>= fun () -> log t "Yes. On t!"
    >>= fun () -> S.merge (info "Merging x into t") x ~into:t
    >|= function Ok () -> () | Errror _ -> failwith "merge conflict!"
  end

Helpers

Backends

API to create new Irmin backends. A backend is an implementation exposing either a concrete implementation of S or a functor providing S once applied.

There are two ways to create a concrete Irmin.S implementation:

• Make creates a store where all the objects are stored in the same store, using the same internal keys format and a custom binary format based on bin_prot, with no native synchronization primitives: it is usually what is needed to quickly create a new backend.
• Make_ext creates a store with a deep embedding of each of the internal stores into separate store, with a total control over the binary format and using the native synchronization protocols when available.