package irmin

Irmin public API.

Irmin is a library to design and use persistent stores with built-in snapshot, branching and reverting mechanisms. Irmin uses concepts similar to Git but it exposes them as a high level library instead of a complex command-line frontend. It features a bidirectional Git backend, fully-compatible with the usual Git tools and workflows.

Irmin is designed to use a large variety of backends. It is written in pure OCaml and does not depend on external C stubs; it is thus very portable and aims to run everywhere, from Linux to Xen unikernels.

Consult the basics and Examples of use for a quick start. See also the documentation for the unix backends.

Release %%VERSION%% - %%MAINTAINER%%

Preliminaries

Stores

An Irmin store is automatically built from a number of lower-level stores, implementing fewer operations, such as append-only and read-write stores. These low-level stores are provided by various backends.

User-Defined Contents

High-level Stores

An Irmin store is a branch-consistent store where keys are lists of steps.

An example is a Git repository where keys are filenames, i.e. list of '/'-separated strings. More complex examples are structured values, where steps might contain first-class field accessors and array offsets.

Irmin provides the following features:

• Support for fast clones, branches and merges, in a fashion very similar to Git.
• Efficient staging areas for fast, transient, in-memory operations.
• Fast synchronization primitives between remote stores, using native backend protocols (as the Git protocol) when available.

Synchronization

Examples

These examples are in the examples directory of the distribution.

Synchronization

A simple synchronization example, using the Git backend and the Sync helpers. The code clones a fresh repository if the repository does not exist locally, otherwise it performs a fetch: in this case, only the missing contents is downloaded.

open Lwt
open Irmin_unix

module S = Irmin_git.FS(Irmin.Contents.String)(Irmin.Ref.String)(Irmin.Hash.SHA1)
module Sync = Irmin.Sync(S)
let config = Irmin_git.config ~root:"/tmp/test" ()

let upstream =
  if Array.length Sys.argv = 2 then (Irmin.remote_uri Sys.argv.(1))
  else (Printf.eprintf "Usage: sync [uri]\n%!"; exit 1)

let test () =
  S.Repo.create config >>= S.master task
  >>= fun t  -> Sync.pull_exn (t "Syncing with upstream store") upstream `Update
  >>= fun () -> S.read_exn (t "get the README") ["README.md"]
  >>= fun r  -> Printf.printf "%s\n%!" r; return_unit

let () =
  Lwt_main.run (test ())

Mergeable logs

We will demonstrate the use of custom merge operators by defining mergeable debug log files. We first define a log entry as a pair of a timestamp and a message, using the combinator exposed by mirage-tc:

module Entry = struct
  include Tc.Pair (Tc.Int)(Tc.String)
  let compare (x, _) (y, _) = Pervasives.compare x y
  let time = ref 0
  let create message = incr time; !time, message
end

A log file is a list of entries (one per line), ordered by decreasing order of timestamps. The 3-way merge operator for log files concatenates and sorts the new entries and prepend them to the common ancestor's ones.

module Log: Irmin.Contents.S with type t = Entry.t list = struct
  module Path = Irmin.Path.String_list
  module S = Tc.List(Entry)
  include S

  (* Get the timestamp of the latest entry. *)
  let timestamp = function
    | [] -> 0
    | (timestamp, _ ) :: _ -> timestamp

  (* Compute the entries newer than the given timestamp. *)
  let newer_than timestamp entries =
    let rec aux acc = function
      | [] -> List.rev acc
      | (h, _) :: _ when h <= timestamp -> List.rev acc
      | h::t -> aux (h::acc) t
    in
    aux [] entries

  let merge_log _path ~old t1 t2 =
    let open Irmin.Merge.OP in
    old () >>| fun old ->
    let old = match old with None -> [] | Some o -> o in
    let ts = timestamp old in
    let t1 = newer_than ts t1 in
    let t2 = newer_than ts t2 in
    let t3 = List.sort Entry.compare (List.rev_append t1 t2) in
    ok (List.rev_append t3 old)

  let merge path = Irmin.Merge.option (module S) (merge_log path)

end

Note: The serialisation primitives provided by mirage-tc: are not very efficient in this case as they parse the file every-time. For real usage, you would write buffered versions of Log.read and Log.write.

To persist the log file on disk, we need to choose a backend. We show here how to use the on-disk Git backend on Unix.

(* Bring [Irmin_unix.task] and [Irmin_unix.Irmin_git] in scope. *)
open Irmin_unix

(* Build an Irmin store containing log files. *)
module S = Irmin_git.FS(Log)(Irmin.Ref.String)(Irmin.Hash.SHA1)

(* Set-up the local configuration of the Git repository. *)
let config = Irmin_git.config ~root:"/tmp/irmin/test" ~bare:true ()

We can now define a toy example to use our mergeable log files.

open Lwt

(* Name of the log file. *)
let file = [ "local"; "debug" ]

(* Read the entire log file. *)
let read_file t =
  S.read (t "Reading the log file") file >>= function
  | None   -> return_nil
  | Some l -> return l

(* Persist a new entry in the log. *)
let log t fmt =
  Printf.ksprintf (fun message ->
      read_file t >>= fun logs ->
      let logs = Entry.create message :: logs in
      S.update (t "Adding a new entry") file logs
    ) fmt

let () =
  Lwt_unix.run begin
    S.Repo.create config >>= S.master task
    >>= fun t  -> log t "Adding a new log entry"
    >>= fun () -> Irmin.clone_force task (t "Cloning the store") "x"
    >>= fun x  -> log x "Adding new stuff to x"
    >>= fun () -> log x "Adding more stuff to x"
    >>= fun () -> log x "More. Stuff. To x."
    >>= fun () -> log t "I can add stuff on t also"
    >>= fun () -> log t "Yes. On t!"
    >>= fun () -> Irmin.merge_exn "Merging x into t" x ~into:t
    >>= fun () -> return_unit
  end

Helpers

Backends

API to create new Irmin backends. A backend is an implementation exposing either a concrete implementation of S or a functor providing S once applied.

There are two ways to create a concrete Irmin.S implementation:

• Make creates a store where all the objects are stored in the same store, using the same internal keys format and a custom binary format based on bin_prot, with no native synchronization primitives: it is usually what is needed to quickly create a new backend.
• Make_ext creates a store with a deep embedding of each of the internal stores into separate store, with a total control over the binary format and using the native synchronization protocols when available. This is mainly used by the Git backend, but could be used for other similar backends as well in the future.