datakit

Orchestrate applications using a 9P dataflow
README

DataKit is a tool to orchestrate applications using a 9P dataflow. It
revisits the UNIX pipeline concept, with a modern twist: streams of
tree-structured data instead of raw text. DataKit allows you to define
complex build pipelines over version-controlled data, using shell
scripts interacting with the filesystem.

DataKit is currently used as the coordination
layer for HyperKit, the
hypervisor component of
Docker for Mac and Windows.



Quick Start

The easiest way to use DataKit is to start both the server and the client in a
container.

To expose a Git repository as a 9p endpoint on port 5640 on a private network,
just run:

$ docker network create datakit-net # create a private network
$ docker run -it --net datakit-net --name datakit -v <path/to/git/repo>:/data docker/datakit

Note: The --name datakit option is mandatory. It will allow the client
to connect to a known name on the private network.

You can then start a DataKit client, which will mount the 9p endpoint and
expose the database as a filesystem API:

# In an other terminal
$ docker run -it --privileged --net datakit-net docker/datakit:client
$ ls /db
branch     remotes    snapshots  trees

Note: the --privileged option is needed because the container will have
to mount the 9p endpoint into its local filesystem.

Now you can explore, edit and script /db. See the
Filesystem API
for more details.

Experimental GitHub API bindings

To start DataKit with the experimental GitHub bindings:

$ docker run -it --net datakit-net --name datakit -v <path/to/git/repo>:/data docker/datakit:github
$ docker run -it --privileged --net datakit-net docker/datakit:client
$ ls /db
branch      github.com  remotes     snapshots   trees

Building

The easiest way to build the DataKit project is to use docker,
(which is what the
start-datakit.sh script
does under the hood):

$ docker build -t datakit .
$ docker run datakit

These commands will expose the database's 9p endpoint on port 5640.

If you really want to build the project from source, you will need to install
ocaml and opam. Then write:

$ opam pin add datakit . -n -y
$ opam depext datakit -y
$ opam install alcotest datakit --deps-only -y
$ make && make test

Usage

$ datakit --help

Filesystem API

The /branch directory contains one subdirectory for each branch. Use
mkdir to create a new branch and rm to delete one.

Each branch directory contains:

  • fast-forward will do a fast-forward merge to any commit ID written
    to this file.

  • head gives the commit ID of the head of the branch when
    read (or the empty string if the branch is empty).

  • head.live is a stream which produces a list of commit IDs, one per
    line, starting with the current commit and returning new commits as the
    branch is updated. A branch with no commits is
    represented by a blank line.

  • reflog is a stream which outputs a new line each time the current
    HEAD is updated. The line gives the commit hash (or is blank if the branch
    has been deleted). Unlike head.live, reflog does not start by outputting
    the current commit and it does not skip commits.

  • ro is a live read-only view of the current contents of the head of
    the branch.

  • transactions is used to update the branch.

  • watch can be used to watch specific files or directories for changes.

Note that reading from head.live will skip directly to the latest
commit: even if you read continuously from it, you will not
necessarily see all intermediate commits.

The root also contains /snapshots, which can be used to explore any
commit in the repository, if you know its ID. The directory will
always appear empty, but attempting to access a subdirectory named by
a commit ID will work.

The /trees directory works in a similar way to /snapshots, but is
indexed by directory tree or file hashes (as read from tree.live)
rather than by commit hashes.

Transactions

Read/write transactions can be created by making a new directory for
the transaction in transactions. The newly created directory will
contain:

  • rw, a directory with the current contents of the
    transaction. Initially, this is a copy of the branch's ro
    directory. Modify this as desired.

  • msg, the commit message to use.

  • parents, the list of commit hashes of the parents, one per line.
    Initially, this is the single head commit at the time the transaction
    was created, but it can be modified to produce other effects.
    Simply appending another branch's 'head' here is equivalent to doing a Git
    merge with strategy 'ours' (which is not the same as "recursive/ours").

  • ctl, which can be used to commit the transaction (by writing commit to
    it) or to cancel it (by writing close).

  • merge, which can be used to start a merge (see below).

  • diff/ is a directory containing hidden files. diff/<commit-id>
    contains the diff between the given commit-id and the current state
    of the transaction.

For example, to create a file somefile:

~/db $ mkdir branch/master/transactions/foo
~/db $ echo somedata > branch/master/transactions/foo/rw/somefile
~/db $ echo commit > branch/master/transactions/foo/ctl

If the branch has been updated since the transaction was created then,
when you try to commit, Irmin will try to merge the changes.

If there is a conflict (two edits to the same file) then the commit
will fail. Merge errors are reported as 9p error strings. When a
commit succeeds the transaction directory is automatically removed.

Each 9p connection has its own set of transactions, and the changes in
a transaction cannot be seen by other clients until the transaction is committed.

Merging

Within a transaction, write a commit ID to the merge file to begin a merge.
The transaction directory will change slightly:

  • ours is a read-only directory, containing whatever was previously in rw

  • theirs is the commit being merged

  • base is a common ancestor (or empty, if the commits share no history)

  • rw contains irmin9p's initial attempt at a merge

  • conflicts is a list of files in rw that need to be resolved manually

  • parents has the new commit appended to it

Note that, unlike Git, irmin9p does not attempt to merge within files.
It simply replaces files with conflicting changes with a message noting
the conflict.

For each file in conflicts you should resolve the problem by either deleting
the file or doing your own three-way merge using ours, theirs and base.
When a file has been edited, it is removed from conflicts. You cannot commit
the transaction while conflicts is non-empty.

You may merge several commits in a single transaction, if desired.
However, doing multiple non-trivial merges at once will make viewing
the resulting merge commit difficult with most tools.

Snapshots

A snapshot for a given commit can be opened by accessing the
directory /snapshots/COMMIT_ID, which is created on demand.

~/db $ cd snapshots/4b6557542ec9cc578d5fe09b664110ba3b68e2c2
~/d/s/4b6557542ec9cc578d5fe09b664110ba3b68e2c2 $ ls
hash  ro/
~/d/s/4b6557542ec9cc578d5fe09b664110ba3b68e2c2 $ cat hash
4b6557542ec9cc578d5fe09b664110ba3b68e2c2
~/d/s/4b6557542ec9cc578d5fe09b664110ba3b68e2c2 $ ls ro
somefile
~/d/s/4b6557542ec9cc578d5fe09b664110ba3b68e2c2 $

The contents of a snapshot directory are:

  • ro is the read-only snapshot, which will never change.

  • hash contains the commit hash.

  • msg contains the commit message.

  • parents contains the hashes of the parent commits, one per line.

  • diff/ is a directory containing hidden files. diff/<commit-id>
    contains the diff between the given commit-id and this snapshot.

Watches

To watch for changes affecting a specific file or subdirectory in a
branch, use the branch's watch directory.

Each directory under watch contains a tree.live file that outputs
the current hash of the object that directory watches. The top
watch/tree.live file tracks changes to all files and directories.
To watch for changes under src/ui, read the file
watch/src.node/ui.node/tree.live. That is, add .node to each path
component to get a directory for that node.

Reading from a tree.live file outputs first one line for the current
state of the path. This can be:

  • A blank line, if the path does not currently exist.

  • D-HASH if the path is a directory (the hash is the tree hash).

  • F-HASH if the path is a file (the hash is the hash of the blob).

  • X-HASH if the path is an executable file (the hash is the hash of the blob).

  • L-HASH if the path is a symlink (the hash is the hash of the blob containing the target string).

When the branch head changes so that the path has a different output,
a new line will be produced, in the same format. As with head.live,
watching for changes is triggered by reading on the open file, so if
several changes occur between reads then you will only see the latest
one.

Note: Listing a watch directory shows .node subdirectories for paths
that currently exist. However, these are just suggestions; you can
watch any path, whether it currently exists or not.

Diff

To see the difference between a given commit ID and the head of a
branch, use the branch's diff directory.

Each file under the diff directory contains a line per change of the form:

  • + <path> means that the file path has been added between commit-id and
    HEAD;

  • - <path> means that the file path has been removed between commid-id and
    HEAD;

  • * <path> means that the file path has been modified betweeen commit-id
    and HEAD.

For instance:

~/db $ cat branches/master/diff/6b2e00a0be59c0335568dd9415a7d93640e7099c
+ foo
* bar

Means that foo have been added and bar modified in the master branch since
the commit 6b2e00a0be59c0335568dd9415a7d93640e7099c took place.

Note: this also works when you are inside a transaction.

Fetch

To fetch from a remote repository, use the /remotes root directory.
This directory is not persisted so will disappear across reboots.

Each directory under /remotes/<name> corresponds to the configuration
of a remote server called <name>. Create a new directory (with mkdir)
to add a new configuration. Every configuration folder contains:

  • A writable file: url, which contains the remote url.

  • A control file: fetch, which is used to fetch branches from the
    remote server.

  • A read-only stream file: head which contains the last known
    commit ID of the remote. On every fetch, a new line is added
    with the commit ID of the remote branch.

To fetch https://github.com/docker/datakit's master branch using the
git protocol:

~/db $ cd remotes
~/db/remotes $ mkdir origin
~/db/remotes $ echo git://github.com/docker/datakit > origin/url
~/db/remotes $ echo master > origin/fetch
~/db/remotes $ cat origin/head
4b6557542ec9cc578d5fe09b664110ba3b68e2c2

GitHub PRs

There is basic support for interacting with GitHub PRs.

~/db $ ls github.com/docker/datakit
41  42
~/db $ cat github.com/docker/datakit/pr/41/status/default/state
pending
~/db $ echo success > github.com/docker/datakit/pr/41/status/default/state

This first queries the status of the pull request on the GitHub interface,
then updates the default status to success.

To create a new status and set its description, url and status:

~/db $ PR=github.com/docker/datakit/pr/41
~/db $ mkdir $PR/status/test
~/db $ echo "My status" > $PR/status/test/descr
~/db $ echo "http://example.com" > $PR/status/test/url
~/db $ echo success > $PR/status/test/state

To read the last GitHub events related to a repository:

~/db $ cat github.com/docker/datakit/events

This is a non-blocking read, and will produce a file where every line is a new
event.

How do I...

Create a new branch
mkdir branch/foo
Fork an existing branch
cd branch
mkdir new-branch
cp old-branch/head new-branch/fast-forward
Rename a branch
mv branch/old-name branch/new-name
Delete a branch
rmdir branch/foo
Merge a branch
cd branch/master/transactions
mkdir my-merge
cd my-merge
cat ../../../feature/head > merge
cat conflicts
meld --auto-merge ours base theirs --output rw
echo commit > ctl

Language bindings

  • Go bindings are in the api/go directory.

  • OCaml bindings are in the api/ocaml directory. See examples/ocaml-client for an example.

Licensing

DataKit is licensed under the Apache License, Version 2.0. See
LICENSE for the full
license text.

Install
Published
06 Dec 2016
Sources
datakit-0.8.0.tbz
md5=eee43f96d223465e4759015aba3ffa00
Dependencies
datakit-server
>= "0.8.0" & < "0.9.0"
irmin-watcher
>= "0.2.0"
mtime
< "1.0.0"
asl
>= "0.10"
logs
>= "0.5.0"
hvsock
>= "0.11.1" & < "0.14.0"
named-pipe
>= "0.4.0"
mirage-flow
< "2.0.0"
cstruct
>= "2.2"
camlzip
>= "1.06"
irmin
= "0.12.0"
mirage-types
< "3.0.0"
git
>= "1.9.3"
cmdliner
< "1.0.0"
topkg
build
ocaml
>= "4.02.0"
Reverse Dependencies