The first beta release of Coq 8.10 is available for testing.

Coq version 8.10 contains two major new features: support for a native machine integer type and a new sort SProp, a definitionally proof irrelevant universe. It is also the result of refinements and stabilization of previous features, deprecations or removals of deprecated features, cleanups of the internals of the system and API, and many documentation improvements. This release includes many user-visible changes, including deprecat…

The first beta release of Coq 8.10 is available for testing.

Coq version 8.10 contains two major new features: support for a native machine integer type and a new sort SProp, a definitionally proof irrelevant universe. It is also the result of refinements and stabilization of previous features, deprecations or removals of deprecated features, cleanups of the internals of the system and API, and many documentation improvements. This release includes many user-visible changes, including deprecations and new features. A more detailed list of changes appears in the reference manual.

Feedback and bug reports are extremely welcome.

Over the past few months, we have been heavily engaged in release engineering the Irmin 2.0 release, which covers multiple years of work on all of its constituent elements. We first began Irmin in late 2013 to act as a Git-like distributed and branchable storage substrate that would let us escape the perils of POSIX filesystems.

The Irmin libraries provide snapshotting, branching and merging operations over storage and can communicate via Git both on-disk and remotely. Irmin today therefore cons…

Over the past few months, we have been heavily engaged in release engineering the Irmin 2.0 release, which covers multiple years of work on all of its constituent elements. We first began Irmin in late 2013 to act as a Git-like distributed and branchable storage substrate that would let us escape the perils of POSIX filesystems.

The Irmin libraries provide snapshotting, branching and merging operations over storage and can communicate via Git both on-disk and remotely. Irmin today therefore consists of many discrete OCaml libraries that compose together to form a set of mergeable data structures that can be used in MirageOS unikernels and normal OCaml daemons such as Tezos.

In this blog post, we wanted to explain some of the release engineering ongoing, and to highlight some areas where we could use help from the community to test out pieces (and hopefully find your own uses in your own infrastructure for it). The overall effort is tracked in mirage/irmin#658, so feel free to comment on there as well.

ocaml-git

Irmin is parameterised over the exact communication mechanisms it uses between nodes, both as an on-disk format and also the remoting protocol. The most important concrete implementation is Git, which has turned into the world’s most popular version control system. In order to seamlessly integrate with Irmin, we embarked on an effort to build a complete re-implementation of Git from scratch in pure OCaml.

You can read details of the git 2.0 release on this blog, but from a release engineering perspective we have steadily been fixing corner cases in this implementation. The development ocaml-git trees feature fixes to https+git, for listing remotes, supporting authenticated URIs and more.

These fixes are possible because users tried end-to-end usecases that found these corner cases, so we’d really like to see more. For example, our friends at Robur have submitted fixes from their integration of it into their upcoming CalDAV engine. The Mirage canopy blog engine can now also push/pull reliably from pure MirageOS unikernels between nodes, which is a huge step.

If you get a chance to try ocaml-git in your infrastructure, please let us know how you get along as we prepare a release of the git libraries with all these fixes (which will be used in Irmin 2.0).

Wodan

Irmin’s storage layer is also well abstracted, so backends other than a Unix filesystem or Git are supported. Irmin can run in highly diverse and OS-free environments, and so we began engineering the Wodan filesystem as a domain-specific filesystem designed for MirageOS, Irmin and modern flash drives. See the OCaml Workshop 2017 abstract on it for more design rationale)

As part of the Irmin 2.0 release, Wodan is also being prepared for a release, and you can find Irmin 2.0 support in the source. If you’d like a standalone block-device based persistence environment for Irmin, please try this out. This is the preferred backend for using Irmin storage in a unikernel.

Tezos and irmin-pack

Another big user of Irmin is the Tezos blockchain, and we have been optimising the persistent space usage of Irmin as their network grows. Because Tezos doesn’t require full Git format support, we created a hybrid backend that grabs the best bits of Git (e.g. the packfile mechanism) and engineered a domain-specific backend tailored for Tezos usage. Crucially, because of the way Irmin is split into clean libraries and OCaml modules, we only had to modify a small part of the codebase and could also re-use elements of the Git 2.0 engineering effort we described above.

The irmin-pack backend is currently being reviewed and integrated ahead of Irmin 2.0 to provide a significant improvement in disk usage -- more information to come soon. There is a corresponding Tezos branch using the Irmin 2.0 code that will be integrated downstream in Tezos once we complete the Irmin 2.0 tests.

Irmin-GraphQL and “browser Irmin”

Another new area of huge interest to us is GraphQL in order to provide frontends a rich query language for Irmin hosted applications. Irmin 2.0 includes a builtin GraphQL server so you can manipulate your Git repo via GraphQL.

If you are interested in (for example) compiling elements of Irmin to JavaScript or wasm, for usage in frontends, then the Irmin 2.0 release makes it significantly easier to support this architecture. We’ve already seen some exploratory efforts report issues when doing this, and we’ve had it working ourselves in Irmin 1.0 Cuekeeper so we are excited by the potential power of applications built using this model. If you have ideas/questions, please get in touch on the issue tracker with your usecase.

This post is just the precursor to the Irmin 2.0 release, so expect to hear more about it in the coming weeks and months. This is primarily a call for help from early adopters interested in helping the project out. All of our code is liberally licensed open source, and so this is a good time to tie together end-to-end usecases and help ensure we don’t make any decisions in Irmin 2.0 that go counter to some product you’d like to build. That’s only possible with your feedback, so either get in touch via the issue tracker, on discuss.ocaml.org via the mirageos tag, or just email us.

A huge thank you to all our commercial customers, end users and open source developers who have contributed their time, expertise and financial support to help us achieve our goal of delivering a modern storage stack in the spirit of Git. We look forward to getting Irmin 2.0 into your hands very soon!

These last few months, I spent some time writing new OCaml PPX rewriters or contributing to existing ones. It's a really fun experience. Toying around with the AST taught me a lot about a language I thought I knew really well. Turns out I actually had no idea what I was doing all these years.

All jokes aside, I was surprised that the most helpful tricks I learned while writing PPX rewriters weren't properly documented. There already exist a few very good introduction articles on the subj…

These last few months, I spent some time writing new OCaml PPX rewriters or contributing to existing ones. It's a really fun experience. Toying around with the AST taught me a lot about a language I thought I knew really well. Turns out I actually had no idea what I was doing all these years.

All jokes aside, I was surprised that the most helpful tricks I learned while writing PPX rewriters weren't properly documented. There already exist a few very good introduction articles on the subject, like that 2014's article from Whitequark, this more recent one from Rudi Grinberg or even this last one from Victor Darvariu I only discovered after I actually started writing my own. I still felt like they were slighlty outdated or weren't answering all the questions I had when I started playing with PPX and writing my first rewriters.

I decided to share my PPX adventures in the hope that it can help others familiarize with this bit of the OCaml ecosystem and eventually write their first rewriters. The scope of this article is not to cover every single detail about the PPX internals but just to give a gentle introduction to beginners to help them get settled. That also means I might omit things that I don't think are worth mentioning or that might confuse the targetted audience but feel free to comment if you believe this article missed an important point.

It's worth mentioning that a lot of the nice tricks mentionned in these lines were given to me by a wonderful human being called Étienne Millon, thanks Étienne!

What is a PPX?

PPX rewriters or PPX-es are preprocessors that are applied to your code before passing it on to the compiler. They don't operate on your code directly but on the Abstract Syntax Tree or AST resulting from its parsing. That means that they can only be applied to syntactically correct OCaml code. You can think of them as functions that take an AST and return a new AST.

That means that in theory you can do a lot of things with a PPX, including pretty bad and cryptic things. You could for example replace every instance of true by false, swap the branches of any if-then-else or randomize the order of every pattern-matching case. Obviously that's not the kind of behaviour that we want as it would make it impossible to understand the code since it would be so far from the actual AST the compiler would get. In practice PPX-es have a well defined scope and only transform parts you explicitly annotated.

Understanding the OCaml AST

First things first, what is an AST. An AST is an abstract representation of your code. As the name suggests it has a tree-like structure where the root describes your entire file. It has children for each bits such as a function declaration or a type definition, each of them having their own children, for example for the function name, its argument and its body and that goes on until you reach a leaf such as a literal 1, "abc" or a variable for instance. In the case of OCaml it's a set of recursive types allowing us to represent OCaml code as an OCaml value. This value is what the parser passes to the compiler so it can type check and compile it to native or byte code. Those types are defined in OCaml's Parsetree module. The entry points there are the structure type which describes the content of an .ml file and the signature type which describes the content of an .mli file.

As mentionned above, a PPX can be seen as a function that transforms an AST. Writing a PPX thus requires you to understand the AST, both to interpret the one you'll get as input and to produce the right one as output. This is probably the trickiest part as unless you've already worked on the OCaml compiler or written a PPX rewriter, that will probably be the first time you two meet. Chances are also high that'll be a pretty bad first date and you will need some to time to get to know each others.

The Parsetree module documentation, is a good place to start. The above mentioned structure and signature types are at the root of the AST but some other useful types to look at at first are:

• expression which describes anything in OCaml that evaluates to a value, the right hand side of a let binding for instance.
• pattern which is what you use to deconstruct an OCaml value, the left hand side of a let binding or a pattern-matching case for example.
• core_type which describes type expressions ie what you would find on the right hand side of a value description in a .mli, ie val f : <what_goes_there>.
• structure_item and signature_item which describe the top level AST nodes you can find in a structure or signature such as type definitions, value or module declarations.

Thing is, it's a bit a rough and there's no detailed explanation about how a specific bit of code is represented, just type definitions. Most of the time, the type, field, and variant names are self-explanatory but it can get harder with some of the more advanced language features. It turns out there are plenty of comments that are really helpful in the actual parsetree.mli file and that aren't part of the generated documentation. You can find them on github but I personally prefer to have it opened in a VIM tab when I work on a PPX so I usually open ~/.opam/<current_working_switch>/lib/ocaml/compiler-libs/parsetree.mli.

This works well while exploring but you might also want a more straightforward approach to discovering what the AST representation is for some specific OCaml code. The ppx_tools opam package comes with a dumpast binary that pretty prints the AST for any given piece of valid OCaml code. You can install it using opam:

$opam install ppx_tools

and then run it using ocamlfind:

$ocamlfind ppx_tools/dumpast some_file.ml

You can use it on .ml and .mli files or to quickly get the AST for an expression with the -e option:

$ocamlfind ppx_tools/dumpast -e "1 + 1"

Similarly, you can use the -t or -p options to respectively pretty prints ASTs from type expressions or patterns.

Using dumpast to get both the ASTs of a piece of code using your future PPX and the resulting preprocessed code is a good way to start and will help you figure out what are the steps required to get there.

Note that you can use the compiler or utop have a similar feature with the -dparsetree flag. Running ocamlc/ocamlopt -dparsetree file.ml will pretty print the AST of the given file while running utop -dparsetree will pretty print the AST of the evaluated code alongside it's evaluation. I tend to prefer the pretty printed AST from dumpast but any of these tools will prove helpful in understanding the AST representation of a given piece of OCaml code.

Language extensions interpreted by PPX-es

OCaml 4.02 introduced syntax extensions meant to be used by external tools such as PPX-es. Knowing their syntax and meaning is important to understand how most of the existing rewriters work because they usually look for those language extenstions in the AST to know which part of it they need to modify.

The two language extensions we're interested in here are extension nodes and attributes. They are defined in detail in the OCaml manual (see the attributes and extension nodes sections) but I'll try to give a good summary here.

Extension nodes are used in place of expressions, module expressions, patterns, type expressions or module type expressions. Their syntax is [%extension_name payload]. We'll come back to the payload part a little later. You can also find extension nodes at the top level of modules or module signatures with the syntax [%%extension_name payload]. Hopefully the following cheatsheet can help you remember the basics of how and where you can use them:

type t =
  { a : int
  ; b : [%ext pl]
  }

let x =
  match 1 with
  | 0 -> [%ext pl]
  | [%ext pl] -> true

[%%ext pl]

Because extension nodes stand where regular AST nodes should, the compiler won't accept them and will give you an Uninterpreted extension error. Extension nodes have to be expanded by a PPX for your code to compile.

Attributes are slightly different although their syntax is very close from extensions. Attributes are attached to existing AST nodes instead of replacing them. That means that they don't necessarily need to be transformed and the compiler will ignore unknown attributes by default. They can come with a payload just like extensions and use @ instead of %. The number of @ preceding the attribute name specifies which kind of node they are attached to:

let a = 12 [@attr pl]

let b = "some string" [@@attr pl]

[@@@attr pl]

In the first example, the attribute is attached to the expression 12 while in the second example it is attached to the whole let b = "some string" value binding. The third one is of a slightly different nature as it is a floating attribute. It's not attached to anything per-se and just ends up in the AST as a structure item. Because there is a wide variety of nodes to which you can attach attributes, I won't go too far into details here but a good rule of thumb is that you use @@ attributes when you want them attached to structure or signature items, for anything deeper within the AST structure such as patterns, expressions or core types, use the single @ syntax. Looking at the Parsetree documentation can help you figure out where you can find attributes.

Now let's talk about those payloads I mentioned earlier. You can think of them as "arguments" to the extension points and attributes. You can pass different kinds of arguments and the syntax varies for each of them:

let a = [%ext expr_or_str_item] 
let b = [%ext: type_expr_or_sig_item]
let c = [%ext? pattern]

As suggested in the examples, you can pass expressions or structure items using a space character, type expressions or signature items (anything you'd find at the top level of a module signature) using a : or a pattern using a ?.

Attributes' payload use the same syntax:

let a = 'a' [@attr expr_or_str_item]
let b = 'b' [@attr: type_expr_or_sig_item]
let a = 'a' [@attr? pattern]

Some PPX-es rely on other language extensions such as the suffix character you can attach to int and float litterals (10z could be used by a PPX to turn it into Z.of_string "10" for instance) or quoted strings with a specific identifier ({ppx_name|some quoted string|ppx_name} can be used if you want your PPX to operate on arbitrary strings and not only syntactically correct OCaml) but attributes and extensions are the most commonly used ones.

Attributes and extension points can be expressed using an infix syntax. The attribute version is barely used but some forms of the infix syntax for extension points are used by popular PPX-es and it is likely you will encounter some of the following:

let infix_let_extension =
  let%ext x = 2 in
  ...

let infix_match_extension =
  match%ext y with ...

let infix_try_extension =
  try%ext f z with _ -> ...

which are syntactic sugar for:

let infix_let_extension =
  [%ext let x = 2 in ...]

let infix_match_extension =
  [%ext match y with ...]

let infix_try_extension =
  [%ext try f z with _ -> ...]

A good example of a PPX making heavy use of these if lwt_ppx. The OCaml manual also contains more examples of the infix syntax in the Attributes and Extension points sections mentionned above.

The two main kind of PPX-es

There is a wide variety of PPX rewriters but the ones you'll probably see the most are Extensions and Derivers.

Extensions

Extensions will rewrite tagged parts of the AST, usually extension nodes of the form [%<extension_name> payload]. They will replace them with a different AST node of the same nature ie if the extension point was located where an expression should be, the rewriter will produce an expression. Good examples of extensions are:

• ppx_getenv2 which replaces [%getenv SOME_VAR] with the value of the environment variable SOME_VAR at compile time.
• ppx_yojson which allows you to write Yojson values using OCaml syntax to mimic actual json. For instance you'd use [%yojson {a = None; b = 1}] to represent {"a": null, "b": 1} instead of the Yojson's notation: Assoc [("a", Null); ("b", Int 1)].

Derivers

Derivers or deriving plugins will "insert" new nodes derived from type definitions annotated with a [@@deriving <deriver_name>] attribute. They have various applications but are particularly useful to derive functions that are tedious and error prone to write by hand such as comparison functions, pretty printers or serializers. It's really convenient as you don't have to update those functions every time you update your type definitions. They were inspired by Haskell Type classes. Good examples of derivers are:

• ppx_deriving itself comes with a bunch of deriving plugins such as eq, ord or show which respectively derives, as you might have guessed, equality, comparison and pretty-printing functions.
• ppx_deriving_yojson which derives JSON serializers and deserializers.
• ppx_sexp_conv which derives s-expressions converters.

Derivers often let you attach attributes to specify how some parts of the AST should be handled. For example when using ppx_deriving_yojson you can use [@default some_val] to make a field of an object optional:

type t =
  { a: int
  ; b: string [@default ""]
  }
[@@deriving of_yojson]

will derive a deserializer that will convert the JSON value {"a": 1} to the OCaml {a = 1; b = ""}

How to write a PPX using ppxlib

Historically there was a few libraries used by PPX rewriter authors to write their PPX-es, including ppx_tools and ppx_deriving but as the eco-system evolved, ppxlib emerged and is now the most up-to-date and maintained library to write and handle PPX-es. It wraps the features of those libraries in a single one. I encourage you to use ppxlib to write new PPX-es as it is also easier to make various rewriters work together if they are all registered through ppxlib and the PPX ecosystem would gain from being unified around a single PPX library and driver.

It is also a great library and has some really powerful features to help you write your extensions and derivers.

Writing an extension

The entry point of ppxlib for extensions is Ppxlib.Extension.declare. You have to use that function to build an Extension.t, from which you can then build a Context_free.Rule.t before registering your transformation so it's actually applied.

The typical my_ppx_extension.ml will look like:

open Ppxlib

let extension =
  Extension.declare
    "my_extension"
    some_context
    some_pattern
    expand_function

let rule = Context_free.Rule.extension extension

let () =
  Driver.register_transformation ~rules:[rule] "my_transformation"

To compile it as PPX rewriter you'll need to put the following in your dune file:

(library
 (public_name my_ppx)
 (kind ppx_rewriter)
 (libraries ppxlib))

Now let's go back a little and look at the important part:

let extension =
  Extension.declare
    "my_extension"
    some_context
    some_pattern
    expand_function

Here "my_extension" is the name of your extension and that define how you're going to invoke it in your extension point. In other words, to use this extension in our code we'll use a [%my_extension ...] extension point.

some_context is a Ppxlib.Extension.Context.t and describes where this extension can be found in the AST, ie can you use [%my_extension ...] as an expression, a pattern, a core type. The Ppxlib.Extension.Context module defines a constant for each possible extension context which you can pass as some_context. This obviously means that it also describes the type of AST node to which it must be converted and this property is actually enforced by the some_pattern argument. But we'll come back to that later.

Finally expand_function is our actual extension implementation, which basically takes the payload, a loc argument which contains the location of the expanded extension point, a path argument which is the fully qualified path to the expanded node (eg. "file.ml.A.B") and returns the generated code to replace the extension with.

Ast_pattern

Now let's get back to that some_pattern argument.

This is one of the trickiest parts of ppxlib to understand but it's also one its most powerful features. The type for Ast_pattern is defined as ('a, 'b, 'c) t where 'a is the type of AST nodes that are matched, 'b is the type of the values you're extracting from the node as a function type and 'c is the return type of that last function. This sounded really confusing to me at first and I'm guessing it might do to some of you too so let's give it a bit of context.

Let's look at the type of Extension.declare:

val declare :
  string ->
  'context Context.t ->
  (payload, 'a, 'context) Ast_pattern.t ->
  (loc:Location.t -> path:string -> 'a) ->
  t

Here, the expected pattern first type parameter is payload which means we want a pattern that matches payload AST nodes. That makes perfect sense since it is used to describe what your extension's payload should look like and what to do with it. The last type parameter is 'context which again seems logical. As I mentioned earlier our expand_function should return the same kind of node as the one where the extension was found. Now what about 'a. As you can see, it describes what comes after the base loc and path parameters of our expand_function. From the pattern point of view, 'a describes the parts of the matched AST node we wish to extract for later consumption, here by our expander.

Ast_pattern contains a whole bunch of combinators to let you describe what your pattern should match and a specific __ pattern that you must use to capture the various parts of the matched nodes. __ has type ('a, 'a -> 'b, 'b) Ast_pattern.t which means that whenever it's used it changes the type of consumer function in the returned pattern.

Let's consider a few examples to try wrapping our heads around this. Say I want to write an extension that takes an expression as a payload and I want to pass this expression to my expander so I can generate code based on its value. I can declare the extension like this:

let extension =
  Extension.declare
    "my_extension"
    Extension.Context.expression
    Ast_pattern.(single_expr_payload __)
    expand_function

In this example, Extension.Context.expresssion has type expression Extension.Context.t, the pattern has type (payload, expression -> expression, expression) Ast_pattern.t. The pattern says we want to allow a single expression in the payload and capture it. If we decompose it a bit, we can see that single_expr_payload has type (expression, 'a, 'b) Ast_pattern.t -> (payload, 'a, 'b) Ast_pattern.t and is passed __ which makes it a (expression, expression -> 'b, 'b) Ast_pattern.t and that's exactly what we want here as our expander will have type loc: Location.t -> path: string -> expression -> expression!

It works similarly to Scanf.scanf when you think about it. Changing the pattern changes the type of the consumer function the same way changing the format string does for Scanf functions.

This was a bit easy since we had a custom combinator just for that purpose so let's take a few more complex examples. Now say we want to only allow pairs of integer and string constants expressions in our payload. Instead of just capturing any expression and dealing with the error cases in the expand_function we can let Ast_pattern deal with that and pass an int and string along to our expander:

Ast_pattern.(single_expr_payload (pexp_tuple ((eint __)^::(estring __)^::nil)))

This one's a bit more elaborate but the idea is the same, we use __ to capture the int and string from the expression and use combinators to specify that the payload should be made of a pair and that gives us a: (payload, int -> string -> 'a, 'a) Ast_pattern.t which should be used with a loc: Location.t -> path: string -> int -> string -> expression expander.

We can also specify that our extension should take something else than an expression as a payload, say a pattern with no when clause so that it's applied as [%my_ext? some_pattern_payload]:

Ast_pattern.(ppat __ none)

or no payload at all and it should just be invoked as [%my_ext]:

Ast_pattern.(pstr nil)

You should play with Ast_pattern a bit if you need to express complex patterns as I think it's the only way to get the hang of it.

Writing a deriver

Registering a deriver is slightly different from registering an extension but in the end it remains relatively simple and you will still have to provide the actual implementation in the form of an expand function.

The typical my_ppx_deriver.ml will look like:

open Ppxlib

let str_type_decl_generator =
  Deriving.Generator.make_no_arg
    ~attributes
    expand_str

let sig_type_decl_generator =
  Deriving.Generator.make_no_arg
    ~attributes
    expand_sig

let my_deriver =
  Deriving.add
    ~str_type_decl:str_type_decl_generator
    ~sig_type_decl:sig_type_decl_generator
    "my_deriver"

Which you'll need to compile with the following library stanza:

(library
 (public_name my_ppx)
 (kind ppx_deriver)
 (libraries ppxlib))

The Deriving.add function is declared as:

val add
  :  ?str_type_decl:(structure, rec_flag * type_declaration list) Generator.t
  -> ?str_type_ext :(structure, type_extension                  ) Generator.t
  -> ?str_exception:(structure, extension_constructor           ) Generator.t
  -> ?sig_type_decl:(signature, rec_flag * type_declaration list) Generator.t
  -> ?sig_type_ext :(signature, type_extension                  ) Generator.t
  -> ?sig_exception:(signature, extension_constructor           ) Generator.t
  -> ?extension:(loc:Location.t -> path:string -> core_type -> expression)
  -> string
  -> t

It takes a mandatory string argument, here "my_deriver", which defines how user are going to invoke your deriver. In this case we'd need to add a [@@deriving my_deriver] to a type declaration in a structure or a signature to use it. Then there's just one optional argument per kind of node to which you can attach a [@@deriving ...] attribute. type_decl correspond to type = ..., type_ext to type += ... and exception to exception My_exc of .... You need to provide generators for the ones you wish your deriver to handle, ppxlib will make sure users get a compile error if they try to use it elsewhere. We can ignore the extension as it's just here for compatibility with ppx_deriving.

Now let's take a look at Generator. Its type is defined as ('output_ast, 'input_ast) t where 'input_ast is the type of the node to which the [@@deriving ...] is attached and 'output_ast the type of the nodes it should produce, ie either a structure or a signature. The type of a generator depends on the expand function it's built from when you use the smart constructo make_no_arg meaning the expand function should have type loc: Location.t -> path: string -> 'input_ast -> 'output_ast. This function is the actual implementation of your deriver and will generate the list of structure_item or signature_item from the type declaration.

Compatibility with ppx_import

ppx_import is a PPX rewriter that lets you import type definitions and spares you the need to copy and update them every time they change upstream. The main reason why you would want to do that is because you need to derive values from those types using a deriver thus the importance of ensuring your deriving plugin is compatible.

Let's take an example to illustrate how ppx_import is used. I'm using a library called blob which exposes a type Blob.t. For some reason I need to be able to serialize and deserialize Blob.t values to JSON. I'd like to use a deriver to do that as I don't want to maintain that code myself. Imagine Blob.t is defined as:

type t =
  { value : string
  ; length : int
  ; id : int
  }

Without ppx_import I would define somewhere a serializable_blob type as follows:

type serializable_blob = Blob.t =
  { value : string
  ; length : int
  ; id : int
  }
[@@deriving yojson]

That works well especially because the type definition is simple but I don't really care about having it here, what I really want is just the to_yojson and of_yojson functions. Also now, if the type definition changes, I have to update it here manually. Maintaining many such imports can be tedious and duplicates a lot of code unecessarily.

What I can do instead, thanks to ppx_import is to write it like this:

type serializable_blob = [%import: Blob.t]
[@@deriving yojson]

which will ultimately be expanded into the above using Blob's definition of the type t.

Now ppx_import works a bit differently from regular PPX rewriters as it needs a bit more information than just the AST. We don't need to understand how it works but what it means is that if your deriving plugin is used with ppx_import, it will be called twice:

• A first time with ocamldep. This is required to determine the dependencies of a module in terms of other OCaml modules. PPX-es need to be applied here to find out about dependencies they may introduce.
• A second time before actually compiling the code.

The issue here is that during the ocamldep pass, ppx_import doesn't have the information it needs to import the type definition yet so it can't copy it and it expands:

type u = [%import A.t]

into:

type u = A.t

Only during the second pass will it actually expand it to the copied type definition.

This may be a concern if your deriving plugin can't apply to abstract types because you will probably raise an error when encountering one, meaning the first phase will fail and the whole compilation will fail without giving your rewriter a chance to derive anything from the copied type definition.

The right way to deal with this is to have different a behaviour in the context of ocamldep. In this case you can ignore such type declaration or eventually, if you know you are going to inject new dependencies in your generated code, to create dummy values referencing them and just behave normally in any other context.

ppxlib versions 0.6.0 and higher allow you to do so through the Deriving.Generator.V2 API which passes an abstract ctxt value to your expand function instead of a loc and a path. You can tell whether it is the ocamldep pass from within the expand function like this:

open Ppxlib

let expand ~ctxt input_ast =
  let omp_config = Expansion_context.Deriver.omp_config ctxt in
  let is_ocamldep_pass = String.equal "ocamldep" omp_config.Migrate_parsetree.Driver.tool_name in
  ...

Deriver attributes

You'll have noted the attributes parameter in the examples. It's an optional parameter that lets you define which attributes your deriver allows the user to attach to various bits of the type, type extension or exception declaration it is applied to.

ppxlib comes with a Attribute module that lets you to properly declare the attributes you want to allow and make sure they are properly used: correctly spelled, placed and with the right payload attached. This is especially useful since attributes are by default ignored by the compiler meaning without ppxlib's care, plugin users wouldn't get any errors if they misused an attribute and it might take them a while to figure out they got it wrong and the generated code wasn't impacted as they hoped. The Attribute module offers another great feature: Attribute.t values can be used to extract the attribute payload from an AST node if it is present. That will spare you the need for inspecting attributes yourself which can prove quite tedious.

Ppxlib.Attribute.t is defined as ('context, 'payload) t where 'context describes to which node the attribute can be attached and 'payload, the type of its payload. To build such an attribute you must use Ppxlib.Attribute.declare:

val declare
  :  string
  -> 'a Context.t
  -> (payload, 'b, 'c) Ast_pattern.t
  -> 'b
  -> ('a, 'c) t

Let's try to declare the default argument from ppx_deriving_yojson I mentioned earlier.

The first string argument is the attribute name. ppxlib support namespaces for the attributes so that users can avoid conflicting attributes between various derivers applied to the same type definitions. For instance here we could use "default". It can prove helpful to use more qualified name such as "ppx_deriving_yojson.of_yojson.default". That means that our attribute can be used as [@@default ...], [@@of_yojson.default ...] or [@@ppx_deriving.of_yojson.default ...]. Now if another deriver uses a [@@default ...], users can apply both derivers and provide different default values to the different derivers by writing:

type t =
  { a : int
  ; b : string [@make.default "abc"] [@of_yojson.default ""]
  }
[@@deriving make,of_yojson]

The context argument works very similarly to the one in Extension.declare. Here we want the attribute to be attached to record field declarations so we'll use Attribute.Context.label_declaration which has type label_declaration Attribute.Context.t.

The pattern argument is an Ast_pattern.t. Now that we know how to work with those this is pretty easy. Here we need to accept any expression as a payload since we should be able to apply the default attribute to any field, regardless of its type and we want to extract that expression from the payload so we can use it in our deserializer so let's use Ast_pattern.(single_expr_payload __).

Finally the last 'b argument has the same type as the pattern consumer function. We can use it to transform what we extracted using the previous Ast_pattern but in this case we just want to keep the expression as we got it so we'll just use the identity function here.

We end up with the following:

let default_attribute =
  Attribute.declare
    "ppx_deriving_yojson.of_yojson.default"
    Attribute.Context.label_declaration
    Ast_pattern.(single_expr_payload __)
    (fun expr -> expr)

and that gives us a (label_declaration, expression) Attribute.t.

You can then use it to collect the attribute payload from a label_declaration:

Attribute.get default_attribute label_decl

which will return Some expr if the attribute was attached to label_decl or None otherwise.

Because of their polymorphic nature, attributes need to be packed, ie to be wrapped with a variant to hide the type parameter, so if you want to pass it to Generator.make_no_arg you'll have to do it like this:

let attributes = [Attribute.T default_attribute]

Writing your expand functions

In the two last sections I mentioned expand functions that would contain the actual deriver or extension implementation but didn't actually said anything about how to write those. It will depend a lot on the purpose of your PPX rewriter and what you're trying to achieve.

Before writing your PPX you should clearly specify what it should be applied to and what code it should produce. That will help you declaring the right deriving or extension rewriter and from there you'll know the type of the expand functions you have to write which should help.

A good way to proceed is to use the dumpast tool to pretty print the AST fragments of both the input of your expander and the output, ie the code it should generate. To take a concrete example, say you want to write a deriving plugin that generates an equal function from a type definition. You can start by running dumpast on the following file:

type some_record =
  { a : int64
  ; b : string
  }

let equal_some_record r r' = Int64.equal r.a r'.a && String.equal r.b r'.b

That will give you the AST representation of a record type definition and the equal function you want to write so you can figure out how to deconstruct your expander's input to be able to generate the right output.

ppxlib exposes smart constructors in Ppxlib.Ast_builder.Default to help you build AST fragments without having to care too much attributes and such fields as well as some convenience constructors to keep your code concise and readable.

Another convenience tool ppxlib exposes to help you build AST fragments is metaquot. I recently wrote a bit of documentation about it here which you should take a look at but to sum it up metaquot is a PPX extension allowing you to write AST nodes using the OCaml syntax they describe instead of the AST types.

Handling code locations in a PPX rewriter

When building AST fragments you should keep in mind that you have to set their location. Locations are part of the AST values that describes the position of the corresponding node in your source file, including the file name and the line number and offset of both the beginning and the end the code bit they represent.

Because your code was generated after the file was parsed, it doesn't have a location so you need to set it yourself. One could think that it doesn't matter and we could use a dummy location but locations are used by the compiler to properly report errors and that's why a PPX rewriter should care about how it locates the generated code as it will help the end user to understand whether the error comes from their code or generated code and how to eventually fix it.

Both Ast_builder and metaquot expect a location. The first explicitly takes it as a labelled loc argument while the second relies on a loc value being available in the scope. It is important to set those with care as errors in the generated code doesn't necessarily mean that your rewriter is bugged. There are valid cases where your rewriter functioned as intended but the generated code triggers an error. PPX-es often work on the assumption that some values are available in the scope, if the user doesn't properly provide those it's their responsibility to fix the error. To help them do so, it is important to properly locate the generated code to guide them as much as possible.

When writing extensions, using the whole extension point location for the generated code makes perfect sense as that's where the code will sit. That's fairly easy as this what ppxlib passes to the expand function through the loc labelled argument. For deriving plugins it's a bit different as the generated code doesn't replace an existing part of the parsed AST but generate a new one to insert. Currently ppxlib gives you the loc of the whole type declaration, extension or exception declaration your deriving plugin is applied to. Ideally it would be nice to be able to locate the generated code on the plugin name in the deriving attribute payload, ie here:

[@@deriving my_plugin,another_plugin]
            ^^^^^^^^^

I'm currently working on making that location available to the expand function. In the meantime, you should choose a convention. I personally locate all the generated code on the type declaration. Some choose to locate the generated code on the part of the input AST they're handling when generating it.

Reporting errors to your rewriter users

You won't always be able to handle all the AST nodes passed to your expand functions, either because the end user misused your rewriter or because there are some cases you simply can't deal with.

In those cases you can report the error to the user with Ppxlib.Location.raise_errorf. It works similarly to printf and you can build your error message from a format string and extra arguments. It will then raise an exception which will be caught and reported by the compiler. A good practice is to prefix the error message with the name of your rewriter to help users understand what's going on, especially with deriving plugin as they might use several of them on the same type declaration.

Another point to take care of here is, again, locations. raise_errorf takes a labelled loc arguments. It is used so that your error is reported as any compiler error. Having good locations in those error messages is just as important as sending clear error messages. Keep in mind that both the errors you report yourself or errors coming from your generated code will be highlighted by merlin so when properly set they make it much easier to work with your PPX rewriter.

Testing your PPX

Just as most pieces of code do, a PPX diserves to be tested and it has become easier over the years to test rewriters.

I personally tend to write as many unit test as possible for my PPX-es internal libraries. I try to extract helper functions that can easily be unit-tested but I can't test it all that way. Testing the ast -> ast functions would be tedious as ppxlib and ocaml-migrate-parsetree don't provide comparison and pretty printing functions that you can use with alcotest or oUnit. That means you'd have to import the AST types and derive them on your own. That would make a lot of boiler plate and even if those functions were exposed, writing such tests would be really tedious. There's a lot of things to take into account. How are you going to build the input AST values for instance? If you use metaquot, every node will share the same loc, making it hard to test that your errors are properly located. If you don't, you will end up with insanely long and unreadable test code or fixtures. While that would allow extremely accurate testing for the generated code and errors, it will almost certainly make your test code unmaintainable, at least given the current tooling.

Don't panic, there is a very good and simple alternative. ppxlib makes it very easy to build a binary that will parse OCaml code, preprocess the AST with your rewriter and spit it out, formatted as code again.

You just have to write the following pp.ml:

let () = Ppxlib.Driver.standalone ()

and build the binary with the following dune stanza, assuming your rewriter is called my_ppx_rewriter:

(executable
 (name pp)
 (modules pp)
 (libraries my_ppx_rewriter ppxlib))

Because we're humans and the OCaml syntax is meant for us to write and read, it makes for much better test input/output. You can now write your test input in a regular .ml file, use the pp.exe binary to "apply" your preprocessor to it and compare the output with another .ml file containing the code you expect it to generate. This kind of test pattern is really well supported by dune thanks to the diff user action.

I usually have the following files in a rewriter/deriver folder within my test directory:

test/rewriter/
├── dune
├── test.expected.ml
├── pp.ml
└── test.ml

Where pp.ml is used to produce the rewriter binary, test.ml contains the input OCaml code and test.expected.ml the result of preprocessing test.ml. The dune file content is generally similar to this:

(executable
 (name pp)
 (modules pp)
 (libraries my_ppx_rewriter ppxlib))

(rule
 (targets test.actual.ml)
 (deps (:pp pp.exe) (:input test.ml))
 (action (run ./%{pp} -deriving-keep-w32 both --impl %{input} -o %{targets})))

(alias
 (name runtest)
 (action (diff test.expected.ml test.actual.ml)))

(test
  (name test)
  (modules test)
  (preprocess (pps my_ppx_rewriter)))

The first stanza is the one I already introduced above and specifies how to build the rewriter binary.

The rule stanza that comes after that indicates to dune how to produce the actual test output by applying the rewriter binary to test.ml. You probably noticed the -deriving-keep-w32 both CLI option passed to pp.exe. By default, ppxlib will generate values or add attributes so that your generated code doesn't trigger a "Unused value" warning. This is useful in real life situation but here it will just polute the test output and make it harder to read so we disable that feature.

The following alias stanza is where all the magic happens. Running dune runtest will now generate test.actual.ml and compare it to test.expected.ml. It will not only do that but show you how they differ from each other in a diff format. You can then automatically update test.expected.ml if you're happy with the results by running dune promote.

Finally the last test stanza is there to ensure that the generated code compiles without type errors.

This makes a very convenient test setup to write your PPX-es TDD style. You can start by writing an identity PPX, that will just return its input AST as it is. Then you add some OCaml code using your soon to be PPX in test.ml and run dune runtest --auto-promote to prefill test.expected.ml. From there you can start implementing your rewriter and run dune runtest to check on your progress and update the expected result with dune promote. Going pure TDD by writing the test works but it's tricky cause you'd have to format your code the same way pp.exe will format the AST. It would be great to be able to specify how to format the generated test.actual.ml so that this approach would be more viable and the diff more readable. Being able to use ocamlformat with a very diff friendly configuration would be great there. pp.exe seems to offer CLI options to change the code style such as -styler but I haven't had the chance to experiment with those yet.

Now you can test successful rewriting this way but what about errors? There's a lot of value ensuring you produce the right errors and on the right code location because that's the kind of things you can get wrong when refactoring your rewriter code or when people try to contribute. That isn't as likely to happen if your CI yells when you break the error reporting. So how do we do that?

Well pretty much the exact same way! We write a file with an erroneous invocation of our rewriter, run pp.exe on it and compare stderr with what we expect it to be. There are two major differences here. First we want to collect the stderr output of the rewriter binary instead of using it to generate a file. The second is that we cant write all of our test cases in a single file since pp.exe will stop at the first error. That means we need one .ml file per error test case. Luckily for us, dune offers ways to do both.

For every error test file we will want to add the following stanzas:

(rule
  (targets test_error.actual)
  (deps (:pp pp.exe) (:input test_error.ml)) 
  (action
    (with-stderr-to
      %{targets}
      (bash "./%{pp} -no-color --impl %{input} || true")
    )
  )
)

(alias
  (name runtest)
  (action (diff test_error.expected test_error.actual))
)

but obviously we don't want to do that by hand every time we add a new test case so we're gonna need a script to generate those stanzas and then include them into our dune file using (include dune.inc).

To achieve that while keeping things as clean as possible I use the following directory structure:

test/rewriter/
├── errors
│   ├── dune
│   ├── dune.inc
│   ├── gen_dune_rules.ml
│   ├── pp.ml
│   ├── test_some_error.expected
│   ├── test_some_error.ml
│   ├── test_some_other_error.expected
│   └── test_some_other_error.ml
├── dune
├── test.expected.ml
├── pp.ml
└── test.ml

Compared to our previous setup, we only added the new errors folder. To keep things simple it has its own pp.ml copy but in the future I'd like to improve it a bit and be able to use the same pp.exe binary.

The most important files here are gen_dune_rules.ml and dune.inc. The first is just a simple OCaml script to generate the above stanzas for each test cases in the errors directory. The second is the file we'll include in the main dune. It's also the file to which we'll write the generated stanza.

I personally use the following gen_dune_rules.ml:

let output_stanzas filename =
  let base = Filename.remove_extension filename in
  Printf.printf
    {|
(library
  (name %s)
  (modules %s)
  (preprocess (pps ppx_yojson))
)

(rule
  (targets %s.actual)
  (deps (:pp pp.exe) (:input %s.ml))
  (action
    (with-stderr-to
      %%{targets}
      (bash "./%%{pp} -no-color --impl %%{input} || true")
    )
  )
)

(alias
  (name runtest)
  (action (diff %s.expected %s.actual))
)
|}
    base
    base
    base
    base
    base
    base

let is_error_test = function
  | "pp.ml" -> false
  | "gen_dune_rules.ml" -> false
  | filename -> Filename.check_suffix filename ".ml"

let () =
  Sys.readdir "."
  |> Array.to_list
  |> List.sort String.compare
  |> List.filter is_error_test
  |> List.iter output_stanzas

Nothing spectacular here, we just build the list of all the .ml files in the directory except pp.ml and gen_dune_rules.ml itself and then generate the right stanzas for each of them. You'll note the extra library stanza which I add to get dune to generate the right .merlin so that I can see the error highlights when I edit the files by hand.

With that we're almost good, add the following to the dune file and you're all set:

(executable
  (name pp)
  (modules pp)
  (libraries
    ppx_yojson
    ppxlib
  )
)

(include dune.inc)

(executable
  (name gen_dune_rules)
  (modules gen_dune_rules)
)

(rule
  (targets dune.inc.gen)
  (deps
    (:gen gen_dune_rules.exe)
    (source_tree .)
  )
  (action
    (with-stdout-to
      %{targets}
      (run %{gen})
    ) 
  )
)

(alias
  (name runtest)
  (action (diff dune.inc dune.inc.gen))
)

The first stanza is here to specify how to build the rewriter binary, same as before, while the second stanza just tells dune to include the content of dune.inc within this dune file.

The interesting part comes next. As you can guess the executable stanza builds our little OCaml script into a .exe. The rule that comes after that sepcifies how to generate the new stanzas by running gen_dune_rules and capturing its standard output into a dune.inc.gen file. The last rule allows you to review the changes to the generated stanza and use promotion to accept them. Once this is done, the new stanzas will be included to the dune file and the test will be run for every test cases.

Adding a new test case is then pretty easy, you can simply run:

$ touch test/rewriter/errors/some_explicit_test_case_name.{ml,expected} && dune runtest --auto-promote

That will create the new empty test case and update the dune.inc with the corresponding rules. From there you can proceed the same way as with the successful rewriting tests, update the .ml, run dune runtest to take a sneak peek at the output and dune promote once you're satisfied with the result.

I've been pretty happy with this setup so far although there's room for improvement. It would be nice to avoid duplicating pp.ml for errors testing. This also involves quite a bit of boilerplate that I have to copy into all my PPX rewriters repositories every time. Hopefully dune plugins should help with that and I can't wait for a first version to be released so that I can write a plugin to make this test pattern more accessible and easier to set up.

Let's talk sun, mint tea and OCaml: Yes, you got it, the MirageOS biennial retreat at Marrakesh!

For the 7th iteration of the retreat, the majority of the Tarides team took part in the trip to the camels country. This is a report about what we produced and enjoyed while there.

Charles-Edouard Lecat

That's it, my first MirageOS retreat is coming soon, let's jump in the plane and here I come. After a nice cab trip and an uncountable number of similar streets, I'm finally at the Ri…

Let's talk sun, mint tea and OCaml: Yes, you got it, the MirageOS biennial retreat at Marrakesh!

For the 7th iteration of the retreat, the majority of the Tarides team took part in the trip to the camels country. This is a report about what we produced and enjoyed while there.

Charles-Edouard Lecat

That's it, my first MirageOS retreat is coming soon, let's jump in the plane and here I come. After a nice cab trip and an uncountable number of similar streets, I'm finally at the Riad which will host me for the next 5 days.

It now begins the time to do what I came for: Code, Eat, Sleep and Repeat

I mostly worked on Colombe, the OCaml implementation of the SMTP protocol for which I developed a simple client. Except some delayed problems (like the integration of the MIME protocol, the TLS wrapping and some others), the client was working perfectly :) Implementing it was actually really easy as the core of the SMTP protocol was done by @dinosaure who developed over time a really nice way of implementing this kind of API. And as I spend most of my time at Tarides working on his code, I feel really comfortable with it.

One of the awesome thing about this retreat was the people who came: There was so many interesting people, doing various thing, so each time someone had an interrogation, you could almost be sure that someone could help you in a way or another.

But sadly, as I arrived few days after everyone, and just before the week-end, the time flew away reaaaaaally fast, and I did not have the time to do some major code, but I'm already looking forward to the next retreat which, I am sure, will be even more fruitful and attract a lot of nice OCaml developers.

Until then, I will just dream about the awesome food I ate there ;)

Lucas Pluvinage

Second Mirage retreat for me, and this time I had plans: make a small web game with Mirage hosted by an ESP32 device. I figured out that there was not canonical way to make an HTTP/Websocket server with Mirage and I didn't want to stick to a particular library.

Instead, I took my time to develop mirage-http, an abstraction of HTTP that can either have cohttp or httpaf as a backend. On top of that, I've build mirage-websocket which is therefore an HTTP server-independant implementation of websockets (indeed this has a lot of redundancies with ocaml-websocket but for now it's a proof of concept). While making all this I discussed with @anmonteiro who's the Webservers/protocols expert for Mirage ! However I didn't have the time to build something on top of that, but this is still something that I would like achieve at some point.

I also became the "dune guy" as I'm working on the Mirage/dune integration, and helped some people with their build system struggles.

It was definitely a rich week, I've learnt a lot of things, enjoyed the sun, ate good food and contributed to the Mirage universe !

Jules Aguillon

This was my first retreat. It was the occasion to meet OCaml developers from all over the world. The food was great and the weather perfect.

I submitted some PRs to the OCaml compiler !

• Hint on type error on int literal PR #2301

  It's adding an hint when using int literals instead of other number literals (eg. 3 instead of 3. or 3L):

  Line 2, characters 20-21:
  2 | let _ = Int32.add a 3
                          ^
  Error: This expression has type int but an expression was expected of type
           int32
         Hint: Did you mean `3l'?

• Hint on type error on int operators PR #2307

  Hint the user when using numerical operators for ints (eg. +) on other kind of numbers (eg. float, int64, etc..)

  For example:

  Line 8, characters 8-9:
  8 | let _ = x + 1.
              ^
  Error: This expression has type float but an expression was expected of type
           int
  Line 8, characters 10-11:
  8 | let _ = x + 1.
                ^
    Hint: Did you mean to use `+.'?

• Clean up int literal hint PR #2313

  A little cleanup of the 2 previous PRs.

• Hint when the expected type is wrapped in a ref PR #2319

  An other PR adding an hint: When the user forgot to use the ! operator on ref values:

  Line 2, characters 8-9:
  2 | let b = a + 1
              ^
  Error: This expression has type int ref
         but an expression was expected of type int
    Hint: This is a `ref', did you mean `!a'?

The first 3 are merged now.

Gabriel de Perthuis

For this retreat my plan was to do something a little different and work on Solo5.

Wodan, the storage layer I'm working on, needs two things from its backends which are not commonly implemented:

• support for discarding unused blocks (first implemented in mirage-block-unix), and
• support for barriers, which are ordering constraints between writes

Solo5 provides relevant mirage backends, which are themselves provided by various virtualised implementations. Discard was added to most of those, at least those that were common enough to be easily tested; we just added an "operation not supported" error code for the other cases.

The virtio implementation was interesting; recent additions to the spec allow discard support, but few virtual machine managers actually implement that on the backend side. I tried to integrate with the Chromium OS "crosvm" for that, and had a good time figuring out how it found the bootloader entry point (turns out the cpu was happily skipping past invalid instructions to find a slightly misaligned entry point), but ran out of time to figure out the rest of the integration, which seemed to be more complex that anticipated. Because of this virtio discard support will be skipped over for now.

I also visited the souk, which was an interesting experience. Turns out I'm bad at haggling, but I brought back interesting things anyway.

Conclusion

We'd like to thank Hannes Mehnert who organized this retreat and all the attendees who contributed to make it fruitful and inspiring. You want to take part in the next MirageOS retreat? Stay tuned here.

OCamlPro started working on blockchains in 2014, when Arthur Breitman came to us with an initial idea to develop the Tezos ledger. The idea was very challenging with a lot of innovations. So, we collaborated with him to write a specification, and to turn the specification into OCaml code. Since then, we continually improved our skills in this domain, trained more engineers, introduced the technology to students and to professionals, advised a dozen projects, developed tools and libraries, made …

OCamlPro started working on blockchains in 2014, when Arthur Breitman came to us with an initial idea to develop the Tezos ledger. The idea was very challenging with a lot of innovations. So, we collaborated with him to write a specification, and to turn the specification into OCaml code. Since then, we continually improved our skills in this domain, trained more engineers, introduced the technology to students and to professionals, advised a dozen projects, developed tools and libraries, made some improvements and extensions to the official Tezos node, and conducted several private deployments of the Tezos ledger.

For an overview of OCamlPro’s blockchain activities see http://blockchains.ocamlpro.com

TzScan: A complete Block Explorer for Tezos

TzScan is considered today to be the best block explorer for Tezos. It’s made of three main components:

• an indexer that queries the Tezos node and fills a relational database,
• an API server that queries the database to retrieve various informations,
• a web based user interface (a Javascript application)

We deployed the indexer and API to freely provide the community with an access to all the content of the Tezos blockchain, already used by many websites, wallets and apps. In addition, we directly use this API within our TzScan.io instance. Our deployment spans on multiple Tezos nodes, multiple API servers and a distributed database to scale and reply to millions of queries per day. We also regularly release open source versions under the GPL license, that can be easily deployed on private Tezos networks. TzScan’s development has been initiated in September 2017. It represents today an enormous investment, that the Tezos Foundation helped partially fund in July 2018.

Contact us for support, advanced features, advertisement, or if you need a private deployment of the TzScan infrastructure.

Liquidity: a Smart Contract Language for Tezos

Liquidity is the first high-level language for Tezos over Michelson. Its development began in April 2017, a few months before the Tezos fundraising in July 2017. It is today the most advanced language for Tezos: it offers OCaml-like and ReasonML-like syntaxes for writing smart contracts, compilation and de-compilation to/from Michelson, multiple-entry points, static type-checking à la ML, etc. Its online editor allows to develop smart contracts and to deploy them directly into the alphanet or mainnet. Liquidity has been used before the mainnet launch to de-compile the Foundation’s vesting smart contracts in order to review them. This smart contract language represents more than two years of work, and is fully funded by OCamlPro. It has been developed with formal verification in mind, formal verification being one of the selling points of Tezos. We have elaborated a detailed roadmap mixing model-checking and deductive program verification to investigate this feature. We are now searching for funding opportunities to keep developing and maintaining Liquidity.

See our online editor to get started ! Contact us if you need support, training, writing or in-depth analysis of your smart contracts.

Techelson: a testing framework for Michelson and Liquidity

Techelson is our newborn in the set of tools for the Tezos blockchain. It is a test execution engine for the functional properties of Michelson and Liquidity contracts. Techelson is still in its early development stage. The user documentation is available here. An example on how to use it with Liquidity is detailed in this post.

Contact us to customize the engine to suit your own needs!

IronTez: an optimized Tezos node by OCamlPro

IronTez is a tailored node for private (and public) deployments of Tezos. Among its additional features, the node adds some useful RPCs, improves storage, enables garbage collection and context pruning, allows an easy configuration of the private network, provides additional Michelson instructions (GET_STORAGE, CATCH…). One of its nice features is the ability to enable adaptive baking in private / proof-of-authority setting (eg. baking every 5 seconds in presence of transactions and every 10 minutes otherwise, etc.).

A simplified version of IronTez has already been made public to allow testing its improved storage system, Ironmin, showing a 10x reduction in storage. Some TzScan.io nodes are also using versions of IronTez. We’ve also successfully deployed it along with TzScan for a big foreign company to experiment with private blockchains. We are searching for projects and funding opportunities to keep developing and maintaining this optimized version of the Tezos node.

Don’t hesitate to contact us if you want to deploy a blockchain with IronTez, or for more information !

Looking for more services from OCamlPro?

• Custom adaptation of our tools for your specific needs: TzScan, Liquidity, IronTez, Techelson
• Research projects, on verification of Tezos smart contracts
• Development of smart contracts fitted to your needs
• In-depth and comprehensive analysis of your smart contracts
• Training Liquidity or Tezos sessions for your team
• Prototyping and development of blockchains related solutions
• Development of new protocol amendments for Tezos
• Deployment of private Tezos networks
• Development of full end-to-end blockchain applications
• Tailored services for Tezos : node hosting, premium TzScan APIs, etc.

Services around other blockchains

Though we have built a huge experience of Tezos, our work also benefits from studying and experimenting other blockchains, such as Bitcoin or Ethererum. For example, we prototyped and developed the Tezos fundraising platform from the ground up, encompassing the web app (back-end and front-end), the Ethereum and Bitcoin (p2sh) multi-signature contracts, as well as the hardware Ledger based process for transferring funds. We also helped prototype and conduct research for some of our partners on the Bitcoin and Ethereum blockchains, and we are also following the recent developments on emerging blockchains like Cardano and Cosmos. Currently, we are deploying a notarization service on top of multiple blockchains.

We can help develop and deploy applications interacting with Bitcoin and Ethereum Blockchains.

The Blockchains@OCamlPro page: http://blockchains.ocamlpro.com

MirageOS Security Advisory 02 - grant unshare vulnerability in mirage-xen

• Module: mirage-xen
• Announced: 2019-04-25
• Credits: Thomas Leonard, Mindy Preston
• Affects: mirage-xen < 3.3.0, mirage-block-xen < 1.6.1, mirage-net-xen < 1.10.2, mirage-console < 2.4.2, ocaml-vchan < 4.0.2, ocaml-gnt (no longer supported)
• Corrected: 2019-04-22: mirage-xen 3.4.0, 2019-04-05: mirage-block-xen 1.6.1, 2019-04-02: mirage-net-xen 1.10.2, 2019-03-27: mirage-console 2.4.2, …

MirageOS Security Advisory 02 - grant unshare vulnerability in mirage-xen

• Module: mirage-xen
• Announced: 2019-04-25
• Credits: Thomas Leonard, Mindy Preston
• Affects: mirage-xen < 3.3.0, mirage-block-xen < 1.6.1, mirage-net-xen < 1.10.2, mirage-console < 2.4.2, ocaml-vchan < 4.0.2, ocaml-gnt (no longer supported)
• Corrected: 2019-04-22: mirage-xen 3.4.0, 2019-04-05: mirage-block-xen 1.6.1, 2019-04-02: mirage-net-xen 1.10.2, 2019-03-27: mirage-console 2.4.2, 2019-03-27: ocaml-vchan 4.0.2

For general information regarding MirageOS Security Advisories, please visit https://mirage.io/security.

Background

MirageOS is a library operating system using cooperative multitasking, which can be executed as a guest of the Xen hypervisor. Virtual machines running on a Xen host can communicate by sharing pages of memory. For example, when a Mirage VM wants to use a virtual network device provided by a Linux dom0:

1. The Mirage VM reserves some of its memory for this purpose and writes an entry to its grant table to say that dom0 should have access to it.
2. The Mirage VM tells dom0 (via XenStore) about the grant.
3. dom0 asks Xen to map the memory into its address space.

The Mirage VM and dom0 can now communicate using this shared memory. When dom0 has finished with the memory:

1. dom0 tells Xen to unmap the memory from its address space.
2. dom0 tells the Mirage VM that it no longer needs the memory.
3. The Mirage VM removes the entry from its grant table.
4. The Mirage VM may reuse the memory for other purposes.

Problem Description

Mirage removes the entry by calling the gnttab_end_access function in Mini-OS. This function checks whether the remote domain still has the memory mapped. If so, it returns 0 to indicate that the entry cannot be removed yet. To make this function available to OCaml code, the stub_gntshr_end_access C stub in mirage-xen wrapped this with the OCaml calling conventions. Unfortunately, it ignored the return code and reported success in all cases.

Impact

A malicious VM can tell a MirageOS unikernel that it has finished using some shared memory while it is still mapped. The Mirage unikernel will think that the unshare operation has succeeded and may reuse the memory, or allow it to be garbage collected. The malicious VM will still have access to the memory.

In many cases (such as in the example above) the remote domain will be dom0, which is already fully trusted. However, if a unikernel shares memory with an untrusted domain then there is a problem.

Workaround

No workaround is available.

Solution

Returning the result from the C stub required changes to the OCaml grant API to deal with the result. This turned out to be difficult because, for historical reasons, the OCaml part of the API was in the ocaml-gnt package while the C stubs were in mirage-xen, and because the C stubs are also shared with the Unix backend.

We instead created a new grant API in mirage-xen, migrated all existing Mirage drivers to use it, and then dropped support for the old API. mirage-xen 3.3.0 added support for the new API and 3.4.0 removed support for the old one.

The recommended way to upgrade is:

opam update
opam upgrade mirage-xen

Correction details

The following PRs were part of the fix:

• mirage-xen/pull/9 - Add grant-handling code to OS.Xen
• mirage-net-xen/pull/85 - Use new OS.Xen API for grants
• ocaml-vchan/pull/125 - Update to new OS.Xen grant API
• mirage-block-xen/pull/79 - Port to new grant interface provided by mirage-xen
• mirage-console/pull/75 - Use new grant interface in mirage-xen
• mirage-xen/pull/12 - Drop support for old ocaml-gnt package

References

You can find the latest version of this advisory online at https://mirage.io/blog/MSA02.

This advisory is signed using OpenPGP, you can verify the signature by downloading our public key from a keyserver (gpg --recv-key 4A732D757C0EDA74), downloading the raw markdown source of this advisory from GitHub and executing gpg --verify 02.txt.asc.

Between March 6th and March 13th 2019, I attended the Mirage retreat organized by Hannes Mehnert in Marrakesh, Morocco.

The Mirage retreat takes place in an artist residency organized as a hostel (shared rooms with simple beds). Hannes gathers a lot of people whose activity is relevant to the Mirage project; some of them work within the OCaml ecosystem (but not necessarily Mirage), some work on system programming (not necessarily in OCaml). The whole place is for all of us for one week, ex…

Between March 6th and March 13th 2019, I attended the Mirage retreat organized by Hannes Mehnert in Marrakesh, Morocco.

The Mirage retreat takes place in an artist residency organized as a hostel (shared rooms with simple beds). Hannes gathers a lot of people whose activity is relevant to the Mirage project; some of them work within the OCaml ecosystem (but not necessarily Mirage), some work on system programming (not necessarily in OCaml). The whole place is for all of us for one week, excellent food is provided, and we get to do whatever we want. (Thanks to the work of the people there who make this possible.)

This was my second time attending the retreat – first time in November 2017. It is probably the work-related trip I enjoy most. When I get back home, I’m exhausted, thrilled, I met fascinating people and I learned a lot.

Marracheck

This week I came with a specific project (that was decided spontaneously maybe three weeks before that): I would work with Armaël Guéneau, also attending the retreat, on mixing ideas for the existing tools to check opam packages (opam-builder, opamcheck, opam-health-check), with the objective of building a tool that can build the whole opam-repository is less than a day on my laptop machine. The ultimate goal is to make it extremely easy for anyone to test the impact of a change to the OCaml compiler on the OCaml ecosystem.

We started with a lot of discussions on the design, inspired by our knowledge of opam-builder and opamcheck – I had hacked on opam-builder a bit before, and had detailed discussion with Damien about the design of opamcheck – and discussions with Kate, working on opam-health-check and in general the opam CI, also attending the retreat. Then we did a good bit of pair-programming, with Armaël behind the keyboard. We decided to rebuild a tool from scratch and to use the opam-libs (the library-level API of the opam codebase). By the end of the week, we were still quite far from a working release, but we have a skeleton in place.

This would not have been possible without the presence, at the retreat, of Louis Gesbert and Raja Boujbel, who helped us navigating the (sometimes daunting) opam API. (We also noticed a few opportunities for improvements and sent a couple pull-requests to opam itself.) Louis and Raja are impressive in their handling of the opam codebase. There are obscure and ugly and painful things within the opam APIs, but they come from elegance/simplicity/pragmatism compromises that they made, understand well, and are consistently able to justify. It feels like a complex codebase that is growing as it discovers its use-cases, with inevitable cruft, but good hands at work to manage this complexity, within resource limits.

Network drivers

My roommate was Fabian Bonk, who participated to the “ixy project”, at the TUM (Technische Universität München), Munich, Germany. The "ixy project’ aims to implement a simple userland network driver (for some specific Intel network card) in many different languages, and see what work and what doesn’t. Fabian wrote the OCaml implementation, and was interested in finding ways to improve its performances.

At first I preferred to hear about his work from a distance; I know nothing of network card, and there were people at the retreat noticeably more knowledgeable about writing high-performance OCaml code. Then I realized that this was a dangerous strategy: for essientally any topic there is someone more knowledgeable than you at the retreat. So why not work on userland network drivers?

Fabian and I made a few attempts at making the program faster, which had the somewhat hilarious result of making the program about 500x slower. It’s an interesting problem domain. The driver author says “I would really need to remove this copy here, even though that would require changing the whole Mirage API”; the first reaction is to argue that copying memory is actually quite fast, so it’s probably not the bottleneck. “But we have to copy”, they say, “ten gibibytes per second!”. Ouch.

Anyway, after some tinkering, I realized that Fabian working over SSH to a machine in Munich with the network card, and being able to run latency tests because “for this you need an optical splitter and I don’t have the privilege level to access the one our university has”, wasn’t that great for benchmarking. So I decided to convince Fabian to implement a compliant network card, in C, on his machine – he insists on calling it a “simulator” but what’s the difference between software and hardware these days? The idea was, anyone could then use his network card on their own machine to test and benchmark the driver against, and make it faster. Unfortunately, he had an OSX machine, and everything you would need to implement this nicely is sort of broken on OSX (working POSIX-compliant shared memory? nope!).

Crowbar

One thing I realized during the retreat is that I have an amazing (dis)advantage over some other people there (Hannes included): I know that Crowbar is extremely easy to use. (You write a generator, a quickcheck-style test, you listen to the tool tell you how to set a few weird environment variables, and boom, there come the bugs.)

Apparently people who haven’t tried Crowbar yet are unwilling to do so because they’re not sure how easy it is. Unfortunately, having the amazing power to find bugs is also a curse: Hannes persuaded me to write a test for an OCaml implementation of the patch utility instead of sleeping one evening.

I’m not sure what tool authors can do to reduce this particular barrier to entry. Maybe one thing that works is to simply demo the tool in front of a crowded audience, every time you have a chance.

Mystery in a box

I helped Antonio Monteiro track down a failure in his HTTP/2 implementation, and over the course of that I caught a glimpse of the angstrom codebase. Angstrom contains a single-field record (used for first-class polymorphism) in one apparently-central data structure, and I noticed that the record does not carry the [@@unboxed] annotation (so it is one extra indirection at runtime). So I decided to add the annotation, hoping it would improve performances.

There is a dirty secret about [@@unboxed]: despite what most people think, it is extremely rare that it can make programs noticeably faster, because the GC is quite fast and combines allocations together – many allocations of a boxed object are combined with an allocation for their content or container, and and often the indirection points to an immediately adjacent place in memory so is basically free to dereference. It may help in some extreme low-latency scenario where code is written to not allocate at all, but I have never personally seen a program where [@@unboxed] makes a noticeable performance difference.

That is, until angstrom. Adding [@@unboxed] to this record field makes the program noticeably slower. The generated code, at the point where this record is used, is much nicer, each run allocates less words, but the program is noticeably slower – 7% slower. I found it extremely puzzling; Romain @dinosaure Calabiscetta pointed out that he tried the same thing, and was similarly puzzled.

Eventually I lured Pierre Chambart into studying the problem with me, and we figured it out. I won’t get into the technical details here (hopefully later), but I’ll point out that we tested our hypothesis by inserting (); in several places in the program, and Armaël’s baffled look made it more than worthwhile.

Video games

When invited to work on “anything they wanted”, some people had a projects that sounded a bit more fun than a parallel compiler of all OPAM packages – a gameboy emulator, for example. Of course, everyone knows that having fun side-projects to work on from time to time is an excellent thing. Yet those were kept in the recent past after more pressing side-projects on my TODO list (typically releasing ocamlbuild, batteries, ppx_deriving and plugins once in a while, or reviewing a compiler Pull Request). While no one was working on that specifically, this retreat made me want to try (eventually) something that I’ve never done before, namely implement a video game in OCaml. We’ll see whether that happens someday.

Conclusion

Thanks to everyone who was at the retreat (including the people that worked hard to ensure we could be there in the best condition). I had a great time and I’m hoping to come again for one of the next retreats.

We are pleased to announce the release of opam 2.0.4.

This new version contains some backported fixes:

• Sandboxing on macOS: considering the possibility that TMPDIR is unset [#3597 @herbelin – fix #3576]
• display: Fix opam config var display, aligned on opam config list [#3723 @rjbou – rel. #3717]
• pin:
  • update source of (version) pinned directory [#3726 @rjbou – fix #3651]
  • fix --ignore-pin-depends with autopin [#3736 @AltGr]
  • fix pinnings not installing/upgrading already pinned p…

We are pleased to announce the release of opam 2.0.4.

This new version contains some backported fixes:

• Sandboxing on macOS: considering the possibility that TMPDIR is unset [#3597 @herbelin – fix #3576]
• display: Fix opam config var display, aligned on opam config list [#3723 @rjbou – rel. #3717]
• pin:
  • update source of (version) pinned directory [#3726 @rjbou – fix #3651]
  • fix --ignore-pin-depends with autopin [#3736 @AltGr]
  • fix pinnings not installing/upgrading already pinned packages (introduced in 2.0.2) [#3800 @AltGr]
• opam clean: Ignore errors trying to remove directories [#3732 @kit-ty-kate]
• remove wrong “mismatched extra-files” warning [#3744 @rjbou]
• urls: fix hg opam 1.2 url parsing [#3754 @rjbou]
• lint: update message of warning 47, to avoid confusion because of missing synopsis field internally inferred from descr [#3753 @rjbou – fix #3738]
• system:
  • lock & signals: don’t interrupt at non terminal signals [#3541 @rjbou]
  • shell: fix fish manpath setting [#3728 @gregory-nisbet]
  • git: use diff.noprefix=false config argument to overwrite user defined configuration [#3788 @rjbou, #3628 @Blaisorblade – fix #3627]
• dirtrack: fix precise tracking mode [#3796 @rjbou]
• fix some mispellings [#3731 @MisterDA]
• CI enhancement & fixes [#3706 @dra27, #3748 @rjbou, #3801 @rjbou]

Note: To homogenise macOS name on system detection, we decided to keep macos, and convert darwin to macos in opam. For the moment, to not break jobs & CIs, we keep uploading darwin & macos binaries, but from the 2.1.0 release, only macos ones will be kept.

Installation instructions (unchanged):

1. From binaries: run

   sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)

   or download manually from the Github “Releases” page to your PATH. In this case, don’t forget to run opam init --reinit -ni to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.

2. From source, using opam:

   opam update; opam install opam-devel

(then copy the opam binary to your PATH as explained, and don’t forget to run opam init --reinit -ni to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script)

3. From source, manually: see the instructions in the README.

We hope you enjoy this new minor version, and remain open to bug reports and suggestions.

NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!

We are pleased to announce the release of opam 2.0.4.

This new version contains some backported fixes:

• Sandboxing on macOS: considering the possibility that TMPDIR is unset [#3597 @herbelin - fix #3576]
• display: Fix opam config var display, aligned on opam config list [#3723 @rjbou - rel. #3717]
• pin:
  • update source of (version) pinned directory [#3726 @rjbou - #3651]
  • fix --ignore-pin-depends with autopin [#3736 @AltGr]
  • fix pinnings not installing/upgrading already pinned packages (introduced in 2.0.…

We are pleased to announce the release of opam 2.0.4.

This new version contains some backported fixes:

• Sandboxing on macOS: considering the possibility that TMPDIR is unset [#3597 @herbelin - fix #3576]
• display: Fix opam config var display, aligned on opam config list [#3723 @rjbou - rel. #3717]
• pin:
  • update source of (version) pinned directory [#3726 @rjbou - #3651]
  • fix --ignore-pin-depends with autopin [#3736 @AltGr]
  • fix pinnings not installing/upgrading already pinned packages (introduced in 2.0.2) [#3800 @AltGr]
• opam clean: Ignore errors trying to remove directories [#3732 @kit-ty-kate]
• remove wrong "mismatched extra-files" warning [#3744 @rjbou]
• urls: fix hg opam 1.2 url parsing [#3754 @rjbou]
• lint: update message of warning 47, to avoid confusion because of missing synopsis field internally inferred from descr [#3753 @rjbou - fix #3738]
• system:
  • lock & signals: don't interrupt at non terminal signals [#3541 @rjbou]
  • shell: fix fish manpath setting [#3728 @gregory-nisbet]
  • git: use diff.noprefix=false config argument to overwrite user defined configuration [#3788 @rjbou, #3628 @Blaisorblade - fix #3627]
• dirtrack: fix precise tracking mode [#3796 @rjbou]
• fix some mispellings [#3731 @MisterDA]
• CI enhancement & fixes [#3706 @dra27, #3748 @rjbou, #3801 @rjbou]

Note: To homogenise macOS name on system detection, we decided to keep macos, and convert darwin to macos in opam. For the moment, to not break jobs & CIs, we keep uploading darwin & macos binaries, but from the 2.1.0 release, only macos ones will be kept.

1. From binaries: run

   sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)

   or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.

2. From source, using opam:

   opam update; opam install opam-devel

   (then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script)

3. From source, manually: see the instructions in the README.

We hope you enjoy this new minor version, and remain open to bug reports and suggestions.

NOTE: this article is cross-posted on opam.ocaml.org and ocamlpro.com. Please head to the latter for the comments!

Tarides is pleased to have contributed to the dune 1.9.0 release which introduces the concept of library variants. Thanks to this update, unikernels builds are becoming easier and faster in the MirageOS universe! This also opens the door for a better cross-compilation story, which will ease the addition of new MirageOS backends (trustzone, ESP32, RISC-V, etc.)

This post has also been posted to the Dune blog. See also the the discuss forum for more details.

Dune 1.9.0

Changes include:

• Coloring …

Tarides is pleased to have contributed to the dune 1.9.0 release which introduces the concept of library variants. Thanks to this update, unikernels builds are becoming easier and faster in the MirageOS universe! This also opens the door for a better cross-compilation story, which will ease the addition of new MirageOS backends (trustzone, ESP32, RISC-V, etc.)

This post has also been posted to the Dune blog. See also the the discuss forum for more details.

Dune 1.9.0

Changes include:

• Coloring in the watch mode (#1956)
• $ dune init command to create or update project boilerplate (#1448)
• Allow "." in c_names and cxx_names (#2036)
• Experimental Coq support
• Support for library variants and default implementations (#1900)

Variants

In dune 1.7.0, the concept of virtual library was introduced: https://dune.build/blog/virtual-libraries/. This feature allows to mark some abstract library as virtual, and then have several implementations for it. These implementations could be for multiple targets (unix, xen, js), using different algorithms, using C code or not. However each implementation in a project dependency tree had to be manually selected. Dune 1.9.0 introduces features for automatic selection of implementations.

Library variants

Variants is a tagging mechanism to select implementations on the final linking step. There's not much to add to make your implementation use variants. For example, you could decide to design a bar_js library which is the javascript implementation of bar, a virtual library. All you need to do is specificy a js tag using the variant option.

(library
 (name bar_js)
 (implements bar)
 (variant js)); <-- variant specification

Now any executable that depend on bar can automatically select the bar_js library variant using the variants option in the dune file.

(executable
 (name foo)
 (libraries bar baz)
 (variants js)); <-- variants selection

Common variants

Language selection

In your projects you might want to trade off speed for portability:

• ocaml: pure OCaml
• c: OCaml accelerated by C

JavaScript backend

• js: code aiming for a Node backend, using Js_of_ocaml

Mirage backends

The Mirage project (mirage.io) will make extensive use of this feature in order to select the appropriate dependencies according to the selected backend.

• unix: Unikernels as Unix applications, running on top of mirage-unix
• xen: Xen backend, on top of mirage-xen
• freestanding: Freestanding backend, on top of mirage-solo5

Default implementation

To facilitate the transition from normal libraries into virtuals ones, it's possible to specify an implementation that is selected by default. This default implementation is selected if no implementation is chosen after variant resolution.

(library
 (name bar)
 (virtual_modules hello)
 (default_implementation bar_unix)); <-- default implementation selection

Selection mechanism

Implementation is done with respect to some priority rules:

• manual selection of an implementation overrides everything
• after that comes selection by variants
• finally unimplemented virtual libraries can select their default implementation

Libraries may depend on specific implementations but this is not recommended. In this case, several things can happen:

• the implementation conflicts with a manually selected implementation: resolution fails.
• the implementation overrides variants and default implementations: a cycle check is done and this either resolves or fails.

Conclusion

Variant libraries and default implementations are fully documented here. This feature improves the usability of virtual libraries.

This commit shows the amount of changes needed to make a virtual library use variants.

Coq support

Dune now supports building Coq projects. To enable the experimental Coq extension, add (using coq 0.1) to your dune-project file. Then, you can use the (coqlib ...) stanza to declare Coq libraries.

A typical dune file for a Coq project will look like:

(include_subdirs qualified) ; Use if your development is based on sub directories

(coqlib
  (name Equations)                  ; Name of wrapper module
  (public_name equations.Equations) ; Generate an .install file
  (synopsis "Equations Plugin")     ; Synopsis
  (libraries equations.plugin)      ; ML dependencies (for plugins)
  (modules :standard \ IdDec)       ; modules to build
  (flags -w -notation-override))    ; coqc flags

See the documentation of the extension for more details.

Credits

This release also contains many other changes and bug fixes that can be found on the discuss announce.

Special thanks to dune maintainers and contributors for this release: @rgrinberg, @emillon, @shonfeder and @ejgallego!

MirageOS Spring Hack Retreat, Marrakesh 2019

Early March 2019, 31 MirageOS hackers gathered again in Marrakesh for our bi-annual hack retreat. We'd like to thank our amazing hosts, and everyone who participated on-site or remotely, and especially those who wrote up their experiences.

On this retreat, we ate our own dogfood, and used our MirageOS DHCP, recursive DNS resolver, and CalDAV unikernels as isolated virtual machines running on a PC Engines APU with FreeBSD as host system. The CalDAV s…

MirageOS Spring Hack Retreat, Marrakesh 2019

Early March 2019, 31 MirageOS hackers gathered again in Marrakesh for our bi-annual hack retreat. We'd like to thank our amazing hosts, and everyone who participated on-site or remotely, and especially those who wrote up their experiences.

On this retreat, we ate our own dogfood, and used our MirageOS DHCP, recursive DNS resolver, and CalDAV unikernels as isolated virtual machines running on a PC Engines APU with FreeBSD as host system. The CalDAV server persisted its data in a git repository on the host system, using the raw git protocol for communication, the smart HTTP protocol could have been used as well. Lynxis wrote a detailed blog post about our uplink situation.

Lots of interesting discussions took place, code was developed, knowledge was exchanged, and issues were solved while we enjoyed the sun and the Moroccan food. The following list is not exhaustive, but gives an overview what was pushed forward.

Imagelib

Imagelib is a library that can parse several image formats, and eye-of-mirage uses it to display those images in a framebuffer.

During the retreat, imagelib was extended with support for the BMP format, it's build system was revised to split off Unix-dependent functionality, and preliminary support for the GIF format was implemented.

ActivityPub

ActivityPub is an open, decentralized social networking protocol, as used by mastodon. It provides a client/server API for creating, updating, and deleting content, and a federated server-to-server API for notifications and content delivery. During the retreat, an initial prototype of a protocol implementation was drafted.

opam

Opam, the OCaml package manager, was extended in several directions:

• External (OS package system) dependency integration
• Interleaving download with build/install actions
• Generalisation of the job scheduler
• JSON serialisation, including crowbar round-trip tests
• Plugin evaluating (binary) reproducibility of opam packages
• some smaller cleanup PRs (return values, locking code)

marracheck

Work was started on a new utility to install as many opam packages as possible on a machine (there just wasn't enough choice with opam-builder, opamcheck and opam-check-all). It uses opam-lib and Z3 to accomplish this.

Conex

Conex is used for signing community repositories, esp. the opam-repository. Any opam package author can cryptographically sign their package releases, and users can verify that the downloaded tarball and build instructions are identical to what the author intended.

Conex has been developed since 2015, but is not yet widely deployed. We extended opam-publish to invoke the conex_targets utility and sign before opening a pull request on the opam-repository.

SMTP

The simple mail transfer protocol is an Internet standard for sending and receiving eMail. Our OCaml implementation has been improved, and it is possible to send eMails from OCaml code now.

HTTP2

The hypertext transfer protocol is an Internet standard widely used for browsing the world wide web. HTTP 1.1 is a line-based protocol which was specified 20 years ago. HTTP2 is an attempt to fix various shortcomings, and uses a binary protocol with multiplexing, priorities, etc. An OCaml implementation of HTTP2 has been actively worked on in Marrakesh.

Irmin

Irmin is a distributed database that follows the same design principles as git. Soon, Irmin 2.0 will be released, which includes GraphQL, HTTP, chunk support, and can use the git protocol for interoperability. Irmin provides a key-value interface for MirageOS.

OCaml compiler

Some hints on type errors for int literals and int operators were developed and merged to the OCaml compiler.

# 1.5 +. 2;;
         ^
Error: This expression has type int but an expression was expected of type
         float
       Hint: Did you mean `2.'?

# 1.5 + 2.;;
  ^^^ ^
Error: This expression has type float but an expression was expected of type
         int
Line 1, characters 4-5:
  Hint: Did you mean to use `+.'?

Also, the whole program dead code elimination PR was rebased onto trunk.

BGP / lazy trie

The mrt-format library which can parse multi-threaded routing toolkit traces, has been adapted to the modern OCaml ecosystem. The border gateway protocol (BGP) library was slightly updated, one of its dependencies, lazy-trie was adapted to the modern ecosystem as well.

Xen PVH

Xen provides several modes for virtualization. MirageOS's first non-Unix target was the para-virtualized (PV) mode for Xen, which does not require hardware support from the hypervisor's host operating system but has a weak security profile (static mapping of addresses, large attack surface). However, PV mode provides an attractive target for unikernels because it provides a simple software-based abstraction for dealing with drivers and a simple memory model; this is in contrast to hardware-virtualization mode, which provides greater security but requires more work from the guest OS.

A more modern virtualization mode combining the virtues of both approaches is PVH (formerly referred to as HVMLite), which is not yet supported by MirageOS. Marek Marczykowski-Górecki from the QubesOS project visited to help us bring PVH support to the unikraft project, a common platform for building unikernels which we hope to use for MirageOS's Xen support in the future.

During the retreat, lots of bugs porting MirageOS to PVH were solved. It boots and crashes now!

Learn OCaml as a unikernel

The platform learn OCaml embeds an editor, top-level, and exercises into a HTTP server, and allows students to learn OCaml, and submit solutions via the web interface, where an automated grader runs unit tests etc. to evaluate the submitted solutions. Teachers can assign mandatory exercises, and have an overview how the students are doing. Learn OCaml used to be executable only on a Unix host, but is now beeing ported into a MirageOS unikernel, executable as a standalone virtual machine.

Network device driver (ixy)

The ixy network driver supports Intel 82599 network interface cards, and is implemented in OCaml. Its performance has been improved, including several failing attempts which degraded its performance. Also, it has been integrated into the mirage tool and is usable as a mirage-net implementation.

DNS client API

Our proposed API is described here. Unix, Lwt, and MirageOS implementations are already available.

mirage-http unified HTTP API

Since we now have two HTTP servers, cohttp and httpaf, in OCaml and MirageOS available, the new interface mirage-http provides a unified interface, and also supports connection upgrades to websockets.

cstruct capabilities

We use cstruct, a wrapper around OCaml's Bigarray, quite a lot in MirageOS. Until now, cstruct is a readable and writable byte array. We used phantom types to add capabilities to the interface to distinct read-only and write-only buffers.

patch

An OCaml implementation to apply unified diffs. This code has been extracted from conex (since we found some issues in it), and still needs to be fixed.

Statistical memory profiler

Since 2016, Jacques-Henri Jourdan has been working on a statistical memory profiler for OCaml (read the OCaml 2016 paper). An Emacs user interface is available since some years. We integrated statmemprof into MirageOS unikernels using the statmemprof-mirage library, marshal the data via TCP, and provide a proxy that communicates with Emacs over a Unix domain socket, and the unikernel.

P2Pcollab

P2Pcollab is a collection of composable libraries implementing protocols for P2P collaboration. So far various P2P gossip protocols has been implemented. At this retreat the focus was on a gossip-based publish-subscribe dissemination protocol. Future plans include building P2P unikernels and adding P2P pub/sub sync functionality to Irmin.

It has been 3 months since I joined IIT Madras and it has been good fun so far. Along with the members of the RISE group, we’ve initiated a project to build secure applications on top of secure extensions of the open-source Shakti RISC-V processor ecosystem. Unsurprisingly, my language of choice to build the applications is OCaml. Given the availability of rich ecosystem of libraries under the MirageOS library operating system for building unikernels, we hope to minimise the amount of unsafe C…

It has been 3 months since I joined IIT Madras and it has been good fun so far. Along with the members of the RISE group, we’ve initiated a project to build secure applications on top of secure extensions of the open-source Shakti RISC-V processor ecosystem. Unsurprisingly, my language of choice to build the applications is OCaml. Given the availability of rich ecosystem of libraries under the MirageOS library operating system for building unikernels, we hope to minimise the amount of unsafe C code that the hardware has to contend with and protect exploits against. As a first step, we have managed to get OCaml programs to run on directly on top of the Shakti processor running in simulation under QEMU and Spike ISA simulators without an intervening operating system.

A custom bootloader performs the necessary hardware initialisation and transfers control directly to the OCaml program. We have open-sourced all of the tools necessary to build your own kernel. This handy dockerfile documents the entire process. For the impatient, an image is available in the dockerhub:

$ docker run -it iitmshakti/riscv-ocaml-baremetal:0.1.0

# Write your program
$ echo 'let _ = print_endline "A camel treads on hardware!"' > hello.ml
# Compile for Shakti
$ ocamlopt -output-obj -o payload.o hello.ml
$ file payload.o
payload.o: ELF 64-bit LSB relocatable, UCB RISC-V, version 1 (SYSV), not stripped

# Link with bootcode and build the kernel
$ make -C ../build
make: Entering directory '/root/ocaml-baremetal-riscv/build'
make[1]: Entering directory '/root/ocaml-baremetal-riscv/build'
make[2]: Entering directory '/root/ocaml-baremetal-riscv/build'
make[2]: Leaving directory '/root/ocaml-baremetal-riscv/build'
[ 64%] Built target boot
make[2]: Entering directory '/root/ocaml-baremetal-riscv/build'
make[2]: Leaving directory '/root/ocaml-baremetal-riscv/build'
[ 78%] Built target freestanding-compat
make[2]: Entering directory '/root/ocaml-baremetal-riscv/build'
make[2]: Leaving directory '/root/ocaml-baremetal-riscv/build'
[ 85%] Built target asmrun_t
make[2]: Entering directory '/root/ocaml-baremetal-riscv/build'
make[2]: Leaving directory '/root/ocaml-baremetal-riscv/build'
[ 92%] Built target nolibc_t
make[2]: Entering directory '/root/ocaml-baremetal-riscv/build'
make[2]: Leaving directory '/root/ocaml-baremetal-riscv/build'
[100%] Built target kernel
make[1]: Leaving directory '/root/ocaml-baremetal-riscv/build'
make: Leaving directory '/root/ocaml-baremetal-riscv/build'
$ file kernel 
kernel: ELF 64-bit LSB executable, UCB RISC-V, version 1 (SYSV), statically linked, with debug_info, not stripped

# Run under spike RISC-V ISA simulator
$ spike kernel
ocaml-boot: heap@0x80042be8 stack@0x8002fbc0
A camel treads on hardware!
ocaml-boot: caml runtime returned. shutting down!

# Run under QEMU
$ qemu-system-riscv64 -machine spike_v1.10 -smp 1 -m 1G -serial stdio -kernel kernel
VNC server running on 127.0.0.1:5900
ocaml-boot: heap@0x80042be8 stack@0x8002fbc0
A camel treads on hardware!
ocaml-boot: caml runtime returned. shutting down!

The immediate next step will be getting the code to run on a Shakti softcore on an FPGA. In addition to targeting high-end FPGAs, we will also be targeting the $100 Arty A7 hobbyist board and release all of the software under liberal open-source licenses.

Further along, we will port mirage libraries to Shakti following similar to the setup in Well-typed lightbulbs and implementing hardware security enhancements in Shakti for preventing spatial and temporal attacks while running unsafe C code (with the ability to dynamically turn it off when running OCaml!), hardware-assisted compartments, etc. Lots of exciting possibilities on the horizon!

Acknowledgements

Much of this work was done by the incredible Malte, who is a visiting student at IIT Madras on a semester away from Leibniz University Hannover, Arjun, Lavanya, Ambika, Chester, and the rest of the Shakti team. The RISC-V port of OCaml is developed and maintained by Nicolás Ojeda Bär.

We are pleased to announce the release of OCamlFormat (available on opam). There have been numerous changes since the last release, so here is a comprehensive list of the new features and breaking changes to help the transition from OCamlFormat 0.8.

Additional dependencies

OCamlFormat now requires:

• ocaml >= 4.06 (up from 4.04.1)
• dune >= 1.1.1
• octavius >= 1.2.0
• uutf

OCamlFormat_Reason now requires:

• ocaml >= 4.06
• dune >= 1.1.1
• ocaml-migrate-parsetree >= 1.0.10 (up from 1.0.6)
• octav…

We are pleased to announce the release of OCamlFormat (available on opam). There have been numerous changes since the last release, so here is a comprehensive list of the new features and breaking changes to help the transition from OCamlFormat 0.8.

Additional dependencies

OCamlFormat now requires:

• ocaml >= 4.06 (up from 4.04.1)
• dune >= 1.1.1
• octavius >= 1.2.0
• uutf

OCamlFormat_Reason now requires:

• ocaml >= 4.06
• dune >= 1.1.1
• ocaml-migrate-parsetree >= 1.0.10 (up from 1.0.6)
• octavius >= 1.2.0
• uutf
• reason >= 3.2.0 (up from 1.13.4)

New preset profiles

The ocamlformat profile aims to take advantage of the strengths of a parsetree-based auto-formatter, and to limit the consequences of the weaknesses imposed by the current implementation. This is a style which optimizes for what the formatter can do best, rather than to match the style of any existing code. General guidelines that have directed the design include:

• Legibility, in the sense of making it as hard as possible for quick visual parsing to give the wrong interpretation, is of highest priority;
• Whenever possible the high-level structure of the code should be obvious by looking only at the left margin, in particular, it should not be necessary to visually jump from left to right hunting for critical keywords, tokens, etc;
• All else equal compact code is preferred as reading without scrolling is easier, so indentation or white space is avoided unless it helps legibility;
• Attention has been given to making some syntactic gotchas visually obvious. ocamlformat is the new default profile.

The conventional profile aims to be as familiar and \"conventional\" appearing as the available options allow.

The default profile is ocamlformat with break-cases=fit. default is deprecated and will be removed in version 0.10.

OCamlFormat diff tool

ocamlformat-diff is a tool that uses OCamlFormat to apply the same formatting to compared OCaml files, so that the formatting differences between the two files are not displayed. Note that ocamlformat-diff comes in a separate opam package and is not included in the ocamlformat package.

The file comparison is then performed by any diff backend.

The options' documentation is available through ocamlformat-diff --help.

The option --diff allows you to configure the diff command that is used to compare the formatted files. The default value is the vanilla diff, but you can also use patdiff or any other similar comparison tool.

ocamlformat-diff can be integrated with git diff, as explained in the online documentation.

Formatting docstrings

Previously, the docstrings (** This is a docstring *) could only be formatted like regular comments, a new option --parse-docstrings has been added so that docstrings can be nicely formatted.

Here is a small example:

(** {1 Printers and escapes used by Cmdliner module} *)

val subst_vars : subst:(string -> string option) -> Buffer.t -> string -> string
(** [subst b ~subst s], using [b], substitutes in [s] variables of the form
    "$(doc)" by their [subst] definition. This leaves escapes and markup
    directives $(markup,...) intact.
    @raise Invalid_argument in case of illegal syntax. *)

Note that this option is disabled by default and you have to set it manually by adding --parse-docstrings to your command line or parse-docstrings=true to your .ocamlformat file. If you get the following error message:

Error: Formatting of (** ... *) is unstable (e.g. parses as a list or not depending on the margin), please tighten up this comment in the source or disable the formatting using the option --no-parse-docstrings.

It means the original docstring cannot be formatted (e.g. because it does not comply with the odoc syntax) and you have to edit it or disable the formatting of docstrings.

Of course if you think your docstring complies with the odoc syntax and there might be a bug in OCamlFormat, feel free to file an issue on github.

Print the configuration

The new --print-config flag prints the configuration determined by the environment variable, the configuration files, preset profiles and command line. Attributes are not considered.

It provides the full list of options with the values they are set to, and the source of this value. For example ocamlformat --print-config prints:

profile=ocamlformat (file .ocamlformat:1)
quiet=false (profile ocamlformat (file .ocamlformat:1))
max-iters=10 (profile ocamlformat (file .ocamlformat:1))
comment-check=true (profile ocamlformat (file .ocamlformat:1))
wrap-fun-args=true (profile ocamlformat (file .ocamlformat:1))
wrap-comments=true (file .ocamlformat:5)
type-decl=compact (profile ocamlformat (file .ocamlformat:1))
space-around-collection-expressions=false (profile ocamlformat (file .ocamlformat:1))
single-case=compact (profile ocamlformat (file .ocamlformat:1))
sequence-style=separator (profile ocamlformat (file .ocamlformat:1))
parse-docstrings=true (file .ocamlformat:4)
parens-tuple-patterns=multi-line-only (profile ocamlformat (file .ocamlformat:1))
parens-tuple=always (profile ocamlformat (file .ocamlformat:1))
parens-ite=false (profile ocamlformat (file .ocamlformat:1))
ocp-indent-compat=false (profile ocamlformat (file .ocamlformat:1))
module-item-spacing=sparse (profile ocamlformat (file .ocamlformat:1))
margin=77 (file .ocamlformat:3)
let-open=preserve (profile ocamlformat (file .ocamlformat:1))
let-binding-spacing=compact (profile ocamlformat (file .ocamlformat:1))
let-and=compact (profile ocamlformat (file .ocamlformat:1))
leading-nested-match-parens=false (profile ocamlformat (file .ocamlformat:1))
infix-precedence=indent (profile ocamlformat (file .ocamlformat:1))
indicate-nested-or-patterns=space (profile ocamlformat (file .ocamlformat:1))
indicate-multiline-delimiters=true (profile ocamlformat (file .ocamlformat:1))
if-then-else=compact (profile ocamlformat (file .ocamlformat:1))
field-space=tight (profile ocamlformat (file .ocamlformat:1))
extension-sugar=preserve (profile ocamlformat (file .ocamlformat:1))
escape-strings=preserve (profile ocamlformat (file .ocamlformat:1))
escape-chars=preserve (profile ocamlformat (file .ocamlformat:1))
doc-comments-tag-only=default (profile ocamlformat (file .ocamlformat:1))
doc-comments-padding=2 (profile ocamlformat (file .ocamlformat:1))
doc-comments=after (profile ocamlformat (file .ocamlformat:1))
disable=false (profile ocamlformat (file .ocamlformat:1))
cases-exp-indent=4 (profile ocamlformat (file .ocamlformat:1))
break-struct=force (profile ocamlformat (file .ocamlformat:1))
break-string-literals=wrap (profile ocamlformat (file .ocamlformat:1))
break-sequences=false (profile ocamlformat (file .ocamlformat:1))
break-separators=before (profile ocamlformat (file .ocamlformat:1))
break-infix-before-func=true (profile ocamlformat (file .ocamlformat:1))
break-infix=wrap (profile ocamlformat (file .ocamlformat:1))
break-fun-decl=wrap (profile ocamlformat (file .ocamlformat:1))
break-collection-expressions=fit-or-vertical (profile ocamlformat (file .ocamlformat:1))
break-cases=fit (file .ocamlformat:2)

If many input files are specified, only print the configuration for the first file. If no input file is specified, print the configuration for the root directory if specified, or for the current working directory otherwise.

Parentheses around if-then-else branches

A new option parens-ite has been added to decide whether to use parentheses around if-then-else branches that spread across multiple lines.

If this option is set, the following function:

let rec loop count a =
  if count >= self#len
  then a
  else
    let a' = f cur#get count a in
    cur#incr ();
    loop (count + 1) a'

will be formatted as:

let rec loop count a =
  if count >= self#len
  then a
  else (
    let a' = f cur#get count a in
    cur#incr ();
    loop (count + 1) a' )

Parentheses around tuple patterns

A new option parens-tuple-patterns has been added, that mimics parens-tuple but only applies to patterns, whereas parens-tuples only applies to expressions. parens-tuple-patterns=multi-line-only mode will try to skip parentheses for single-line tuple patterns, this is the default value. parens-tuple-patterns=always always uses parentheses around tuples patterns.

For example:

(* with parens-tuple-patterns=always *)
let (a, b) = (1, 2)

(* with parens-tuple-patterns=multi-line-only *)
let a, b = (1, 2)

Single-case pattern-matching expressions

The new option single-case defines the style of pattern-matching expressions with only a single case. single-case=compact will try to format a single case on a single line, this is the default value. single-case=sparse will always break the line before a single case.

For example:

(* with single-case=compact *)
try some_irrelevant_expression
with Undefined_recursive_module _ -> true

(* with single-case=sparse *)
try some_irrelevant_expression
with
| Undefined_recursive_module _ -> true

Space around collection expressions

The new option space-around-collection-expressions decides whether to add a space inside the delimiters of collection expressions (lists, arrays, records).

For example:

(* by default *)
type wkind = {f : 'a. 'a tag -> 'a kind}
let l = ["Nil", TCnoarg Thd; "Cons", TCarg (Ttl Thd, tcons)]

(* with space-around-collection-expressions *)
type wkind = { f : 'a. 'a tag -> 'a kind }
let l = [ "Nil", TCnoarg Thd; "Cons", TCarg (Ttl Thd, tcons) ]

Break separators

The new option break-separators decides whether to break before or after separators such as ; in list or record expressions, * in tuples or -> in arrow types. break-separators=before breaks the expressions before the separator, this is the default value. break-separators=after breaks the expressions after the separator. break-separators=after-and-docked breaks the expressions after the separator and docks the brackets for records.

For example:

(* with break-separators=before *)
type t =
  { foooooooooooooooooooooooo: foooooooooooooooooooooooooooooooooooooooo
  ; fooooooooooooooooooooooooooooo: fooooooooooooooooooooooooooo }

(* with break-separators=after *)
type t =
  { foooooooooooooooooooooooo: foooooooooooooooooooooooooooooooooooooooo;
    fooooooooooooooooooooooooooooo: fooooooooooooooooooooooooooo }

(* with break-separators=after-and-docked *)
type t = {
  foooooooooooooooooooooooo: foooooooooooooooooooooooooooooooooooooooo;
  fooooooooooooooooooooooooooooo: fooooooooooooooooooooooooooo
}

Not breaking before bind/map operators

The new option break-infix-before-func decides whether to break infix operators whose right arguments are anonymous functions specially. This option is set by default, if you disable it with --no-break-infix-before-func, it will not break before the operator so that the first line of the function appears docked at the end of line after the operator.

For example:

(* by default *)
f x
>>= fun y ->
g y
>>= fun () ->
f x >>= fun y -> g y >>= fun () -> f x >>= fun y -> g y >>= fun () -> y ()

(* with break-infix-before-func = false *)
f x >>= fun y ->
g y >>= fun () ->
f x >>= fun y -> g y >>= fun () -> f x >>= fun y -> g y >>= fun () -> y ()

Break toplevel cases

There is a new value for the break-cases option: toplevel, that forces top-level cases (i.e. not nested or-patterns) to break across lines, otherwise breaks naturally at the margin.

For example:

let f =
  let g = function
    | H when x y <> k -> 2
    | T | P | U -> 3
  in
  fun x g t h y u ->
    match x with
    | E -> 4
    | Z | P | M -> (
      match y with
      | O -> 5
      | P when h x -> (
          function
          | A -> 6 ) )

Number of spaces before docstrings

The new option doc-comments-padding controls how many spaces are printed before doc comments in type declarations. The default value is 2.

For example:

(* with doc-comments-padding = 2 *)
type t = {a: int  (** a *); b: int  (** b *)}

(* with doc-comments-padding = 1 *)
type t = {a: int (** a *); b: int (** b *)}

Ignore files

An .ocamlformat-ignore file specifies files that OCamlFormat should ignore. Each line in an .ocamlformat-ignore file specifies a filename relative to the directory containing the .ocamlformat-ignore file. Lines starting with # are ignored and can be used as comments.

Here is an example of such .ocamlformat-ignore file:

#This is a comment
dir2/ignore_1.ml

Tag-only docstrings

The new option doc-comments-tag-only controls the position of doc comments only containing tags. doc-comments-tag-only=default means no special treatment is done, this is the default value. doc-comments-tag-only=fit puts doc comments on the same line if it fits.

For example:

(* with doc-comments-tag-only = default *)

(** @deprecated  *)
open Module

(* with doc-comments-tag-only = fit *)

open Module (** @deprecated  *)

Fit or vertical mode for if-then-else

There is a new value for the option if-then-else: fit-or-vertical. fit-or-vertical vertically breaks all branches if they do not fit on a single line. Compared to the compact (default) value, it breaks all branches if at least one of them does not fit on a single line.

For example:

(* with if-then-else = compact *)
let _ =
  if foo then
    let a = 1 in
    let b = 2 in
    a + b
  else if foo then 12
  else 0

(* with if-then-else = fit-or-vertical *)
let _ =
  if foo then
    let a = 1 in
    let b = 2 in
    a + b
  else if foo then
    12
  else
    0

Check mode

A new --check flag has been added. It checks whether the input files already are formatted. This flag is mutually exclusive with --inplace and --output. It returns 0 if the input files are indeed already formatted, or 1 otherwise.

Break function declarations

The new option break-fun-decl controls the style for function declarations and types. break-fun-decl=wrap breaks only if necessary, this is the default value. break-fun-decl=fit-or-vertical vertically breaks arguments if they do not fit on a single line. break-fun-decl=smart is like fit-or-vertical but try to fit arguments on their line if they fit. The wrap-fun-args option now only controls the style for function calls, and no more for function declarations.

For example:

(* with break-fun-decl = wrap *)
let ffffffffffffffffffff aaaaaaaaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbb
    cccccccccccccccccccccc =
  g

(* with break-fun-decl = fit-or-vertical *)
let ffffffffffffffffffff
    aaaaaaaaaaaaaaaaaaaaaa
    bbbbbbbbbbbbbbbbbbbbbb
    cccccccccccccccccccccc =
  g

(* with break-fun-decl = smart *)
let ffffffffffffffffffff
    aaaaaaaaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbb cccccccccccccccccccccc =
  g

Disable configuration in files and attributes

Two new options have been added so that .ocamlformat configuration files and attributes in OCaml files do not change the configuration. These options can be useful if you use some preset profile and you do not want attributes and .ocamlformat files to interfere with your preset configuration. --disable-conf-attrs disables the configuration in attributes, and --disable-conf-files disables .ocamlformat configuration files.

Preserve module items spacing

There is a new value for the option module-item-spacing: preserve, that will not leave open lines between one-liners of similar sorts unless there is an open line in the input.

For example the line breaks are preserved in the following code:

let cmos_rtc_seconds = 0x00
let cmos_rtc_seconds_alarm = 0x01
let cmos_rtc_minutes = 0x02

let x = o

let log_other = 0x000001
let log_cpu = 0x000002
let log_fpu = 0x000004

Breaking changes

• When --disable-outside-detected-project is set, disable ocamlformat when no .ocamlformat file is found.
• Files are not parsed when ocamlformat is disabled.
• Disallow - with other input files.
• The wrap-fun-args option now only controls the style for function calls, and no more for function declarations.
• The default profile is now named ocamlformat.
• The deprecated syntax for .ocamlformat files: option value is no more supported anymore and you should use the option = value syntax instead.

Miscellaneous bugfixes

• Preserve shebang (e.g. #!/usr/bin/env ocaml) at the beginning of a file.
• Improve the formatting when ocp-indent-compat is set.
• UTF8 characters are now correctly printed in comments.
• Add parentheses around a constrained any-pattern (e.g. let (_ : int) = x1).
• Emacs: the temporary buffer is now killed.
• Emacs: add the keybinding in tuareg's map instead of merlin's.
• Lots of improvements on the comments, docstrings, attributes formatting.
• Lots of improvements on the formatting of modules.
• Lots of improvements in the Reason support.
• Do not rely on the file-system to format sources.
• The --debug mode is more user-friendly.

Credits

This release also contains many other changes and bug fixes that we cannot detail here.

Special thanks to our maintainers and contributors for this release: Jules Aguillon, Mathieu Barbin, Josh Berdine, Jérémie Dimino, Hugo Heuzard, Ludwig Pacifici, Guillaume Petiot, Nathan Rebours and Louis Roché.

If you wish to get involved with OCamlFormat development or file an issue, please read the contributing guide, any contribution is welcomed.

MirageOS Security Advisory 01 - memory disclosure in mirage-net-xen

• Module: netchannel
• Announced: 2019-03-21
• Credits: Thomas Leonard, Hannes Mehnert, Mindy Preston
• Affects: netchannel = 1.10.0
• Corrected: 2019-03-20 1.10.1 release

For general information regarding MirageOS Security Advisories, please visit https://mirage.io/security.

Background

MirageOS is a library operating system using cooperative multitasking, which can be executed as a guest of the Xen hypervisor. Virtu…

MirageOS Security Advisory 01 - memory disclosure in mirage-net-xen

• Module: netchannel
• Announced: 2019-03-21
• Credits: Thomas Leonard, Hannes Mehnert, Mindy Preston
• Affects: netchannel = 1.10.0
• Corrected: 2019-03-20 1.10.1 release

For general information regarding MirageOS Security Advisories, please visit https://mirage.io/security.

Background

MirageOS is a library operating system using cooperative multitasking, which can be executed as a guest of the Xen hypervisor. Virtual devices, such as a network device, share memory between MirageOS and the hypervisor. To maintain adequate performance, the virtual device managing network communication between MirageOS and the Xen hypervisor maintains a shared pool of pages and reuses them for write requests.

Problem Description

In version 1.10.0 of netchannel, the API for handling network requests changed to provide higher-level network code with an interface for writing into memory directly. As part of this change, code paths which exposed memory taken from the shared page pool did not ensure that previous data had been cleared from the buffer. This error resulted in memory which the user did not overwrite staying resident in the buffer, and potentially being sent as part of unrelated network communication.

The mirage-tcpip library, which provides interfaces for higher-level operations like IPv4 and TCP header writes, assumes that buffers into which it writes have been zeroed, and therefore may not explicitly write some fields which are always zero. As a result, some packets written with netchannel v1.10.0 which were passed to mirage-tcpip with nonzero data will have incorrect checksums calculated and will be discarded by the receiver.

Impact

This issue discloses memory intended for another recipient and corrupts packets. Only version 1.10.0 of netchannel is affected. Version 1.10.1 fixes this issue.

Version 1.10.0 was available for less than one month and many upstream users had not yet updated their own API calls to use it. In particular, no version of qubes-mirage-firewall or its dependency mirage-nat compatible with version 1.10.0 was released.

Workaround

No workaround is available.

Solution

Transmitting corrupt data and disclosing memory is fixed in version 1.10.1.

The recommended way to upgrade is:

opam update
opam upgrade netchannel

Or, explicitly:

opam upgrade
opam reinstall netchannel=1.10.1

Affected releases (version 1.10.0 of netchannel and mirage-net-xen) have been marked uninstallable in the opam repository.

Correction details

The following list contains the correction revision numbers for each affected branch.

Memory disclosure on transmit:

master: 6c7a13a5dae0f58dcc0653206a73fa3d8174b6d2

1.10.0: bd0382eabe17d0824c8ba854ec935d8a2e5f7489

References

netchannel

You can find the latest version of this advisory online at https://mirage.io/blog/MSA01.

This advisory is signed using OpenPGP, you can verify the signature by downloading our public key from a keyserver (gpg --recv-key 4A732D757C0EDA74), downloading the raw markdown source of this advisory from GitHub and executing gpg --verify 01.txt.asc.

This blog post looks back on some of the improvements in opam 2.0, and gives tips on the new workflows available.

Package development environment management

Opam 2.0 has been vastly improved to handle locally defined packages. Assuming you have a project ~/projects/foo, defining two packages foo-lib and foo-bin, you would have:

~/projects/foo
|-- foo-lib.opam
|-- foo-bin.opam
`-- src/ ...

(See also about computed dependency constraints for handling multiple package definitions with mutual co…

This blog post looks back on some of the improvements in opam 2.0, and gives tips on the new workflows available.

Package development environment management

Opam 2.0 has been vastly improved to handle locally defined packages. Assuming you have a project ~/projects/foo, defining two packages foo-lib and foo-bin, you would have:

~/projects/foo
|-- foo-lib.opam
|-- foo-bin.opam
`-- src/ ...

(See also about computed dependency constraints for handling multiple package definitions with mutual constraints)

Automatic pinning

The underlying mechanism is the same, but this is an interface improvement that replaces most of the opam 1.2 workflows based on opam pin.

The usual commands (install, upgrade, remove, etc.) have been extended to support specifying a directory as argument. So when working on project foo, just write:

cd ~/projects/foo
opam install .

and both foo-lib and foo-bin will get automatically pinned to the current directory (using git if your project is versioned), and installed. You may prefer to use:

opam install . --deps-only

to just get the package dependencies ready before you start hacking on it. See below for details on how to reproduce a build environment more precisely. Note that opam depext . will not work at the moment, which will be fixed in the next release when the external dependency handling is integrated (opam will still list you the proper packages to install for your OS upon failure).

If your project is versioned and you made changes, remember to either commit, or add --working-dir so that your uncommitted changes are taken into account.

Local switches

Opam 2.0 introduced a new feature called "local switches". This section explains what it is about, why, when and how to use them.

Opam switches allow to maintain several separate development environments, each with its own set of packages installed. This is particularly useful when you need different OCaml versions, or for working on projects with different dependency sets.

It can sometimes become tedious, though, to manage, or remember what switch to use with what project. Here is where "local switches" come in handy.

How local switches are handled

A local switch is simply stored inside a _opam/ directory, and will be selected automatically by opam whenever your current directory is below its parent directory.

NOTE: it's highly recommended that you enable the new shell hooks when using local switches. Just run opam init --enable-shell-hook: this will make sure your PATH is always set for the proper switch.

You will otherwise need to keep remembering to run eval $(opam env) every time you cd to a directory containing a local switch. See also how to display the current switch in your prompt

For example, if you have ~/projects/foo/_opam, the switch will be selected whenever in project foo, allowing you to tailor what it has installed for the needs of your project.

If you remove the switch dir, or your whole project, opam will forget about it transparently. Be careful not to move it around, though, as some packages still contain hardcoded paths and don't handle relocation well (we're working on that).

Creating a local switch

This can generally start with:

cd ~/projects/foo
opam switch create . --deps-only

Local switch handles are just their path, instead of a raw name. Additionally, the above will detect package definitions present in ~/projects/foo, pick a compatible version of OCaml (if you didn't explicitely mention any), and automatically install all the local package dependencies.

Without --deps-only, the packages themselves would also get installed in the local switch.

Using an existing switch

If you just want an already existing switch to be selected automatically, without recompiling one for each project, you can use opam switch link:

cd ~/projects/bar
opam switch link 4.07.1

will make sure that switch 4.07.1 is chosen whenever you are in project bar. You could even link to ../foo here, to share foo's local switch between the two projects.

Reproducing build environments

Pinnings

If your package depends on development versions of some dependencies (e.g. you had to push a fix upstream), add to your opam file:

depends: [ "some-package" ] # Remember that pin-depends are depends too
pin-depends: [
  [ "some-package.version" "git+https://gitfoo.com/blob.git#mybranch" ]
]

This will have no effect when your package is published in a repository, but when it gets pinned to its dev version, opam will first make sure to pin some-package to the given URL.

Lock-files

Dependency contraints are sometimes too wide, and you don't want to explore all the versions of your dependencies while developing. For this reason, you may want to reproduce a known-working set of dependencies. If you use:

opam lock .

opam will check what version of the dependencies are installed in your current switch, and explicit them in *.opam.locked files. opam lock is a plugin at the moment, but will get automatically installed when needed.

Then, assuming you checked these files into version control, any user can do

opam install . --deps-only --locked

to instruct opam to reproduce the same build environment (the --locked option is also available to opam switch create, to make things easier).

The generated lock-files will also contain added constraints to reproduce the presence/absence of optional dependencies, and reproduce the appropriate dependency pins using pin-depends. Add the --direct-only option if you don't want to enforce the versions of all recursive dependencies, but only direct ones.

Liquidity version 1.0

We are pleased to announce the release of the first major version of the Liquidity smart-contract language and associated tools.

Some of the highlights of this version are detailed below.

Multiple Entry Points

In the previous versions of Liquidity, smart contracts were limited to a single entry point (named main). But traditionally smart contracts executions path depend strongly on the parameter and in most cases they are completely distinct.

Having different entry points a…

Liquidity version 1.0

We are pleased to announce the release of the first major version of the Liquidity smart-contract language and associated tools.

Some of the highlights of this version are detailed below.

Multiple Entry Points

In the previous versions of Liquidity, smart contracts were limited to a single entry point (named main). But traditionally smart contracts executions path depend strongly on the parameter and in most cases they are completely distinct.

Having different entry points allows to separate code that do not overlap and which usually accomplish vastly different tasks. Encoding entry points with complex pattern matching constructs before was tedious and made the code not extremely readable. This new feature gives you readability and allows to call contracts in a natural way.

Internally, entry points are encoded with sum types and pattern matching so that you keep the strong typing guarantees that come over from Michelson. This means that you cannot call a typed smart contract with the wrong entry point or the wrong parameter (this is enforced statically by both the Liquidity typechecker and the Michelson typechecker).

Modules and Contract System

Organizing, encapsulating and sharing code is not always easy when you need to write thousand lines files. Liquidity now allows to write modules (which contain types and values/functions) and contracts (which define entry points in addition). Types and non-private values of contracts and modules in scope can be accessed by other modules and contracts.

You can even compile several files at once with the command line compiler, so that you may organize your multiple smart contract projects in libraries and files.

Polymorphism and Type Inference

Thanks to a new and powerful type inference algorithm, you can now get rid of almost all type annotations in the smart contracts.

Instead of writing something like

let%entry main (parameter : bool) (storage : int) =
  let ops = ([] : operation list) in
  let f (c : bool) = if not c then 1 else 2 in
  ops, f parameter

you can now write

let%entry main parameter _ =
  let ops = [] in
  let f c = if not c then 1 else 2 in
  ops, f parameter

And type inference works with polymorhpism (also a new feature of this release) so you can now write generic and reusable functions:

type 'a t = { x : 'a set; y : 'a }

let mem_t v = Set.mem v.y v.x

Inference also works with contract types and entry points.

ReasonML Syntax

We originally used a modified version of the OCaml syntax for the Liquidity language. This made the language accessible, almost for free, to all OCaml and functional language developers. The typing discipline one needs is quite similar to other strongly typed functional languages so this was a natural fit.

However this is not the best fit for everyone. We want to bring the power of Liquidity and Tezos to the masses so adopting a seemingly familiar syntax for most people can help a lot. With this new version of Liquidity, you can now write your smart contracts in both an OCaml-like syntax or a ReasonML-like one. The latter being a lot closer to Javascript on the surface, making it accessible to people that already know the language or people that write smart contracts for other platforms like Solidity/Ethereum.

You can see the full changelog as well as download the latest release and binaries at this address.

Don’t forget that you can also try all these new cool features and more directly in your browser with our online editor.

MirageOS 3.5.0 release

We are happy to announce our MirageOS 3.5.0 release. We didn't announce post 3.0.0 releases too well -- that's why this post tries to summarize the changes in the MirageOS ecosystem over the past two years. MirageOS consists of over 100 opam packages, lots of which are reused in other OCaml projects and deployments without MirageOS. These opam packages are maintained and developed further by lots of developers.

On the OCaml tooling side, since MirageOS 3.0.0 we did severa…

MirageOS 3.5.0 release

We are happy to announce our MirageOS 3.5.0 release. We didn't announce post 3.0.0 releases too well -- that's why this post tries to summarize the changes in the MirageOS ecosystem over the past two years. MirageOS consists of over 100 opam packages, lots of which are reused in other OCaml projects and deployments without MirageOS. These opam packages are maintained and developed further by lots of developers.

On the OCaml tooling side, since MirageOS 3.0.0 we did several major changes:

• moved most packages to dune (formerly jbuilder) and began using dune-release for smooth developer experience and simple releases,
• require opam to be version 2.0.2 or later, allowing pin-depends in config.ml. pin-depends allows you to depend on a development branch of any opam package for your unikernel,
• adjusted documentation to adhere to odoc requirements,
• the mirage command-line utility now emits lower and upper bounds of opam packages, allowing uncompromising deprecation of packages,
• support for OCaml 4.06.0 (and above), where safe-string is enabled by default. Strings are immutable now!!,
• remove usage of result package, which has incorporated into Pervasives since OCaml 4.03.0.

The 3.5.0 release contains several API improvements of different MirageOS interfaces - if you're developing your own MirageOS unikernels, you may want to read this post to adjust to the new APIs.

MirageOS interface API changes:

• mirage-clock has the type t constrained to unit as of 2.0.0;
• mirage-protocols renames the ETHIF module type to the clearer ETHERNET. As of 2.0.0 it also contains keep-alive support, complies with recent TCP/IP layering rework (see below), and IPv4 now supports reassembly and fragmentation;
• mirage-net reflects revised layering API as of 2.0.0 (see below);
• mirage-kv has a revised API and introduction of a read-write key-value store (see below).

Major changes

Key-value store

We improved the key-value store API, and added a read-write store. There is also ongoing work which implements the read-write interface using irmin, a branchable persistent storage that can communicate via the git protocol. Motivations for these changes were the development of CalDAV, but also the development of wodan, a flash-friendly, safe and flexible filesystem. The goal is to EOL the mirage-fs interface in favour of the key-value store.

Major API improvements (in this PR, since 2.0.0):

• The key is now a path (list of segments) instead of a string
• The value type is now a string
• The new function list : t -> key -> (string * [Value|Dictionary], error) result io was added
• The function get : t -> key -> (value, error) result io is now provided (used to be named read and requiring an offset and length parameter)
• The functions last_modified : t -> key -> (int * int64, error) result io and digest : t -> key -> (string, error) result io have been introduced
• The function size was removed.
• The signature RW for read-write key-value stores extends RO with three functions set, remove, and batch

There is now a non-persistent in-memory implementation of a read-write key-value store available. Other implementations (such as crunch, mirage-kv-unix, mirage-fs, tar have been adapted, as well as clients of mirage-kv (dns, cohttp, tls)).

TCP/IP

The IPv4 implementation now has support for fragment reassembly. Each incoming IPv4 fragment is checked for the "more fragments" and "offset" fields. If these are non-zero, the fragment is processed by the fragment cache, which uses a least recently used data structure of maximum size 256kB content shared by all incoming fragments. If there is any overlap in fragments, the entire packet is dropped (avoiding security issues). Fragments may arrive out of order. The code is heavily unit-tested. Each IPv4 packet may at most be in 16 fragments (to minimise CPU DoS with lots of small fragments), the timeout between the first and last fragment is 10 seconds.

The layering and allocation discipline has been revised. ethernet (now encapsulating and decapsulating Ethernet) and arp (the address resolution protocol) are separate opam packages, and no longer part of tcpip.

At the lowest layer, mirage-net is the network device. This interface is implemented by our different backends (xen, solo5, unix, macos, and vnetif). Some backends require buffers to be page-aligned when they are passed to the host system. This was previously not really ensured: while the abstract type page_aligned_buffer was required, write (and writev) took the abstract buffer type (always constrained to Cstruct.t by mirage-net-lwt). The mtu (maximum transmission unit) used to be an optional connect argument to the Ethernet layer, but now it is a function which needs to be provided by mirage-net.

The Mirage_net.write function now has a signature that is explicit about ownership and lifetime: val write : t -> size:int -> (buffer -> int) -> (unit, error) result io. It requires a requested size argument to be passed, and a fill function which is called with an allocated buffer, that satisfies the backend demands. The fill function is supposed to write to the buffer, and return the length of the frame to be send out. It can neither error (who should handle such an error anyways?), nor is it in the IO monad. The fill function should not save any references to the buffer, since this is the network device's memory, and may be reused. The writev function has been removed.

The Ethernet layer does encapsulation and decapsulation now. Its write function has the following signature: val write: t -> ?src:macaddr -> macaddr -> Ethernet.proto -> ?size:int -> (buffer -> int) -> (unit, error) result io. It fills in the Ethernet header with the given source address (defaults to the device's own MAC address) and destination address, and Ethernet protocol. The size argument is optional, and defaults to the MTU. The buffer that is passed to the fill function is usable from offset 0 on. The Ethernet header is not visible at higher layers.

The IP layer also embeds a revised write signature: val write: t -> ?fragment:bool -> ?ttl:int -> ?src:ipaddr -> ipaddr -> Ip.proto -> ?size:int -> (buffer -> int) -> buffer list -> (unit, error) result io. This is similar to the Ethernet signature - it writes the IPv4 header and sends a packet. It also supports fragmentation (including setting the do-not-fragment bit for path MTU discovery) -- whenever the payload is too big for a single frame, it is sent as multiple fragmented IPv4 packets. Additionally, setting the time-to-live is now supported, meaning we now can implement traceroute! The API used to include two functions, allocate_frame and write, where only buffers allocated by the former should be used in the latter. This has been combined into a single function that takes a fill function and a list of payloads. This change is for maximum flexibility: a higher layer can either construct its header and payload, and pass it to write as payload argument (the buffer list), which is then copied into the buffer(s) allocated by the network device, or the upper layer can provide the callback fill function to assemble its data into the buffer allocated by the network device, to avoid copying. Of course, both can be used - the outgoing packet contains the IPv4 header, and possibly the buffer until the offset returned by fill, and afterwards the payload.

The TCP implementation has preliminary keepalive support.

Solo5

• MirageOS 3.0.0 used the 0.2.0 release of solo5
• The ukvm target was renamed to hvt, where solo5-hvt is the monitoring process
• Support for FreeBSD bhyve and OpenBSD VMM hypervisor (within the hvt target)
• Support for ARM64 and KVM
• New target muen.sk, a separation kernel developed in SPARK/Ada
• New target GenodeOS, an operating system framework using a microkernel
• Debugger support: attach gdb in the host system for improved debugging experience
• Core dump support
• Drop privileges on OpenBSD and FreeBSD
• Block device write fixes (in mirage-block-solo5)

random

The default random device from the OCaml standard library is now properly seeded using mirage-entropy. In the future, we plan to make the fortuna RNG the default random number generator.

Argument passing to unikernels

The semantics of arguments passed to a MirageOS unikernel used to vary between different backends, now they're the same everywhere: all arguments are concatenated using the whitespace character as separator, and split on the whitespace character again by parse-argv. To pass a whitespace character in an argument, the whitespace now needs to be escaped: --hello=foo\ bar.

Noteworthy package updates

• cstruct 3.6.0 API changes and repackaging, see this announcement and this announcement
• ipaddr 3.0.0 major API changes, the s-expression serialisation is a separate subpackage, macaddr is now a standalone opam package
• base64 3.0.0 performance and API changes, see this announcement
• git 2.0.0, read this announcement, as well as its design and implementation
• io-page 2.0.0, see this announcement
• cohttp 2.0.0, see this announcement
• dns 1.0.0, see this announcement
• conduit 1.0.0, see this announcement

More features and bugfixes

• More HTTP server choices are supported via a new httpaf device that permits the high performance httpaf stack to run as a unikernel now.
• libvirt.xml is generated for virtio target
• Unix target now include -tags thread (for mirage-framebuffer SDL support)
• Various modules (IPv6, DHCP) are explicit about their dependency to the random device
• QubesDB can be requested in config.ml when the target is Xen

You may also want to read the MirageOS 3.2.0 announcement and the MirageOS 3.3.0 announcement.

Next steps

We are working on further changes which revise the mirage internal build system to dune. At the moment it uses ocamlbuild, ocamlfind, pkg-config, and make. The goal of this change is to make MirageOS more developer-friendly. On the horizon we have MirageOS unikernel monorepos, incremental builds, pain-free cross-compilation, documentation generation, ...

Several other MirageOS ecosystem improvements are on the schedule for 2019, including an irmin 2.0 release, a seccomp target for Solo5, and easier deployment and multiple interface in Solo5.

We are pleased to announce the first release of Techelson, available here.

Techelson is a Test Execution Engine for Michelson. It aims at testing functional properties of Michelson smart contracts. Make sure to check the user documentation to get a sense of Techelson’s workflow and features.

For Liquidity programmers interested in Techelson, take a look at this blog post discussing how to write tests in Liquidity and run them using Techelson.

Techelson is still young: if you have problems…

We are pleased to announce the first release of Techelson, available here.

Techelson is a Test Execution Engine for Michelson. It aims at testing functional properties of Michelson smart contracts. Make sure to check the user documentation to get a sense of Techelson’s workflow and features.

For Liquidity programmers interested in Techelson, take a look at this blog post discussing how to write tests in Liquidity and run them using Techelson.

Techelson is still young: if you have problems, suggestions or feature requests please open an issue on the repository.

Smart contracts calls already provide a built-in authentication mechanism as transactions (i.e. call operations) are cryptographically signed by the sender of the transaction. This is a guarantee on which programs can rely.

However, sometimes you may want more involved or flexible authentication schemes. The ones that rely on signature validity checking can be implemented in Michelson, and Liquidity provide a built-in instruction to do so. (You still need to keep in mind that you cannot store un…

Smart contracts calls already provide a built-in authentication mechanism as transactions (i.e. call operations) are cryptographically signed by the sender of the transaction. This is a guarantee on which programs can rely.

However, sometimes you may want more involved or flexible authentication schemes. The ones that rely on signature validity checking can be implemented in Michelson, and Liquidity provide a built-in instruction to do so. (You still need to keep in mind that you cannot store unencrypted confidential information on the blockchain).

This instruction is Crypto.check in Liquidity. Its type can be written as:

Crypto.check: key -> signature -> bytes -> bool

Which means that it takes as arguments a public key, a signature and a sequence of bytes and returns a Boolean. Crypto.check pub_key signature message is true if and only if the signature signature was obtained by signing the Blake2b hash of message using the private key corresponding to the public key pub_key.

A small smart contract snippet which implements a signature check (against a predefined public key kept in the smart contract’s storage) can be tested online here.

type storage = key

let%entry main ((message : string), (signature : signature)) key =
  let bytes = Bytes.pack message in
  if not (Crypto.check key signature bytes) then
    failwith "Wrong signature";
  ([] : operation list), key

This smart contract fails if the string message was not signed with the private key corresponding to the public key key stored. Otherwise it does nothing.

This signature scheme is more flexible than the default transaction/sender one, however it requires that the signature can be built outside of the smart contract. (And more generally outside of the toolset provided by Liquidity and Tezos). On the other hand, signing a transaction is something you get for free if you use the tezos client or any tezos wallet (as is it essentially their base function).

The rest of this blog post will focus on various ways to sign data, and on getting signatures that can be used in Tezos and Liquidity directly.

Signing Using the Tezos Client

One (straightforward) way to sign data is to use the Tezos client directly. You will need to be connected to a Tezos node though as the client makes RPCs to serialize data (this operation is protocol dependent). We can only sign sequences of bytes, so the first thing we need to do is to serialize whichever data we want to sign. This can be done with the command hash data of the client.

> ./tezos-client -A alphanet-node.tzscan.io -P 80 hash data '"message"' of type string
Raw packed data: 0x0501000000076d657373616765
Hash: exprtXaZciTDGatZkoFEjE1GWPqbJ7FtqAWmmH36doxBreKr6ADcYs
Raw Blake2b hash: 0x01978930fd2d04d0db8c2e4ef8a3f5d63b8e732177c8723135ed0dc7d99ebed3
Raw Sha256 hash: 0x32569319f6517036949bcead23a761bfbfcbf4277b010355884a86ba09349839
Raw Sha512 hash: 0xdfa4ea9f77db3a98654f101be1d33d56898df40acf7c2950ca6f742140668a67fefbefb22b592344922e1f66c381fa2bec48aa47970025c7e61e35d939ae3ca0
Gas remaining: 399918 units remaining

This command gives the result of hashing the data using various algorithms but what we’re really interested in is the first item Raw packed data which is the serialized version of our data ("message") : 0x0501000000076d657373616765.

We can now sign these bytes using the Tezos client as well. This step can be performed completely offline, for that we need to use the option -p of the client to specify the protocol we want to use (the sign bytes command will not be available without first selecting a valid protocol). Here we use protocol 3, designated by its hash PsddFKi3.

> ./tezos-client -p PsddFKi3 sign bytes 0x0501000000076d657373616765 for my_account
Signature: edsigto9QHtXMyxFPyvaffRfFCrifkw2n5ZWqMxhGRzieksTo8AQAFgUjx7WRwqGPh4rXTBGGLpdmhskAaEauMrtM82T3tuxoi8

The account my_account can be any imported account in the Tezos client. In particular, it can be an encrypted key pair (you will need to enter a password to sign) or a hardware Ledger (you will need to confirm the signature on the Ledger). The obtained signature can be used as is with Liquidity or Michelson. This one starts with edsig because it was obtained using an Ed25519 private key, but you can also get signatures starting with spsig1 or p2sig depending on the cryptographic curve that you use.

Signing Manually

In this second section we detail the necessary steps and provide a Python script to sign string messages using an Ed25519 private key. This can be easily adapted for other signing schemes.

These are the steps that will need to be performed in order to sign a string:

1. Assuming that the value you want to sign is a string, you first need to convert its ASCII version to hexa, for the string "message" that is 6d657373616765.
2. You need to produce the packed version of the corresponding Michelson expression. The binary representation can vary depending on the types of the values you want to pack but for strings it is:

   | 0x | 0501 | [size of the string on 4 bytes] | [ascii string in hexa] |

   for "message" (of length 7), it is

   | 0x | 0501 | 00000007 | 6d657373616765 |

   or 0x0501000000076d657373616765.

3. Hash this value using Blake2b (01978930fd2d04d0db8c2e4ef8a3f5d63b8e732177c8723135ed0dc7d99ebed3) which is 32 bytes long.
4. Depending on your public key, you then need to sign it with the corresponding curve (ed25519 for edpk keys), the signature is 64 bytes:

   753e013b8515a7d47eaa5424de5efa2f56620ac8be29d08a6952ae414256eac44b8db71f74600275662c8b0c226f3280e9d24e70a5fa83015636b98059b5180c

5. Optionally convert to base58check. This is not needed because Liquidity and Michelson allow signatures (as well as keys and key hashes) to be given in hex format with a 0x:

   0x753e013b8515a7d47eaa5424de5efa2f56620ac8be29d08a6952ae414256eac44b8db71f74600275662c8b0c226f3280e9d24e70a5fa83015636b98059b5180c

The following Python (3) script will do exactly this, entirely offline. Note that this is just an toy example, and should not be used in production. In particular you need to give your private key on the command line so this might not be secure if the machine you run this on is not secure.

> pip3 install base58check pyblake2 ed25519
> python3 ./sign_string.py "message" edsk2gL9deG8idefWJJWNNtKXeszWR4FrEdNFM5622t1PkzH66oH3r
0x753e013b8515a7d47eaa5424de5efa2f56620ac8be29d08a6952ae414256eac44b8db71f74600275662c8b0c226f3280e9d24e70a5fa83015636b98059b5180c

sign_string.py

from pyblake2 import blake2b
import base58check
import ed25519
import sys

message = sys.argv[1]
seed_b58 = sys.argv[2]

prefix = b'\x05\x01'
len_bytes = (len(message)).to_bytes(4, byteorder='big')
h = blake2b(digest_size=32)
b = bytearray()
b.extend(message.encode())
h.update(prefix + len_bytes + b)
digest = h.digest()

seed = base58check.b58decode(seed_b58)[4:-4]
sk = ed25519.SigningKey(seed)
sig = sk.sign(digest)
print("0x" + sig.hex())