Monorobot: a notification bot for monorepos

A few years ago, we decided to move most of our code into a monorepo. Many advocates have highlighted its upsides, which include better cross-project coordination and simpler dependency management.

But one problem remained: none of the available GitHub integrations for Slack work nicely with monorepos. Slack is vital for day to day communication among Ahrefs’ globally distrib…

Monorobot: a notification bot for monorepos

A few years ago, we decided to move most of our code into a monorepo. Many advocates have highlighted its upsides, which include better cross-project coordination and simpler dependency management.

But one problem remained: none of the available GitHub integrations for Slack work nicely with monorepos. Slack is vital for day to day communication among Ahrefs’ globally distributed team, so a Slack integration was a must-have feature. We needed a service that could map activity from our various subprojects to their corresponding Slack channels — something existing solutions didn’t offer.

That’s why we built our own integration: Monorobot, a notification bot for monorepos. We’ve improved it iteratively since the monorepo transition, incorporating real time feedback from our engineers over time. Today, Monorobot is an active member of Ahrefs’ Slack workspace, dutifully routing GitHub activity notifications to different channels based on the relevance of each activity.

And now we’re open-sourcing Monorobot, for anybody to use in their monorepo setup! The package is available via OPAM:

opam install monorobot

Read on for more details about the motivation, an overview of the main features, and what’s in the pipeline.

Existing Slack integrations lack monorepo support

Within a monorepo, multiple projects have their code located in separate, nested directories. Correspondingly, each project’s Slack channel is only interested in activity from that part of the overall repository. The issue with most GitHub-to-Slack integrations is that once you subscribe a Slack channel to a GitHub repository, the channel receives all activity from that repository.

Suppose we operate various camel-related services, and we’re planning to launch a new camel ride-sharing app called Camel Ride. The directory structure could look like this:

monorepo/
| frontend/
| | camelride_ui/
| | | mobile/
| | | web/
| | cameldance/
| backend/
| | camelride/
| | | routing/
| | | pricing/
| | camelfood/

As you can see, both the frontend/ and backend/ directories contain code for our fictitious ride-sharing service, along with code from other projects.

If we were to connect the GitHub for Slack integration to this repository, notifications for activity from all projects would be sent to the same channel. Even if I were only interested in activity from the Camel Ride project, I’d need to sift through notifications from the other, unrelated projects. Imagine the volume of notifications this would create for a larger monorepo with dozens of projects. What a mess!

Enter Monorobot

Monorobot enables more granular control over where notifications from the same repository get routed, depending on the type of activity. This routing behavior can be defined in a configuration file named .monorobot.json, which should be committed to the root of the monorepo. Once you create a GitHub webhook from the repository to a running instance of Monorobot, it will use the configuration file to route notifications to relevant channels based on the webhook event payload:

• For pushed commits, it checks the path prefixes of the files modified in the commits.
• For activity related to PRs and issues, it checks their labels.
• For status updates on pushed commits (e.g., CI builds), it uses the same path prefix logic as pushed commits.

Additionally, Monorobot supports unfurling GitHub links shared in Slack.

Path prefix routing for commit push notifications

Continuing with our example, suppose we want to route all commit activity related to the Camel Ride project to a Slack channel called #camelride. Our configuration file might look like this:

{
  ...,
  "prefix_rules": {
    "rules": [
      {
        "match": [
          "frontend/camelride_ui/",
          "backend/camelride/"
        ],
        "ignore": [
          "frontend/camelride_ui/images",
        ],
        "channel": "camelride"
      }
    ]
  }
}

Each rule “matches” a file path to a channel. Whenever someone pushes a commit touching files with either of these prefixes, the #camelride channel will be notified. All other commits will be ignored.

If a file prefix appears in a rule’s optional ignore field, the rule won't be matched even if the prefix is is also in the match field. In the above snippet, the Camel Ride frontend team has decided to silence notifications for activity in the images/ subdirectory.

Now, let’s say the project’s price optimization team is growing, and they’ve decided to create their own separate Slack channel called #camelride-pricing. We can simply commit an update to the .monorobot.json file, and Monorobot will detect the configuration change:

{
  ...,
  "prefix_rules": {
    "rules": [
      {
        "match": [
          "frontend/camelride_ui/",
          "backend/camelride/"
        ],
        "ignore": [
          "frontend/camelride_ui/images",
        ],
        "channel": "camelride"
      },
      {
        "match": [
          "backend/camelride/pricing/"
        ],
        "channel": "camelride-pricing"
      }
    ]
  }
}

Since Monorobot will match the rule with the longest matched prefix, only commits related to the price optimization aspect of Camel Ride will notify #camelride-pricing, and all other general Camel Ride commits will notify #camelride.

There are additional configuration options for prefix rules (and for label rules discussed in the next section) that aren’t mentioned here. Visit the repository for the full details.

Label-based routing for PRs and issue notifications

For activity related to pull requests and issues (opening, closing, merging, commenting, and reviewing), Monorobot uses labels to determine routing. The format is largely the same as for path prefix routing:

{
  ...,
  "label_rules": {
    "default_channel": "notifications",
    "rules": [
      {
        "match": [
          "Camel Ride"
        ],
        "channel": "camelride"
      },
      {
        "match": [
          "Price Optimization"
        ],
        "channel": "camelride-pricing"
      }
    ]
  }
}

Here, all PRs and issues with the “Camel Ride” label will have activity sent to #camelride; those with the “Price Optimization” label to #camelride-pricing; and those with both labels to both channels.

The default_channel field provides an option to fall back on a channel if no rule is matched; this option is available for prefix rules as well.

Status notifications

Monorobot also supports build status notifications for CI pipelines. When it receives a status update for a pushed commit, it routes it to the relevant channel(s) by applying the prefix rules to the commit associated with the build. Further filtering based on status (e.g., ignoring canceled builds, and only notifying for a successful build when preceded by a failed one) is also possible.

Link unfurling

Finally, Monorobot can unfurl links to GitHub repositories shared on Slack (including private ones, if a personal access token is provided). This applies to commit, issue, and pull request URLs.

What’s next

Monorobot is actively used at Ahrefs today, but there are lots of promising future directions it could take. Here, we list a few.

Unifying GitHub and Slack identities

It would be useful to allow GitHub user IDs to be mapped to Slack ones. This would enable more personalized features for Monorobot, such as direct messaging a user when their review is requested or when a CI build fails on a feature branch they authored.

Consolidating notifications

Sometimes, a collection of multiple GitHub webhook events makes sense to be grouped and delivered as a single Slack notification. For example, pull request reviews generate discrete webhook events for each review comment, but it would make more sense to pool them together, so as not to spam a channel with many notifications.

Better status notifications

A CI build failure can have multiple potential causes:

1. A bad commit
2. A previous bad commit that has yet to be fixed
3. An issue with the pipeline itself (this is out of scope for Monorobot)

It can be quite tricky to discern between the first two causes from the GitHub webhook event alone. Cause 1 is handled well by our current approach of using path prefix routing on the commit associated with the build. But with cause 2, that same approach doesn’t always send the build failure notification to the channel where it is actually relevant. In that case, the originator of the initial bad commit won’t be nagged about all subsequent failures, and Slack channels with no relevance to the cause of failure will get polluted with unnecessary notifications.

How can we best determine whether a status notification is “relevant” to a Slack channel? This is still an open question, but one possible direction is to track build state per build step rather than per status, and route notifications based on that. For example, if an overall build fails due to a backend build step failure, then it could be sent to a channel where the frontend team won’t be notified.

Wrapping up

The overall goal of Monorobot is to make Slack notifications more relevant for all teams in a large monorepo environment, using the information available from GitHub webhook events. We’ve had fairly positive results with our own internal usage, and now we hope others find it useful as well.

Monorobot is written in OCaml. We welcome your feedback and contributions on GitHub!

P.S. If anyone does make an actual ride sharing service for camels, do let us know…

Thanks to Feihong, Igor, and Louis for feedback on this post.

Monorobot: a Slack bot for monorepos was originally published in ahrefs on Medium, where people are continuing the conversation by highlighting and responding to this story.

We are pleased to announce the minor release of opam 2.1.2.

This opam release consists of backported fixes, including:

• Fallback on dnf if yum does not exist on RHEL-based systems (#4825)
• Use --no-depexts in CLI 2.0 mode. This further improves the use of opam 2.1 as a drop-in replacement for opam 2.0 in CI, for example with setup-ocaml in GitHub Actions. (#4908)

1. From binaries: run

   bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/oc…

We are pleased to announce the minor release of opam 2.1.2.

This opam release consists of backported fixes, including:

• Fallback on dnf if yum does not exist on RHEL-based systems (#4825)
• Use --no-depexts in CLI 2.0 mode. This further improves the use of opam 2.1 as a drop-in replacement for opam 2.0 in CI, for example with setup-ocaml in GitHub Actions. (#4908)

1. From binaries: run

   bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.2"

   or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.

2. From source, using opam:

   opam update; opam install opam-devel

   (then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni to enable sandboxing if you had version 2.0.0~rc manually installed or to update your sandbox script)

3. From source, manually: see the instructions in the README.

We hope you enjoy this new minor version, and remain open to bug reports and suggestions.

November has become MirageOS month! Between the upcoming official MirageOS 4.0 release, making custom Christmas Tree garlands with MirageOS on a Raspberry Pi, and now this "What is an Operating System?" podcast (featuring Tarides advisor and core MirageOS maintainer Anil Madhavapeddy), it truly is MirageOS month!

MirageOS can do much more than program a Raspberry Pi for Christmas decor. From agricultural monitoring to smart buildings, its applications cover a wide range of needs. For example, it…

November has become MirageOS month! Between the upcoming official MirageOS 4.0 release, making custom Christmas Tree garlands with MirageOS on a Raspberry Pi, and now this "What is an Operating System?" podcast (featuring Tarides advisor and core MirageOS maintainer Anil Madhavapeddy), it truly is MirageOS month!

MirageOS can do much more than program a Raspberry Pi for Christmas decor. From agricultural monitoring to smart buildings, its applications cover a wide range of needs. For example, it can also be used in critical pieces of infrastructure where security is of paramount importance, such as the Tezos blockchain. Combined with Irmin, it allows developers to build secure-by-design, offline-first systems and invert the current cloud-centric model for designing applications to securely connect physical spaces with extremely low latency and high bandwidth, using local-area computation capabilities.

You can read the entire transcript here and find links to multiple places to listen through podcast apps.

Proof assistant communities have grown quite a bit lately. They have active Zulip chats: Lean, Coq, Agda, Isabelle. These are good for discussions, but less so for knowledge accumulation and organization, and are not indexed by the search engines.

I have therefore created a proposal for a new “Proof assistants” StackExchange site. I believe that such a site would complement very well various Zulips dedicated to specific proof assistants. If you favor the idea, please support it by visiting …

Proof assistant communities have grown quite a bit lately. They have active Zulip chats: Lean, Coq, Agda, Isabelle. These are good for discussions, but less so for knowledge accumulation and organization, and are not indexed by the search engines.

I have therefore created a proposal for a new “Proof assistants” StackExchange site. I believe that such a site would complement very well various Zulips dedicated to specific proof assistants. If you favor the idea, please support it by visiting the proposal and

• becoming a follower (you have to be a registered user with a verified email account),
• asking sample questions, and
• upvoting good sample questions.

To pass the first stage, we need 60 followers and 40 questions with at least 10 votes to proceed to the next stage.

We are thrilled to announce a partnership between Tarides and Hyper, a technology provider in the agritech space who’s building an "operating system for high-performing farms." Indoor and vertical farms are becoming tech businesses that require scalable, flexible, and easy-to-use tools to facilitate data analysis and thereby increase productivity. According to the State of Indoor Farming 2020 Report, “40% of vertical and indoor farms are implementing data analytics and control automation to …

We are thrilled to announce a partnership between Tarides and Hyper, a technology provider in the agritech space who’s building an "operating system for high-performing farms." Indoor and vertical farms are becoming tech businesses that require scalable, flexible, and easy-to-use tools to facilitate data analysis and thereby increase productivity. According to the State of Indoor Farming 2020 Report, “40% of vertical and indoor farms are implementing data analytics and control automation to increase yield and lower cost of production.”

Hyper’s product offers a developer-friendly platform for modern farms to integrate analytics and automation without worrying about hardware, scaling, or maintenance. There is a natural synergy between Hyper's product and Tarides’s mission to bring robust and scalable functional systems to the industry. Both teams will be working on technology to help solve some big problems the world faces.

First, some context about how agriculture affects our environment:
The world’s population is growing, but the size of our planet is not.

Agriculture is currently responsible for:

• 75% of the world’s deforestation¹
• 50% of the world’s habitable land
• 70% of the world's freshwater withdrawals
• 26% of the greenhouse gas emissions²

It’s an important time for indoor farming, as it’s already producing twenty times (20x) more yield per area while using 95% less water and zero (0) pesticides. Plus, indoor farming will reduce the carbon footprint by cutting their food miles in half (50%)³. Vertical farming complements traditional farming by growing fresh produce for cities in a sustainable way.

Hyper's mission is to simplify the integration of sensors and controllers for data collection and automation. Their roadmap is focused on implementing real-time computation of metrics for environmental data gathered from farms, prototyping computer vision models for crop analysis, and automated crop traceability infrastructure. With their data platform, growers can optimise yield and reduce operating costs by getting access to crop growth metrics and climate automation profiles without a dedicated engineering team.

Hyper's founders are experienced engineers and OCaml hackers who have worked on IoT and data analytics products in the past in the retail, biotech, and multimedia industries. Utilizing our MirageOS ecosystem, Hyper's sensor networks and cameras are continuously collecting millions of data points across large-scale farming operations to ensure consistent crop quality, detect issues early, and automate climate control. Since the technology is offline-first, it enables Hyper to collect this data securely, without the risk of breach or loss, as is sometimes the case in cloud computing.

As a technological partner, Tarides will help Hyper build a scalable IoT platform that leverages the OCaml ecosystem. Our support will help Hyper bring the product to the market faster while also contributing to the open-source IoT ecosystem for MirageOS and related projects. Hyper’s IoT platform will be fully open-source next year and will focus on implementing better support for MQTT, CoAP, and other protocols.

A new version of MirageOS will be released in November, so this exciting announcement couldn't come at a better time! Hyper’s use of the MirageOS ecosystem to build its data intelligence product is yet another real-world example that displays the power and efficiency of MirageOS.

It’s truly exciting technology because we’re at a point in history where even the most adamant climate change critics have begun to admit the climate is indeed changing. There are things we can each do to contribute, so Tarides is proud to collaborate with a company that’s actively seeking solutions to help reduce the amount of greenhouse gas emissions while increasing productivity.

Hyper currently has production deployments in vertical and indoor farms in the UK and East Africa and is planning on scaling the operations in the coming months.

SOURCES

Feedback on this post is welcomed on Discuss!

We are pleased to announce several minor releases: opam 2.0.10, opam 2.1.1, and opam-depext 1.2.

The opam releases consist of backported fixes, while opam-depext has been adapted to be compatible with opam 2.1, to allow for workflows which need to maintain compatibility with opam 2.0. With opam 2.1.1, if you export OPAMCLI=2.0 into your environment then workflows expecting opam 2.0 should now behave even more equivalently.

opam-depext 1.2

Previous…

Feedback on this post is welcomed on Discuss!

We are pleased to announce several minor releases: opam 2.0.10, opam 2.1.1, and opam-depext 1.2.

The opam releases consist of backported fixes, while opam-depext has been adapted to be compatible with opam 2.1, to allow for workflows which need to maintain compatibility with opam 2.0. With opam 2.1.1, if you export OPAMCLI=2.0 into your environment then workflows expecting opam 2.0 should now behave even more equivalently.

opam-depext 1.2

Previous versions of opam-depext were made unavailable when using opam 2.1, since depext handling is done directly by opam 2.1 and later. This is still the recommended way, but this left workflows which wanted to maintain compatibility with opam 2.0 without a single command to install depexts. You can now run OPAMCLI=2.0 opam depext -y package1 package2 and expect this to work correctly with any version of opam 2. If you don't specify OPAMCLI=2.0 then the plugin will remind you that you should be using the integrated depext support! Calling opam depext this way with opam 2.1 and later still exposes the same double-solving problem that opam 2.0 has, but if for some reason the solver returns a different solution at opam install then the correct depexts would be installed.

For opam 2.0, some useful depext handling changes are back-ported from opam 2.1.x to the plugin: With opam 2.0:

• yum-based distributions: force not to upgrade (#137)
• Archlinux: always upgrade all the installed packages when installing a new package (#138)

opam 2.1.1

opam 2.1.1 includes both the fixes in opam 2.0.10.

General fixes:

• Restore support for switch creation with "positional" package arguments and --packages option for CLI version 2.0, e.g. OPAMCLI=2.0 opam switch create . 4.12.0+options --packages=ocaml-option-flambda. In opam 2.1 and later, this syntax remains an error (#4843)
• Fix opam switch set-invariant: default repositories were loaded instead of the switch's repositories selection (#4869)
• Run the sandbox check in a temporary directory (#4783)

Integrated depext support has a few updates:

• Homebrew now has support for casks and full-names (#4800)
• Archlinux now handles virtual package detection (#4833, partially addressing #4759)
• Disable the detection of available packages on RHEL-based distributions. This fixes an issue on RHEL-based distributions where yum list used to detect available and installed packages would wait for user input without showing any output and/or fail in some cases (#4791)

And finally two regressions have been dealt with:

• Regression: avoid calling Unix.environment on load (as a toplevel expression). This regression affected opam's libraries, rather than the binary itself (#4789)
• Regression: handle empty environment variable updates (#4840)

A few issues with the compilation of opam from sources have been fixed as well (e.g. mingw-w64 with g++ 11.2 now works)

opam 2.0.10

Two subtle fixes are included in opam 2.0.10. These actually affect the ocaml package. Both of these are Heisenbugs - investigating what's going wrong on your system may well have fixed them, they were both found on Windows!

$(opam env --revert) is the reverse of the more familiar $(opam env) but it's effectively called by opam whenever you change switch. It has been wrong since 2.0.0 for the case where several values are added to an environment variable in one setenv update. For example, if a package included a setenv field of the form [PATH += "dir1:dir2"], then this would not be reverted, but [[PATH += "dir1"] [PATH += "dir2"]] would be reverted. As it happens, this bug affects the ocaml package, but it was masked by another setenv update in the same package.

The other fix is also to do with setenv. It can be seen immediately after creating a switch but before any additional packages are installed, as this Dockerfile shows:

FROM ocaml/opam@sha256:244b948376767fe91e2cd5caca3b422b2f8d332f105ef2c8e14fcc9a20b66e25
RUN sudo apt-get install -y ocaml-nox
RUN opam --version
RUN opam switch create show-issue ocaml-system
RUN eval $(opam env) ; echo $CAML_LD_LIBRARY_PATH
RUN opam install conf-which
RUN eval $(opam env) ; echo $CAML_LD_LIBRARY_PATH

Immediately after switch creation, $CAML_LD_LIBRARY_PATH was set to /home/opam/.opam/show-issue/lib/stublibs:, rather than /home/opam/.opam/show-issue/lib/stublibs:/usr/local/lib/ocaml/4.08.1/stublibs:/usr/lib/ocaml/stublibs

1. From binaries: run

   bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.1"

   or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.

2. From source, using opam:

   opam update; opam install opam-devel

   (then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni to enable sandboxing if you had version 2.0.0~rc manually installed or to update your sandbox script)

3. From source, manually: see the instructions in the README.

We hope you enjoy this new minor version, and remain open to bug reports and suggestions.

Earlier this week, Romain Calascibetta hosted an in-house MirageOS workshop for employees, both locally and remotely around the world. This interactive workshop taught participants how to build an operating system on a Raspberry Pi 4 using MirageOS. They got to create their own OS and play with projects, like one they dubbed GuirlandeOS for which they programmed an LED garland to trim their Christmas tree, creating their own customized light show! There will be a dedicated blog to GuirlandeOS so…

Earlier this week, Romain Calascibetta hosted an in-house MirageOS workshop for employees, both locally and remotely around the world. This interactive workshop taught participants how to build an operating system on a Raspberry Pi 4 using MirageOS. They got to create their own OS and play with projects, like one they dubbed GuirlandeOS for which they programmed an LED garland to trim their Christmas tree, creating their own customized light show! There will be a dedicated blog to GuirlandeOS soon.

That’s one fun example of what can be done with MirageOS and Raspberry Pi, but the possibilities are endless. For example, one could use this dynamic pair to create solar-powered websites (something we’ll hear more about next week).

The spirit of MirageOS is that anyone can integrate it, even if they don't work at Tarides. Although the workshop was only for employees, MirageOS is for everyone! Since it's autonomous from Tarides, we encourage you to play with MirageOS and see what you can create.

Romain has opened the contents of his informative workshop to the world! Follow along with his slides, which will walk you through the MirageOS toolchain to create your very own projects. You can read more about the Raspberry Pi process in this repo.

Early next week, we’ll continue with MirageOS month by showcasing a podcast where Anil Madhavapeddy talks about those solar-powered websites and more!

Happy hacking!

The official release of MirageOS 4.0 quickly approaches! Learn about some general MirageOS concepts and get a sneak park at the forthcoming changes in MirageOS 4.0 during a LIVE presentation today at 15h CET.

Lucas Pluvinage will lead you through a live-streaming presentation to acquaint you with MirageOS 4.0. You’ll learn what kinds of problems MirageOS can solve and about Functoria, the compilation model. Then Lucas will also discuss the switch to the Dune build system and how that enables…

The official release of MirageOS 4.0 quickly approaches! Learn about some general MirageOS concepts and get a sneak park at the forthcoming changes in MirageOS 4.0 during a LIVE presentation today at 15h CET.

Lucas Pluvinage will lead you through a live-streaming presentation to acquaint you with MirageOS 4.0. You’ll learn what kinds of problems MirageOS can solve and about Functoria, the compilation model. Then Lucas will also discuss the switch to the Dune build system and how that enables cross-compilation, not to mention the creation of new compilation targets, such as the Raspberry Pi!

Watch the live presentation today at 15h CET. Just go to: https://meet.google.com/iqy-urht-rcn, or dial ‪(FR) +33 1 87 40 43 45‬ with the PIN: ‪288 878 885‬#. You can also find other phone numbers here: https://tel.meet/iqy-urht-rcn?pin=3736259978366 if you're not in France.

This presentation will be a great introduction to Romain Calascibetta's forthcoming MirageOS workshop tomorrow. Check back on this blog and follow us on Twitter to find out more about how to watch and participate in this informative workshop tomorrow!

Until then, check out Lucas's presentation today at 15h CET.

In my last post, Qubes-lite with KVM and Wayland, I described setting up a Qubes-inspired Linux system that runs applications in virtual machines. A Wayland proxy running in each VM connects its applications to the host Wayland compositor over virtwl, allowing them to appear on the desktop alongside normal host applications. In this post, I extend this to support X11 applications using Xwayland.

Table of Contents

• Overview
• Introduction to X11
• Running Xwayland
• The X11 protocol
• Initialising t…

In my last post, Qubes-lite with KVM and Wayland, I described setting up a Qubes-inspired Linux system that runs applications in virtual machines. A Wayland proxy running in each VM connects its applications to the host Wayland compositor over virtwl, allowing them to appear on the desktop alongside normal host applications. In this post, I extend this to support X11 applications using Xwayland.

Table of Contents

• Overview
• Introduction to X11
• Running Xwayland
• The X11 protocol
• Initialising the window manager
• Windows
• Performance
• Pointer events
• Keyboard events
• Pointer cursor
• Selections
• Drag-and-drop
• Bonus features
  • HiDPI works
  • Ring-buffer logging
  • Vim windows open correctly
  • Copy-and-paste without ^M characters
• Conclusions

Overview

A graphical desktop typically allows running multiple applications on a single display (e.g. by showing each application in a separate window). Client applications connect to a server process (usually on the same machine) and ask it to display their windows.

Until recently, this service was an X server, and applications would communicate with it using the X11 protocol. However, on newer systems the display is managed by a Wayland compositor, using the Wayland protocol.

Many older applications haven't been updated yet. Xwayland can be used to allow unmodified X11 applications to run in a Wayland desktop environment. However, setting this up wasn't as easy as I'd hoped. Ideally, Xwayland would completely isolate the Wayland compositor from needing to know anything about X11:

Fantasy Xwayland architecture

However, it doesn't work like this. Xwayland handles X11 drawing operations, but it doesn't handle lots of other details, including window management (e.g. telling the Wayland compositor what the window title should be), copy-and-paste, and selections. Instead, the Wayland compositor is supposed to connect back to Xwayland over the X11 protocol and act as an X11 window manager to provide the missing features:

Actual Xwayland architecture

This is a problem for several reasons:

1. It means that every Wayland compositor has to implement not only the new Wayland protocol, but also the old X11 protocol.
2. The compositor is part of the trusted computing base (it sees all your keystrokes and window contents) and this adds a whole load of legacy code that you'd need to audit to have confidence in it.
3. It doesn't work when running applications in VMs, because each VM needs its own Xwayland service and existing compositors can only manage one.

Because Wayland (unlike X11) doesn't allow applications to mess with other applications' windows, we can't have a third-party application act as the X11 window manager. It wouldn't have any way to ask the compositor to put Xwayland's surfaces into a window frame, because Xwayland is a separate application.

There is another way to do it, however. As I mentioned in the last post, I already had to write a Wayland proxy (wayland-proxy-virtwl) to run in each VM and relay Wayland messages over virtwl, so I decided to extend it to handle Xwayland too. As a bonus, the proxy can also be used even without VMs, avoiding the need for any X11 support in Wayland compositors at all. In fact, I found that doing this avoided several bugs in Sway's built-in Xwayland support.

Sommelier already has support for this, but it doesn't work for the applications I want to use. For example, popup menus appear in the center of the screen, text selections don't work, and it generally crashes after a few seconds (often with the error xdg_surface has never been configured). So instead I'd been using ssh -Y vm from the host to forward X11 connections to the host's Xwayland, managed by Sway. That works, but it's not at all secure.

Introduction to X11

Unlike Wayland, where applications are mostly unaware of each other, X is much more collaborative. The X server maintains a tree of windows (rectangles) and the applications manipulate it. The root of the tree is called the root window and fills the screen. You can see the tree using the xwininfo command, like this:

$ xwininfo -tree -root

xwininfo: Window id: 0x47 (the root window) (has no name)

  Root window id: 0x47 (the root window) (has no name)
  Parent window id: 0x0 (none)
     9 children:
     0x800112 "~/Projects/wayland/wayland-proxy-virtwl": ("ROX-Filer" "ROX-Filer")  2184x2076+0+0  +0+0
        1 child:
        0x800113 (has no name): ()  1x1+-1+-1  +-1+-1
     0x800123 (has no name): ()  1x1+-1+-1  +-1+-1
     0x800003 "ROX-Filer": ()  10x10+-100+-100  +-100+-100
     0x800001 "ROX-Filer": ("ROX-Filer" "ROX-Filer")  10x10+10+10  +10+10
        1 child:
        0x800002 (has no name): ()  1x1+-1+-1  +9+9
     0x600002 "main.ml (~/Projects/wayland/wayland-proxy-virtwl) - GVIM1": ("gvim" "Gvim")  1648x1012+0+0  +0+0
        1 child:
        0x600003 (has no name): ()  1x1+-1+-1  +-1+-1
     0x600007 (has no name): ()  1x1+-1+-1  +-1+-1
     0x600001 "Vim": ("gvim" "Gvim")  10x10+10+10  +10+10
     0x200002 (has no name): ()  1x1+0+0  +0+0
     0x200001 (has no name): ()  1x1+0+0  +0+0

This tree shows the windows of two X11 applications, ROX-Filer and GVim, as well as various invisible utility windows (mostly 1x1 or 10x10 pixels in size).

Applications can create, move, resize and destroy windows, draw into them, and request events from them. The X server also allows arbitrary data to be attached to windows in properties. You can see a window's properties with xprop. Here are some of the properties on the GVim window:

$ xprop -id 0x600002
WM_HINTS(WM_HINTS):
		Client accepts input or input focus: True
		Initial state is Normal State.
		window id # of group leader: 0x600001
_NET_WM_WINDOW_TYPE(ATOM) = _NET_WM_WINDOW_TYPE_NORMAL
WM_NORMAL_HINTS(WM_SIZE_HINTS):
		program specified minimum size: 188 by 59
		program specified base size: 188 by 59
		window gravity: NorthWest
WM_CLASS(STRING) = "gvim", "Gvim"
WM_NAME(STRING) = "main.ml (~/Projects/wayland/wayland-proxy-virtwl) - GVIM1"
...

The X server itself doesn't know anything about e.g. window title bars. Instead, a window manager process connects and handles that. A window manager is just another X11 application. It asks to be notified when an application tries to show ("map") a window inside the root, and when that happens it typically creates a slightly larger window (with room for the title bar, etc) and moves the other application's window inside that.

This design gives X a lot of flexibility. All kinds of window managers have been implemented, without needing to change the X server itself. However, it is very bad for security. For example:

1. Open an xterm.
2. Use xwininfo to find its window ID (you need the nested child window, not the top-level one).
3. Run xev -id 0x80001b -event keyboard in another window (using the ID you got above).
4. Use sudo or similar inside xterm and enter a password.

As you type the password into xterm, you should see the characters being captured by xev. An X application can easily spy on another application, send it synthetic events, etc.

Running Xwayland

Xwayland is a version of the xorg X server that treats Wayland as its display hardware. If you run it as e.g. Xwayland :1 then it opens a single Wayland window corresponding to the X root window, and you can use it as a nested desktop. This isn't very useful, because these windows don't fit in with the rest of your desktop. Instead, it is normally used in rootless mode, where each child of the X root window may have its own Wayland window.

$ WAYLAND_DEBUG=1 Xwayland :1 -rootless
[3991465.523]  -> wl_display@1.get_registry(new id wl_registry@2)
[3991465.531]  -> wl_display@1.sync(new id wl_callback@3)
...

When run this way, however, no windows actually appear. If we run DISPLAY=:1 xterm then we see Xwayland creating some buffers, but no surfaces:

[4076460.506]  -> wl_shm@4.create_pool(new id wl_shm_pool@15, fd 9, 540)
[4076460.520]  -> wl_shm_pool@15.create_buffer(new id wl_buffer@24, 0, 9, 15, 36, 0)
[4076460.526]  -> wl_shm_pool@15.destroy()
...

We need to run Xwayland as Xwayland :1 -rootless -wm FD, where FD is a socket we will use to speak the X11 protocol and act as a window manager.

It's a little hard to find information about Xwayland's rootless mode, because "rootless" has two separate common meanings in xorg:

1. Running xorg without root privileges.
2. Using xorg's miext/rootless extension to display application windows on some other desktop.

After a while, it became clear that Xwayland's rootless mode isn't either of these, but a third xorg feature also called "rootless".

The X11 protocol

libxcb provides C bindings to the X11 protocol, but I wanted to program in OCaml. Luckily, the X11 protocol is well documented, and generating the messages directly didn't look any harder than binding libxcb, so I wrote a little OCaml library to do this (ocaml-x11).

At first, I hard-coded the messages. For example, here's the code to delete a property on a window:

1
2
3
4
5
6
7
8
9
10
11
12
13

module Delete = struct
  [%%cstruct
    type req = {
      window : uint32_t;
      property : uint32_t;
    } [@@little_endian]
  ]

  let send t window property =
    Request.send_only t ~major:19 sizeof_req @@ fun r ->
    set_req_window r window;
    set_req_property r property
end

I'm using the cstruct syntax extension to let me define the exact layout of the message body. Here, it generates sizeof_req, set_req_window and set_req_property automatically.

After a bit, I discovered that there are XML files in xcbproto describing the X11 protocol. This provides a Python library for parsing the XML, which you can use by writing a Python script for your language of choice. For example, this glorious 3394 line Python script generates the C bindings. After studying this script carefully, I decided that hard-coding everything wasn't so bad after all.

I ended up having to implement more messages than I expected, including some surprising ones like OpenFont (see x11.mli for the final list). My implementation came to 1754 lines of OCaml, which is quite a bit shorter than the Python generator script, so I guess I still came out ahead!

In the X11 protocol, client applications send requests and the server sends replies, errors and events. Most requests don't produce replies, but can produce errors. Replies and errors are returned immediately, so if you see a response to a later request, you know all previous ones succeeded. If you care about whether a request succeeded, you may need to send a dummy message that generates a reply after it. Since message sequence numbers are 16-bit, after sending 0xffff consecutive requests without replies, you should send a dummy one with a reply to resynchronise (but window management involves lots of round-trips, so this isn't likely to be a problem for us). Events can be sent by the server at any time.

Unlike Wayland, which is very regular, X11 has various quirks. For example, every event has a sequence number at offset 2, except for KeymapNotify.

Initialising the window manager

Using Xwayland -wm FD actually prevents any client applications from connecting at all at first, because Xwayland then waits for the window manager to be ready before accepting any client connections.

To fix that, we need to claim ownership of the WM_S0 selection. A "selection" is something that can be owned by only one application at a time. Selections were originally used to track ownership of the currently-selected text, and later also used for the clipboard. WM_S0 means "Window Manager for Screen 0" (Xwayland only has one screen).

1
2
3

(* Become the window manager. This allows other clients to connect. *)
let* wm_sn = intern t ~only_if_exists:false ("WM_S" ^ string_of_int i) in
X11.Selection.set_owner x11 ~owner:(Some root) ~timestamp:`CurrentTime wm_sn

Instead of passing things like WM_S0 as strings in each request, X11 requires us to first intern the string. This returns a unique 32-bit ID for it, which we use in future messages. Because intern may require a round-trip to the server, it returns a promise, and so we use let* instead of let to wait for that to resolve before continuing. let* is defined in the Lwt.Syntax module, as an alternative to the more traditional >>= notation.

This lets our clients connect. However, Xwayland still isn't creating any Wayland surfaces. By reading the Sommelier code and stepping through Xwayland with a debugger, I found that I needed to enable the Composite extension.

Composite was originally intended to speed up redraw operations, by having the server keep a copy of every top-level window's pixels (even when obscured), so that when you move a window it can draw it right away without asking the application for help. The application's drawing operations go to the window's buffer, and then the buffer is copied to the screen, either automatically by the X server or manually by the window manager. Xwayland reuses this mechanism, by turning each window buffer into a Wayland surface. We just need to turn that on:

1
2

let* composite = X11.Composite.init x11 in
let* () = X11.Composite.redirect_subwindows composite ~window:root ~update:`Manual in

This says that every child of the root window should use this system. Finally, we see Xwayland creating Wayland surfaces:

-> wl_compositor@5.create_surface id:+28

Now we just need to make them appear on the screen!

Windows

As usual for Wayland, we need to create a role object and attach it to the surface. This tells Wayland whether the surface is a window or a dialog, for example, and lets us set the title, etc.

But first we have a problem: we need to know which X11 window corresponds to each Wayland surface. For example, we need the title, which is stored in a property on the X11 window. Xwayland does this by sending the new window a ClientMessage event of type WL_SURFACE_ID containing the Wayland ID. We don't get this message by default, but it seems that selecting SubstructureRedirect on the root does the trick.

SubstructureRedirect is used by window managers to intercept attempts by other applications to change the children of the root window. When an application asks the server to e.g. map a window, the server just forwards the request to the window manager. Operations performed by the window manager itself do not get redirected, so it can just perform the same request the client wanted, or make any changes it requires.

In our case, we don't actually need to modify the request, so we just re-perform the original map operation:

1
2
3
4
5
6
7
8
9

let event_handler = object (_ : X11.Event.handler)
  method map_request ~window = X11.Window.map x11 window

  method client_message ~window ~ty body =
      if ty = wl_surface_id then (
        let wayland_id = Cstruct.LE.get_uint32 body 0 in
        Log.info (fun f -> f "X window %a corresponds to Wayland surface %ld" X11.Window.pp window wayland_id);
        pair_when_ready ~x11 t window wayland_id
      )

Having two separate connections to Xwayland is quite annoying, because messages can arrive in any order. We might get the X11 ClientMessage first and need to wait for the Wayland create_surface, or we might get the create_surface first and need to wait for the ClientMessage.

An added complication is that not all Wayland surfaces correspond to X11 windows. For example, Xwayland also creates surfaces representing cursor shapes, and these don't have X11 windows. However, when we get the ClientMessage we can be sure that a Wayland message is on the way, so I just pause the X11 event handling until that has arrived:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

(* We got an X11 message saying X11 [window] corresponds to Wayland surface [wayland_id].
   Turn [wayland_id] into an xdg_surface. If we haven't seen that surface yet, wait until it appears
   on the Wayland socket. *)
let rec pair_when_ready ~x11 t window wayland_id =
  match Hashtbl.find_opt t.unpaired wayland_id with
  | None ->
    Log.info (fun f -> f "Unknown Wayland object %ld; waiting for surface to be created..." wayland_id);
    let* () = Lwt_condition.wait t.unpaired_added in
    pair_when_ready ~x11 t window wayland_id
  | Some { client_surface = _; host_surface; set_configured } ->
    Log.info (fun f -> f "Setting up Wayland surface %ld using X11 window %a" wayland_id X11.Xid.pp window);
    Hashtbl.remove t.unpaired wayland_id;
    Lwt.async (fun () -> pair t ~set_configured ~host_surface window);
    Lwt.return_unit

Another complication is that Wayland doesn't allow you to attach a buffer to a surface until the window has been "configured". Doing so is a protocol error, and Sway will disconnect us if we try! But Xwayland likes to attach the buffer immediately after creating the surface.

To avoid this, I use a queue:

1. Xwayland asks to create a surface.
2. We forward this to Sway, add its ID to the unpaired map, and create a queue for further events.
3. Xwayland asks us to attach a buffer, etc. We just queue these up.
4. We get the ClientMessage over the X11 connection and create a role for the new surface.
5. Sway sends us a configure event, confirming it's ready for the buffer.
6. We forward the queued events.

However, this creates a new problem: if the surface isn't a window then the events will be queued forever. To fix that, when we get a create_surface we also do a round-trip on the X11 connection. If the window is still unpaired when that returns then we know that no ClientMessage is coming, and we flush the queue.

X applications like to create dummy windows for various purposes (e.g. receiving clipboard data), and we need to avoid showing those. They're normally set as override_redirect so the window manager doesn't handle them, but Xwayland redirects them anyway (it needs to because otherwise e.g. tooltips wouldn't appear at all). I'm trying various heuristics to detect this, e.g. that override redirect windows with a size of 1x1 shouldn't be shown.

If Sway asks us to close a window, we need to relay that to the X application using the WM_DELETE_WINDOW protocol, if it supports that:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

let toplevel = Xdg_surface.get_toplevel xdg_surface @@ object
    inherit [_] Xdg_toplevel.v1

    method on_close _ =
      Lwt.async (fun () ->
          let* x11 = t.x11 in
          let* wm_protocols = X11.Atom.intern x11 "WM_PROTOCOLS"
          and* wm_delete_window = X11.Atom.intern x11 "WM_DELETE_WINDOW" in
          let* protocols = X11.Property.get_atoms x11 window wm_protocols in
          if List.mem wm_delete_window protocols then (
            let data = Cstruct.create 8 in
            Cstruct.LE.set_uint32 data 0 (wm_delete_window :> int32);
            Cstruct.LE.set_uint32 data 4 0l;
            X11.Window.send_client_message x11 window ~fmt:32 ~propagate:false ~event_mask:0l ~ty:wm_protocols data;
          ) else (
            X11.Window.destroy x11 window
          )
        )
  end

Wayland defaults to using client-side decorations (where the application draws its own window decorations). X doesn't do that, so we need to turn it off (if the Wayland compositor supports the decoration manager extension):

1
2
3
4
5
6
7
8

t.decor_mgr |> Option.iter (fun decor_mgr ->
    let decor = Xdg_decor_mgr.get_toplevel_decoration decor_mgr ~toplevel @@ object
        inherit [_] Xdg_decoration.v1
        method on_configure _ ~mode:_ = ()
      end
    in
    Xdg_decoration.set_mode decor ~mode:Xdg_decoration.Mode.Server_side
  )

Dialog boxes are more of a problem. Wayland requires every dialog box to have a parent window, but X11 doesn't. To handle that, the proxy tracks the last window the user interacted with and uses that as a fallback parent if an X11 window with type _NET_WM_WINDOW_TYPE_DIALOG is created without setting WM_TRANSIENT_FOR. That could be a problem if the application closes that window, but it seems to work.

Performance

I noticed a strange problem: scrolling around in GVim had long pauses once a second or so, corresponding to OCaml GC runs. This was surprising, as OCaml has a fast incremental garbage collector, and is normally not a problem for interactive programs. Besides, I'd been using the proxy with the (Wayland) Firefox and xfce4-terminal applications for 6 months without any similar problem.

Using perf showed that Linux was spending a huge amount of time in release_pages. The problem is that Xwayland was sharing lots of short-lived memory pools with the proxy. Each time it shares a pool, we have to ask the VM host for a chunk of memory of the same size. We map both pools into our address space and then copy each frame across (this is needed because we can't export guest memory to the host).

Normally, an application shares a single pool and just refers to regions within it, so we just map once at startup and unmap at exit. But Xwayland was creating, sharing and discarding around 100 pools per second while scrolling in GVim! Because these pools take up a lot of RAM, OCaml was (correctly) running the GC very fast, freeing them in batches of 100 or so each second.

First, I tried adding a cache of host memory, but that only solved half the problem: freeing the client pool was still slow.

Another option is to unmap the pools as soon as we get the destroy message, to spread the work out. Annoyingly, OCaml's standard library doesn't let you free memory-mapped memory explicitly (see the Add BigArray.Genarray.free PR for the current status), but adding this myself with a bit of C code would have been easy enough. We only touch the memory in one place (for the copy), so manually checking it hadn't been freed would have been pretty safe.

Then I noticed something interesting about the repeated log entries, which mostly looked like this:

-> wl_shm@4.create_pool id:+26 fd:(fd) size:8368360
-> wl_shm_pool@26.create_buffer id:+28 offset:0 width:2090 height:1001 stride:8360 format:1
-> wl_shm_pool@26.destroy 
<- wl_display@1.delete_id id:26
-> wl_buffer@28.destroy 
<- wl_display@1.delete_id id:28

Xwayland creates a pool, allocates a buffer within it, destroys the pool (so it can't create more buffers), and then deletes the buffer. But it never uses the buffer for anything!

So the solution was simple: I just made the host buffer allocation and the mapping operations lazy. We force the mapping if a pool's buffer is ever attached to a surface, but if not we just close the FD and forget about it. Would be more efficient if Xwayland only shared the pools when needed, though.

Pointer events

Wayland delivers pointer events relative to a surface, so we simply forward these on to Xwayland unmodified and everything just works.

I'm kidding - this was the hardest bit! When Xwayland gets a pointer event on a window, it doesn't send it directly to that window. Instead, it converts the location to screen coordinates and then pushes the event through the old X event handling mechanism, which looks at the X11 window stack to decide where to send it.

However, the X11 window stack (which we saw earlier with xwininfo -tree -root) doesn't correspond to the Wayland window layout at all. In fact, Wayland doesn't provide us any way to know where our windows are, or how they are stacked.

Sway seems to handle this via a backdoor: X11 applications do get access to location information even though native Wayland clients don't. This is one of the reasons I want to get X11 support out of the compositor - I want to make sure X11 apps don't have any special access. Sommelier has a solution though: when the pointer enters a window we raise it to the top of the X11 stack. Since it's the topmost window, it will get the events.

Unfortunately, the raise request goes over the X11 connection while the pointer events go over the Wayland one. We need to make sure that they arrive in the right order. If the computer is running normally, this isn't much of a problem, but if it's swapping or otherwise struggling it could result in events going to the wrong place (I temporarily added a 2-second delay to test this). This is what I ended up with:

1. Get a wayland pointer enter event from Sway.
2. Pause event delivery from Sway.
3. Flush any pending Wayland events we previously sent to Xwayland by doing a round-trip on the Wayland connection.
4. Send a raise on the X11 connection.
5. Do a round-trip on the X11 connection to ensure the raise has completed.
6. Forward the enter event on the Wayland connection.
7. Unpause the event stream from Sway.

At first I tried queuing up just the pointer events, but that doesn't work because e.g. keyboard events need to be synchronised with pointer events. Otherwise, if you e.g. Shift-click on something then the click gets delayed but the Shift doesn't and it can do the wrong thing. Also, Xwayland might ask Sway to destroy the window while we're entering it, and Sway might confirm the deletion. Pausing the whole event stream from Sway fixes all these problems.

The next problem was how to do the two round-trips. For X11 we just send an Intern request after the raise and wait to get a reply to that. Wayland provides the wl_display.sync method to clients, but we're acting as a Wayland server to Xwayland, not a client. I remembered that Wayland's xdg-shell extension provides a ping from the server to the client (the compositor can use this to detect when an application is not responding). Unfortunately, Xwayland has no reason to use this extension because it doesn't deal with window roles. Luckily, it uses it anyway (it does need it for non-rootless mode and doesn't bother to check).

wl_display.sync works by creating a fresh callback object, but xdg-shell's ping just sends a pong event to a fixed object, so we also need a queue to keep track of pings in flight so we don't get confused between our pings and any pings we're relaying for Sway. Also, xdg-shell's ping requires a serial number and we don't have one. But since Xwayland is the only app this needs to support, and it doesn't look at that, I cheat and just send zero.

And that's how to get pointer events to go to the right window with Xwayland.

Keyboard events

A very similar problem exists with the keyboard. When Wayland says the focus has entered a window we need to send a SetInputFocus over the X11 connection and then send the keyboard events over the Wayland one, requiring another two round-trips to synchronise the two connections.

Pointer cursor

Some applications set their own pointer shape, which works fine. But others rely on the default and for some reason you get no cursor at all in that case. To fix it, you need to set a cursor on the root window, which applications will then inherit by default. Unlike Wayland, where every application provides its own cursor bitmaps, X very sensibly provides a standard set of cursors, in a font called cursor (this is why I had to implement OpenFont). As cursors have two colours and a mask, each cursor is two glyphs: even numbered glyphs are the image and the following glyph is its mask:

1
2
3
4
5
6
7
8
9
10

(* Load the default cursor image *)
let* cursor_font = X11.Font.open_font x11 "cursor" in
let* default_cursor = X11.Font.create_glyph_cursor x11
    ~source_font:cursor_font ~mask_font:cursor_font
    ~source_char:68 ~mask_char:69
    ~bg:(0xffff, 0xffff, 0xffff)
    ~fg:(0, 0, 0)
in
X11.Window.create_attributes ~cursor:default_cursor ()
|> X11.Window.change_attributes x11 root

Selections

The next job was to get copying text between X and Wayland working.

In X11:

• When you select something, the application takes ownership of the PRIMARY selection.
• When you click the middle button or press Shift-Insert, the application requests PRIMARY.
• When you press Ctrl-C, the application takes ownership of the CLIPBOARD selection.
• When you press Ctrl-V it requests CLIPBOARD.

It's quite neat that adding support for a Windows-style clipboard didn't require changing the X server at all. Good forward-thinking design there.

In Wayland, things are not so simple. I have so far found no less than four separate Wayland protocols for copying text:

1. gtk_primary_selection supports copying the primary selection, but not the clipboard.
2. wp_primary_selection_unstable_v1 is identical to gtk_primary_selection except that it renames everything.
3. wl_data_device_manager supports clipboard transfers but not the primary selection.
4. zwlr_data_control_manager_v1 supports both, but it's for a "privileged client" to be a clipboard manager.

gtk_primary_selection and wl_data_device_manager both say they're stable, while the other two are unstable. However, Sway dropped support for gtk_primary_selection a while ago, breaking many applications (luckily, I had a handy Wayland proxy and was able to add some adaptor code to route gtk_primary_selection messages to the new "unstable" protocol).

For this project, I went with wp_primary_selection_unstable_v1 and wl_data_device_manager. On the Wayland side, everything has to be written twice for the two protocols, which are almost-but-not-quite the same. In particular, wl_data_device_manager also has a load of drag-and-drop stuff you need to ignore.

For each selection (PRIMARY or CLIPBOARD), we can be in one of two states:

• An X11 client owns the selection (and we own the Wayland selection).
• A Wayland client owns the selection (and we own the X11 selection).

When we own a selection we proxy requests for it to the matching selection on the other protocol.

• At startup, we take ownership of the X11 selection, since there are no X11 apps running yet.
• When we lose the X11 selection it means that an X11 client now owns it and we take the Wayland selection.
• When we lose the Wayland selection it means that a Wayland client now owns it and we take the X11 selection.

One good thing about the Wayland protocols is that you send the data by writing it to a normal Unix pipe. For X11, we need to write the data to a property on the requesting application's window and then notify it about the data. And we may need to split it into multiple chunks if there's a lot of data to transfer.

A strange problem I had was that, while pasting into GVim worked fine, xterm would segfault shortly after trying to paste into it. This turned out to be a bug in the way I was sending the notifications. If an X11 application requests the special TEXT target, it means that the sender should choose the exact format. You write the property with the chosen type (e.g. UTF8_STRING), but you must still send the notification with the target TEXT. xterm is a C application (thankfully no longer set-uid!) and seems to have a use-after-free bug in the timeout code.

Drag-and-drop

Sadly, I wasn't able to get this working at all. X itself doesn't know anything about drag-and-drop and instead applications look at the window tree to decide where the user dropped things. This doesn't work with the proxy, because Wayland doesn't tell us where the windows really are on the screen.

Even without any VMs or proxies, drag-and-drop from X applications to Wayland ones doesn't work, because the X app can't see the Wayland window and the drop lands on the X window below (if any).

Bonus features

In the last post, I mentioned several other problems, which have also now been solved by the proxy:

HiDPI works

Wayland's support for high resolution screens is a bit strange. I would have thought that applications really only need to know two things:

1. The size in pixels of the window.
2. The size in pixels you want some standard thing (e.g. a normal-sized letter M).

Some systems instead provide the size of the window and the DPI (dots-per-inch), but this doesn't work well. For example, a mobile phone might be high DPI but still want small text because you hold it close to your face, while a display board will have very low DPI but want large text.

Wayland instead redefines the idea of pixel to be a group of pixels corresponding to a single pixel on a typical 1990's display. So if you set your scale factor to 2 then 1 Wayland pixel is a 2x2 grid of physical pixels. If you have a 1000x1000 pixel window, Wayland will tell the application it is 500x500 but suggest a scale factor of 2. If the application supports HiDPI mode, it will double all the numbers and render a 1000x1000 image and things work correctly. If not, it will render a 500x500 pixel image and the compositor will scale it up.

Since Xwayland doesn't support this, it just draws everything too small and Sway scales it up, creating a blurry and unusable mess. This might be made worse by subpixel rendering, which doesn't cope well with being scaled.

With the proxy, the solution is simple enough: when talking to Xwayland we just scale everything back up to the real dimensions, scaling all coordinates as we relay them:

1
2
3

let scale_to_client t (x, y) =
  x * t.config.xunscale,
  y * t.config.xunscale

1
2
3
4
5
6
7
8
9

method on_configure _ ~width ~height ~states:_ =
  let width = Int32.to_int width in
  let height = Int32.to_int height in
  if width > 0 && height > 0 then (
    Lwt.async (fun () ->
        let (width, height) = scale_to_client t (width, height) in
        X11.Window.configure x11 window ~width ~height ~border_width:0
      )
  )

This will tend to make things sharp but too small, but X applications already have their own ways to handle high resolution screens. For example, you can set Xft.dpi to make all the fonts bigger. I run this proxy like this, which works for me:

wayland-proxy-virtwl --x-display=0 --xrdb Xft.dpi:150 --x-unscale=2

However, there is a problem. The Wayland specification says:

The new size of the surface is calculated based on the buffer size transformed by the inverse buffer_transform and the inverse buffer_scale. This means that at commit time the supplied buffer size must be an integer multiple of the buffer_scale. If that's not the case, an invalid_size error is sent.

Let's say we have an X11 image viewer that wants to show a 1001-pixel-high image in a 1001-pixel-high window. This isn't allowed by the spec, which can only handle even-sized windows when the scale factor is 2. Regular Wayland applications already have to deal with that somehow, but for X11 applications it becomes our problem.

I tried rounding down, but that has a bad side-effect: if GTK asks for a 1001-pixel high menu and gets a 1000 pixel allocation, it switches to squashed mode and draws two big bumper arrows at the top and bottom of the menu which you must use to scroll it. It looks very silly.

I also tried rounding up, but tooltips look bad with any rounding. Either one border is missing, or it's double thickness. Luckily, it seems that Sway doesn't actually enforce the rule about surfaces being a multiple of the scale factor. So, I just let the application attach a buffer of whatever size it likes to the surface and it seems to work!

The only problem I had was that when using unscaling, the mouse pointer in GVim would get lost. Vim hides it when you start typing, but it's supposed to come back when you move the mouse. The problem seems to be that it hides it by creating a 1x1 pixel cursor. Sway decides this isn't worth showing (maybe because it's 0x0 in Wayland-pixels?), and sends Xwayland a leave event saying the cursor is no longer on the screen. Then when Vim sets the cursor back, Xwayland doesn't bother updating it, since it's not on screen!

The solution was to stop applying unscaling to cursors. They look better doubled in size, anyway. True, this does mean that the sharpness of the cursor changes as you move between windows, but you're unlikely to notice this due to the far more jarring effect of Wayland cursors also changing size and shape at the same time.

Ring-buffer logging

Even without a proxy to complicate things, Wayland applications often have problems. To make investigating this easier, I added a ring-buffer log feature. When on, the proxy keeps the last 512K or so of log messages in memory, and will dump them out on demand.

To use it, you run the proxy with e.g. -v --log-ring-path ~/wayland.log. When something odd happens (e.g. an application crashes, or opens its menus in the wrong place) you can dump out the ring buffer and see what just happened with:

echo dump-log > /run/user/1000/wayland-1-ctl

I also added some filtering options (e.g. --log-suppress motion,shm) to suppress certain classes of noisy messages.

Vim windows open correctly

One annoyance with Sway is that Vim's window always appears blank (even when running on the host, without any proxy). You have to resize it before you can see the text.

My proxy initially suffered from the same problem, although only intermittently. It turned out to be because Vim sends a ConfigureRequest with its desired size and then waits for the confirmation message. Since Sway is a tiling window manager, it ignores the new size and no event is generated. In this case, an X11 window manager is supposed to send a synthetic ConfigureNotify, so I just got the proxy to do that and the problem disappeared (I confirmed this by adding a sleep to Vim's gui_mch_update).

By the way, the GVim start-up code is quite interesting. The code path to opening the window goes though three separate functions which each define a static int recursive = 0 and then proceed to behave differently depending on how many times they've been reentered - see gui_init for an example!

Copy-and-paste without ^M characters

The other major annoyance with Sway is that copy-and-paste doesn't work correctly (Sway bug #1839). Using the proxy avoids that problem completely.

Conclusions

I'm not sure how I feel about this project. It ended up taking a lot longer than I expected, and I could probably have ported several X11 applications to Wayland in the same time. On the other hand, I now have working X support in the VMs with no need for ssh -Y from the host, plus support for HiDPI in Wayland, mouse cursors that are large enough to see easily, windows that open reliably, text pasting that works, and I can get logs whenever something misbehaves.

In fact, I'm now also running an instance of the proxy directly on the host to get the same benefits for host X11 applications. Setting this up is actually a bit tricky: you want to start Sway with DISPLAY=:0 so that every application it spawns knows it has an X11 display, but if you set that then Sway thinks you want it to run nested inside an X window provided by the proxy, which doesn't end well (or, indeed, at all).

Having all the legacy X11 support in a separate binary should make it much easier to write new Wayland compositors, which might be handy if I ever get some time to try that. It also avoids having many thousands of lines of legacy C code in the highly-trusted compositor code.

If Wayland had an official protocol for letting applications know the window layout then I could make drag-and-drop between X11 applications within the same VM work, but it still wouldn't work between VMs or to Wayland applications, so it's probably not worth it.

Having two separate connections to Xwayland creates a lot of unnecessary race conditions. A simple solution might be a Wayland extension that allows the Wayland server to say "please read N bytes from the X11 socket now", and likewise in the other direction. Then messages would always arrive in the order in which they were sent.

The code is all available at https://github.com/talex5/wayland-proxy-virtwl if you want to try it. It works with the applications I use when running under Sway, but will probably require some tweaking for other programs or compositors. Here's a screenshot of my desktop using it:

Screenshot of my desktop

The windows with [dev] in the title are from my Debian VM, while [com] is a SpectrumOS VM I use for email, etc. Gitk, GVim and ROX-Filer are X11 applications using Xwayland, while Firefox and xfce4-terminal are using plain Wayland proxying.

In April, we announced that the DAPSI initiative accepted the proposal for our Secure-by-Design Communication Protocols (SCoP) project. Today, we are thrilled to announce that SCoP has passed the initiative’s Phase 1, and we are now on our way to Phase 2!

SCoP is an open, secure, and resource-efficient infrastructure to engineer a modern basis for open messaging (for existing and emerging protocols) using type-safe languages and unikernels—to ensure your private information remains secure. A…

In April, we announced that the DAPSI initiative accepted the proposal for our Secure-by-Design Communication Protocols (SCoP) project. Today, we are thrilled to announce that SCoP has passed the initiative’s Phase 1, and we are now on our way to Phase 2!

SCoP is an open, secure, and resource-efficient infrastructure to engineer a modern basis for open messaging (for existing and emerging protocols) using type-safe languages and unikernels—to ensure your private information remains secure. After all, you wouldn’t like your postal carrier reading your snail mail, so why should emails be any different?

Challenges

To operate an email service requires many technical skills and reliable infrastructure. As a result, only a few large companies can handle emails with the proper security levels. Unfortunately, the core business model of these companies is to mine your personal data.

The number of emails exchanged every day is expected to reach 333 billion in 2022. That’s a considerable amount of data, much of it private or sensitive, sent across Cyberspace through portals with questionable security. The ‘memory unsafe’ languages used in most communication services leave far too much room for mistakes that have serious ramifications, like security flaws that turn into security breaches, leaving your personal or business information vulnerable to malicious hackers.

Due to this global challenge, we set out to build a simple, secure, easily deployable solution to preserve users' privacy, and we’re making great strides toward accomplishing that goal. We base our systems on scientific foundations to last for decades and drive positive change for the world. Our robust understanding of both theory and practice enables us to solve these security problems, so we explore ideas where research and engineering meet at the intersection of the domains of operating systems, distributed systems, and programming languages.

Every component of SCoP is carefully designed as independent libraries, using modern development techniques to avoid the common reported threats and flaws. For instance, the implementation of protocol parsers and serializers are written in a type-safe language and tested using fuzzing. Combining these techniques will increase users' trust to migrate their personal data to these new, more secure services.

Architecture

The architecture of the SCoP communication service is composed of an Email Service based on a secure extension of the SMTP protocol, and a decentralised real-time communication system based on Matrix.

The SMTP and Matrix protocols implemented in SCoP follow the separation of concerns design principle, meaning that the SMTP Sender and SMTP Receiver are designed as two distinct units. They’re implemented as isolated micro-services which run as unikernels. The SMTP Sender, Receiver, and Matrix are all configurable, and each configuration comes with a security risk analysis report to understand possible privacy risks

Progress

Not only are we on our way to Phase 2 in the DAPSI Initiative, but we’re also proud to report that we’re on track with our planned milestones!

Our first milestone was to generate a corpus of emails to test our parser implementation against existing projects in order to detect differences between the descriptions specified in the RFCs. We now have 1 million emails that have been parsed/encoded without any issues! Our email corpus keeps isomorphism between the encoder and decoder, and you can find it in this GitHub Repo, as we encourage implementors of other languages to use it to improve their trust in their own implementation.

We set out to implement an SMTP extension mechanism and support for SPF as well as implement DMARC, a security framework, on top of DKIM and SPF for our second milestone, and we are right on target. To date, we’ve completed four components:

• SPF
• DKIM
• SMTP can send and verify emails
• MrMIME can generate the email, then SMTP sends the email (signed by a DKIM private key). We can correctly sign an email, generate a signature, and the DKIM field containing the signature. When the email is received, we check the DKIM signature and the SPF metadata.

For our third milestone, we set out to implement DNSSEC, a set of security extensions over DNS. This security layer verifies the identity of an email sender through DKIM/SPF/DMARC, but it also needs security extensions in the DNS protocol. We completed our initial investigation of a DNSSEC implementation prototype, and we discovered several issues, like some of the elliptic curve cryptography was missing. Those necessary cryptographic primitives are now available, so we should complete this milestone by the end of the month.

Finally, our fourth milestone was to implement the Matrix protocol (client and server). We completed the protocol’s client library, which sends a notification from OCaml CI. Plus, we have a PoC, and Matrix’s server-side, which received the notification, is also complete.

Although we still have much work ahead of us, we’re quite pleased with the progress thus far, and so is the DAPSI Initiative! Follow our progress by subscribing to this blog and our Twitter feed (@tarides_) for the latest updates.

As mentioned in our Tezos Storage / Irmin Summer 2021 Update on the Tezos Agora forum, the Irmin team's goal has been to improve Irmin's performance in order to speed up the Baking Account migration process in Octez, and we managed to make it 10x faster in the first quarter of 2021. Since then, we've been working on a new benchmark program for Irmin that's based on the interactions between Irmin and Octez. This won't just help make Irmin even faster, it will also help speed up the Tezos blockcha…

As mentioned in our Tezos Storage / Irmin Summer 2021 Update on the Tezos Agora forum, the Irmin team's goal has been to improve Irmin's performance in order to speed up the Baking Account migration process in Octez, and we managed to make it 10x faster in the first quarter of 2021. Since then, we've been working on a new benchmark program for Irmin that's based on the interactions between Irmin and Octez. This won't just help make Irmin even faster, it will also help speed up the Tezos blockchain process and enable us to monitor Irmin's behavior in Octez.

Octez is the Tezos node implementation that uses Irmin to store the blockchain state, so Irmin is a core component of Octez that's responsible for virtually all the filesystem operations. Whether a node is launched to produce new blocks (aka “bake”) or just to participate in peer-to-peer sharing of existing blocks, it must first update itself by rebuilding blocks individually until it reaches the head of the blockchain. This first phase is called bootstrapping, and once it reaches the blockchain head, we say it has been bootstrapped. Currently, the bootstrapped phase processes 2 block per minute, which is the rate at which the Tezos blockchain progresses. The next goal is to increase that rate to 12 blocks per minute.

Irmin stores the content of the Tezos blockchain on a disk using the irmin-pack library. There is one-to-one correspondence between the Tezos block and the Irmin commits. Each time Tezos produces a block, Irmin produces a commit, and then the Tezos block hash is computed using the Irmin commit hash. The Irmin developers are working on improving the irmin-pack performance which in turn will improve the performance of Octez.

A benchmark program is considered “fair” when it's representative of how the benchmarked code is used in the real world—for example, the access-patterns to Irmin. A standard database benchmark would first insert random data and then remove it. Such a synthetic benchmark would fail to reproduce the bottlenecks that occur when the insertions and removal are interleaved. Our solution to “fairness” is radical: replaying. Within a sandboxed environment, we replay a real world situation.

Basically, our new benchmark program makes use of a benchmarked code and records statistics for later analysis. The program is stored in the irmin-bench library and makes use of operation traces (called action traces) when Octez runs with Irmin. Later, the program replays the recorded operations one at a time while simultaneously recording tonnes of statistics (called stat traces). Data analysis of the stat traces may reveal many interesting facts about the behaviour of Irmin, especially if we tweak:

• the configuration of Irmin (e.g., what’s the impact of doubling the size of a certain cache?)
• the replay parameters (e.g., does Irmin's performance decay over time? Does irmin-pack perform as well after 24 hours of replay as after 1 minute of replay?)
• the hardware (e.g., does irmin-pack perform well on a Raspberry Pi?)
• the code of Irmin (e.g., does this PR have an impact on performance?)

This benchmarking process is similar to the record-replay feature available with TezEdge.

Recording the Action Trace

By adding logs to Tezos, we can record the Tezos-Irmin interactions and thus capture the Irmin “view” of Tezos. We’ve recorded action traces during the bootstrapping phase of Tezos nodes, which started from Genesis—the name of the very first Tezos block inserted into an empty Irmin store.

The interaction surface between Irmin and Octez is quite simple, so we were able to reduce it to eight (8) elementary operations:

• checkout, to pull an Irmin tree from disk;
• find, mem and mem_tree, read only operations on an Irmin tree;
• add, remove and copy, write only operations on an Irmin tree;
• commit, to push an Irmin tree to disk.

It’s important to remember that Irmin behaves much like Git. It has built-in snapshotting and is compatible with Git itself when using the irmin-git library. In fact, these operations are very similar to Git, too.

Sequence of Operations

To illustrate further, here's a concrete example of an operation sequence inside an action trace:

This shows Octez’s first interaction with Irmin at the very beginning of the blockchain! The first block, Genesis, is quite small (it ends at operation #5), but the second one is massive (it ends at operation #309273). It contains no transactions because it only sets up the entire structure of the tree. It precedes the beginning of Tezos's initial protocol called “Alpha I”.

Benchmark Benefits

Our benchmark results convey the sheer magnitude of the Tezos blockchain and the role that Irmin plays within it. We’ve recorded a trace that covers the blocks from the beginning the blockchain in June 2018 all the way up to May 2021. It weighs 96GB.

Although it took 34 months for Tezos to reach that state, bootstrapping so far takes only 170 hours, and replaying it takes a mere 37 hours on a section of the blockchain that contains 1,343,486 blocks. On average, this corresponds to 1 per minute when the blocks were created, 132 per minute when bootstrapping, and 611 per minute during replay.

On this particular section of the blockchain, Octez had 1,089,853,521 interactions with Irmin. On average, this corresponds to 12 per second when the blocks were created, 1782 per second during bootstrapping, and 8258 per second during replay.

The chart below demonstrates how many of each Irmin operation occur per block (on average):

This next chart displays where the time is spent during replay:

With irmin-pack, an OCaml thread managed by the index library is running concurrently to the main thread (i.e., the merge thread), a fraction of the durations (shown above) are actually spent in that thread. Refer to this blog post for more details on index's merges.

The following chart illustrates how memory usage evolves during replay:

On a logarithmic scale, this last chart shows the evolution of the write amplification, which indicates the amount of rewriting (e.g., at the end of the replay, 20TB of data have been written to disk in order to create a store that weighs 73GB).

The merge operations of the index library are the source of this poor write amplification. The Irmin team is working hard on improving this metric:

• on the one hand, the new structured keys feature of the upcoming Irmin 3.0 release will help to reduce the pressure on the index library,
• on the other hand, we are working on algorithmic improvements of index itself.

Another nice way to use the trace is for testing. When replaying a trace, we can recompute the commit hashes and check that they correspond to the trace hashes, so the benchmark acts as additional tests to ensure we don't compromise the hashes computed in Tezos.

Complex changes to Tezos can be simulated first in Irmin. For example, the path flattening in Tezos feature (merged in August 2021) can now be tested earlier in the process with our benchmark. Prior to the trace benchmarks, we first had to make the changes in Tezos to understand their repercussions on Irmin directly from the Tezos benchmarks.

Lastly, we continue to test alternative libraries and compare them with the ones integrated in Tezos; however, using these alternative libraries to build Tezos nodes has proven to be more complicated than merely adding them in Irmin and running our benchmarks. While testing continues on most new libraries, we can definitely use replays to compare our new cactus library as a replacement for our index library.

Future Directions

While the action trace recording was only made possible on a development branch of Octez, we would next like to upstream the feature to the main branch of Octez, which would give all users the option to record Tezos-Irmin interactions. This would simplify bug reporting overall.

Although the first version only deals with the bootstapping phase of Tezos, an upcoming goal is to make it possible to benchmark the boostrapped phase of Tezos as well. Additionally, we plan to replay the multiprocess aspects of a Tezos node in the near future.

The first stable version of this benchmark has existed in Irmin’s development branch since Q2 2021, and we will release it as part of irmin-bench for Irmin 3.0 in Q4 2021. This release will allow integration into the Sandmark OCaml benchmarking suite.

Follow the Tarides blog for future Irmin updates.

The OCaml typechecker is an important piece of the OCaml compiler pipeline which accounts for a significant portion of time spent on compiling an OCaml program (see the appendices).

The code of the typechecker is also quite optimised, sometime…

The OCaml typechecker is an important piece of the OCaml compiler pipeline which accounts for a significant portion of time spent on compiling an OCaml program (see the appendices).

The code of the typechecker is also quite optimised, sometimes to the detriment of the readability of the code. Recently, Jacques Garrigue and Takafumi Saikawa have worked on a series of pull requests to improve the readability of the typechecker (#10337, #10474, #10541). Unfortunately, those improvements are also expected to increase the typechecking time of OCaml programs because they add abstraction barriers, and remove some optimisations that were breaking the abstraction barriers.

The effect is particularly pronounced on #10337. Due to the improvement of the readability of the typechecker, this pull request has been merged after some quick tests to check that the compilation time increase was not too dire.

However, the discussion on this pull request highlighted the fact that it was difficult to measure OCaml compilation time on a scale large enough to enable good statistical analysis and that it would be useful.

Consequently, I decided to try my hand at a statistical analysis of OCaml compilation time, using this pull request #10337 as a case study. Beyond this specific PR, I think that it is interesting to write down a process and a handful of tools for measuring OCaml compilation time on the opam ecosystem.

Before doing any kind of analysis, the first step is to find an easy way to collect the data of interest. Fortunately, the OCaml compiler can emit timing information with flag -dtimings. However, this information is emitted on stdout, whereas my ideal sampling process would be to just pick an opam package, launch a build process and recover the timing information for each file. This doesn’t work if the data is sent to the stdout, and never see again. This first step is thus to create a version of the OCaml compiler that can output the timing information of the compilation to a specific directory. With this change (#10575), installing an opam package with

OCAMLPARAM=",_,timings=1,dump-dir= /tmp/pkgnname" opam install pkgname

outputs all profiling information to /tmp/pkgname.

This makes it possible to collect large number of data points on compilation times by using opam, and the canonical installation process of each package without the need of much glue code.

For this case study, I am using 5 core packages containers, dune, tyxml, coq and base. Once their dependencies are added, I end up with

• ocamlfind
• num
• zarith
• seq
• containers
• coq
• dune
• re
• ocamlbuild
• uchar
• topkg
• uutf
• tyxml
• sexplib0
• base

Then it is a matter of repeatedly installing those packages, and measuring the compilation times before and after #10337.

In order to get more reliable statistics on each file, each package was compiled 250 times leading to 1,6 millions of data points (available here) after slightly more than a week-end of computation.

In order to try to reduce the noise induced by the operating system scheduler, the compilation process is run with OPAMJOBS=1. Similarly, the compilation process was isolated as much as possible from the other process using the cset Linux utility to reserve one full physical core to the opam processes.

The code for collecting samples, analyzing them, and plotting the graphs below is available at https://github.com/Octachron/ocaml-perfomance-monitoring.

Comparing averages, files by files

With the data at hand, we can compute the average compilation by files, and by stage of the OCaml compiler pipeline. In our case, we are mostly interested in the typechecking stage, and global compilation time, since #10337 should only alter the time spent on typechecking. It is therefore useful to split the compilation time into typechecking + other=total. Then for each files in the 15 packages above, we can can compute the average time for each of those stages and the relative change of average compilation time: time after/time before.

Rendering those relative changes for the typechecking time, file by file (with the corresponding 90% confidence interval) yields

To avoid noise, I have removed files for which the average typechecking time was inferior to one microsecond on the reference version of the compiler.

In the graph above, there are few remarkable points:

• As expected, the average typechecking time increased for almost all files
• A significant portion of points are stuck to the line “after/before=1”. This means that for those files there was no changes at all of the typechecking times.
• The standard deviation time varies wildly across packages. The typechecking of some dune files tend to have a very high variances. However outside of those files, the standard deviation seems moderate, and the mean estimator seem to have converged.
• For a handful a files for which the typechecking time more than doubled. However the relative typechecking time does seem to be confined in the [1,1.2] range for a majority of files.

Since the data is quite noisy, it is useful before trying to interpret it to check that we are not looking only at noise. Fortunately, we have the data on the time spent outside of the typechecking stage available, and those times should be mostly noise. We have thus a baseline, that looks like

This cloud of points look indeed much noisier. More importantly, it seems centred around the line after/before=1. This means that our hypothesis that the compilation time outside of the typechecking stage has not been altered is not visibly invalidated by our data points. An other interesting point is that the high variance points seems to be shared between the typechecking and other graphs.

We can even check on the graphs for the average total compilation (file by file)

that those points still have a high variance here. However, outside of this cluster of points, we have a quite more compact distribution of points for the total compilation time: it seems that we have a quite consistent increase of the total compilation time of around 3%.

And this is reflected in the averages:

There is thus an increase of around 6.6% of typechecking time which translates to an increase of 3.3% of total time. However, the non-typechecking time also increased by 1.7% in average. The average is thus either tainted by some structural bias or the relative variance (mean/ratio) is still enough for the distribution of the ratio to be ill-behaved (literature seems to indicate that a relative variance < 10% is required for the distribution of ratio to be Gaussian-like). Anyway, we probably cannot count on a precision of more than 1.7%. Even with this caveat, we still have a visible effect on the total compilation time.

We might better served by comparing the geometric average. Indeed, we are comparing ratio of time, with possibly a heavy-tailed noise. By using the geometric average (which compute the exponential of the arithmetic mean of the logarithms of our ratio), we can check that rare events don’t have an undue influence on the average. In our case the geometric means looks like

All geometric averages have decreased compared to the arithmetic means, which is a sign that the compilation time distribution is skewed towards high compilation times. However, the changes are small and do not alter our previous interpretation.

We can somewhat refine those observations by looking at the medians (which are even less affected by the heavy-tailness of distributions)

Here, the non-typechecking times seems far less affected by the structural bias (with an increase of 0.9%) whereas the increase of typechecking time and total compilation time are reduced but still here at 3.8% and 2.5% respectively.

Comparing averages, quantiles

We can refine our analysis by looking at the quantiles of those relative changes of compilation time

Here we see that the typechecking time of around 25% of files is simply not affected at all by the changes. And for half of the files, the compilation time is inferior to 9%. Contrarily, there is 1% of files for which the typechecking time increases by more than 50% (with outliers around 200%-400% increase).

However, looking at the total compilation does seems to reduce the overall impact of the change

Indeed, we still have 25% of files not impacted, but for 65% of files the relative increase of compilation time is less than 8%. (and the outliers stalls at a 50% increase)

We can also have a quick look at the quantiles for the non-typechecking time

but here the only curiosity if that the curve is more symmetric and we have 25% of files for which the non-typechecking compilation time decrease randomly.

Noise models and minima

One issue with our previous analysis is the high variance which is observable in the non-typechecking average times across files. A possibility to mitigate this issue is to change our noise model. Using an average, we implicitly assumed that the compilation time was mostly:

observable_compilation_time = theoretical_computation_time + noise

where noise is a random variable with at least a finite variance and a mean of 0. Indeed, with this symmetry hypothesis the expectation of the observable computation time aligns with the theoretical compilation time:

E[observable_computation_time] = E[theoretical_computation_time] + E[noise] = theoretical_computation_time

and the variance Var[observable_computation_time] is exactly the variance of the noise Var[noise]. Then our finite variance hypothesis ensure that the empirical average hypothesis converges relatively well towards the theoretical expectation.

However, we can imagine another noise model with a multiplicative noise (due to CPU scheduling for instance),

observable_compilation_time = scheduling_noise * theoretical_computation_time + noise

with both scheduling_noise>1 and noise>1. With this model, the expectation of the observable compilation time does not match up with the theoretical computation time:

E[observable_computation_time] - theoretical_computation_time =
  (E[scheduling_noise]-1) * theoretical_computation_time + E[noise]

Thus, in this model, the average observable computation time is a structurally biased estimator for the theoretical computation time. This bias might be compensated by the fact that we are only looking to ratio. Nevertheless, this model also induces a second source of variance

Var[observable_computation_time] = theoretical_computation_time^2 Var[scheduling_noise] + Var[noise]

(/assuming the two noises are not correlated), and this variance increases with the theoretical computation time. This relative standard deviation might be problematic when computing ratio.

If this second noise model is closer to reality, using the empirical average estimators might be not ideal. However, the positivity of the noise opens another avenue for estimators: we can consider the minima of a series of independent realisations. Then, we have

min(observable_compilation_time) = min(scheduling_noise * theoretical_computation_time) + min(noise) = theoretical_computation_time

if the min(scheduling_noise)=1 and min(noise)=0.

This model has another advantage: by assuming that the (essential) support of the noise distribution has finite lower bound, we know that the empirical minima will converge towards a three-parameter Weibull distribution with a strictly positive support. (To be completely explicit, we also need to assume some regularity of the distribution around this lower bound too).

This means that the distribution ratio of the empirical minima will not exhibits the infinite moments of the ratio of two Gaussians. Without this issue, our estimator should have less variance.

However, we cannot use Gaussian confidence intervals for the empirical minima. Moreover, estimating the confidence interval for the Weibull distribution is more complex. Since we are mostly interested in corroborating our previous result, we are bypassing the computation of those confidence intervals.

Comparing minima

We can then restart out analysis using the minimal compilation time file-by-file. Starting with the minimal typechecking time, we get

There are notable differences with the average version: - a very significant part of our points takes the same time to typecheck before and after #10337 - there is a discretization effects going on: data points tend to fall on exactly the same value of the ratio

Beyond those changes, there is still a visible general increase of typechecking time.

The same differences are visible for the non typechecking compilation time

and the total compilation time

but overall the minimal total compilation and non-typechecking time mirrors what we had seen with the average. The distribution of the non-typechecking times is maybe more evenly centred around a ratio of 1.

We can have a look at the averages and median (across files) to have more global point of view

A striking change is that the median for the typechecking and total compilation time is equal to one: more than half of files are not affected by the changes when looking at the minimal compilation time. This might be an issue with the granularity of time measurement, or it could be a genuine fact.

More usefully, we still have an increase of average typechecking time between 6% and 6.9% depending on the averaging methods, which translates to a total compilation time increase between 2.7% and 3.3%. And this time, the increase of unrelated compilation time is between 0.7% to 0.9%. This seems to confirms that do have a real increase of average compilation time and 3% increase time is a reasonable number.

Comparing minima, quantiles

With the discretization, the quantiles of the compilation time are quite interesting and uncharacteristic.

For instance the typechecking quantiles,

are stuck to 1 between the first and 50th centile. In other words the minimal typechecking time of more than 50% of the files in our experiment is unchanged. For 40% of the files, the increase is less than 20%. And the most extreme files see only an increase of 100% of the typechecking time. On the higher quantiles, the presence of multiple jumps is the consequence of the discretization of ratio that was already visible on the raw data.

When looking at the time spent outside of typechecking,

we observe that the non-typechecking relation compilation time for more than 80% of file is unaffected by the change (or somehow accelerated for 10% of files).

The quantiles for the total compilation time,

mostly reflects the trends set by the typechecking time: 55% of files are unaffected. For 90% of file the increase is less than 10%, and the maximal impact on the compilation time peaks at a 40% relative increase.

Conclusion

To sum up, with the available data at hands, it seems sensible to conclude that #10337 resulted in an average increase of compilation time of the order of 3%, while the average relative increase of typechecking time is around 6%. Moreover, for the most impacted files (at the ninth decile), the relative increase in compilation time ranges between 10% to 40%.

Appendices

Compilation profile

Since we have data for both typechecking time and non-typechecking times for a few thousand files, it is interesting to check how much time is spent on typechecking. We can start by looking at the data points files by files:

We have here a relatively uniform cloud of points between 20-60% of time spent in typechecking compared to total compilation time. This is is reflected on the average and median

Both value are quite comparable, the distribution doesn’t seem significantly skewed.

However, we have a clear cluster of files for which typechecking accounts for 90% of the total compilation time. Interestingly, this cluster of points corresponds to the dune cluster of files with a very variance that we had identified earlier. This explains why those files have essentially the same profile when looking at the total and typechecking compilation time: in their case, typechecking accounts for most of the work done during compilation.

This relatively uniform distribution is visible both on the quantiles (with an affine part of the quantiles)

and on the histogram of the relative time spent in typechecking

Histograms

Histogram versions for the quantile diagrams are also available. Due to the mixture of continuous and discrete distributions they are not that easy to read. Note that those histograms have equiprobable bins (in other words, constant area) rather than constant width bins.

Average compilation time histograms

An interesting take-away for those histograms is that the typechecking and total compilation time distribution are clearly skewed to the right: with very few exceptions, compilation increases. Contrarily the non-typechecking time distribution is much more symmetric. Since the change here is due to noise, there is no more reason for the compilation time to increase or decrease.

Minimal compilation time histograms

There is no much change when looking at the histogram for the minimal compilation time for a file

The most notable difference is that the non-typechecking histogram is completely dominated by the dirac distribution centred at x=1.

Defining instructions semantics using Primus Lisp (Tutorial)

Introduction

So you found a machine instruction that is not handled by BAP and you wonder how to add it to BAP. This is the tutorial that will gently guide you through the whole process of discovering the instruction, studying its semantics, encoding it, testing, and finally submitting to BAP. The latter is optional but highly appreciated.

In modern BAP, the easiest option is to use Primus Lisp to define new instructions. The idea i…

Defining instructions semantics using Primus Lisp (Tutorial)

Introduction

So you found a machine instruction that is not handled by BAP and you wonder how to add it to BAP. This is the tutorial that will gently guide you through the whole process of discovering the instruction, studying its semantics, encoding it, testing, and finally submitting to BAP. The latter is optional but highly appreciated.

In modern BAP, the easiest option is to use Primus Lisp to define new instructions. The idea is that for each instruction, you describe its effects using Primus Lisp. No recompilation or OCaml coding is required. For example here is the semantics of the tst instruction in the thumb mode taken from BAP,

(defun tTST (rn rm _ _)
  "tst rn, rm"
  (let ((rd (logand rn rm)))
    (set ZF (is-zero rd))
    (set NF (msb rd))))

You now probably have the question: what is tTST and why it has four parameters? We use LLVM (and now Ghidra) as the disassembler backend and when we write a lifter (i.e., when we define the semantics of instructions) we rely on the representation of instructions that are provided by the backend. In LLVM it is MC Instructions, which more or less corresponds to LLVM MIR.

Picking an instruction

So let’s find some ARM instruction that is not yet handled by BAP and use it as a working example. The bap mc command is an ideal companion in writing lifters (mc - stands for machine code), as it lets you work with individual instructions. Besides, I wasn’t able to find any ARM instruction that we do not handle (except co-processors instructions, which are not interesting from the didactic perspective) so we will be working with the thumb2 instruction set. So first of all, we need a binary encoding for the instruction that we would like to lift. If you don’t have one then use llvm-mc (or any other assembler). The encoding (which I found from some wild-caught arm binary is 2d e9 f8 43 and we can disassemble it using bap mc

$ bap mc --arch=thumb --show-insn=asm -- 2d e9 f8 43
push.w {r3, r4, r5, r6, r7, r8, r9, lr}

Or, if we want to go the other way around, from assembly to binary encoding then here is the llvm-mc command-line,

$ echo "push.w {r3, r4, r5, r6, r7, r8, r9, lr}" | llvm-mc -triple=thumb -mattr=thumb2 --show-inst -show-encoding
    .text
    push.w    {r3, r4, r5, r6, r7, r8, r9, lr} @ encoding: [0x2d,0xe9,0xf8,0x43]
                                        @ <MCInst #4118 t2STMDB_UPD
                                        @  <MCOperand Reg:15>
                                        @  <MCOperand Reg:15>
                                        @  <MCOperand Imm:14>
                                        @  <MCOperand Reg:0>
                                        @  <MCOperand Reg:75>
                                        @  <MCOperand Reg:76>
                                        @  <MCOperand Reg:77>
                                        @  <MCOperand Reg:78>
                                        @  <MCOperand Reg:79>
                                        @  <MCOperand Reg:80>
                                        @  <MCOperand Reg:81>
                                        @  <MCOperand Reg:13>>

Now we need to get full information about the instruction name. In BAP, all instruction names are packaged in namespaces to enable multiple backends. To get full information about the instruction encoding and decoding we will use the –show-knowledge option of bap mc. This command-line option will instruct BAP to dump the whole knowledge base, so it will have everything that bap knows so far about the instruction. The property that we’re looking for, is bap:lisp-name and bap:insn. The first will give us the true name and the last will show us how the instruction and operands are represented, e.g.,

$ bap mc --arch=thumb --show-knowledge -- 2d e9 f8 43 | grep lisp-name
   (bap:lisp-name (llvm-thumb:t2STMDB_UPD))
$ bap mc --arch=thumb --show-knowledge -- 2d e9 f8 43 | grep bap:insn
   (bap:insn ((t2STMDB_UPD SP SP 0xe Nil R3 R4 R5 R6 R7 R8 R9 LR)))

Writing the stub semantics

Okay, now we have nearly all the knowledge so we can start writing the semantics. Let’s start with some stub semantics, we will later look into the instruction set manual and learn the instruction semantics, but we want to make sure that BAP loads our files and calls our semantics function.

BAP searches for the semantics file in a number of predefined locations (see bap --help and search for --primus-lisp-semantics option for more details). The default locations include you $HOME/.local/share/bap/primus/semantics and the system-wide location that is usually dependent on your installation (usually it is in an opam switch in <opam-switch>/share/bap/primus/semantics. So you can either place your file in the home location or just pick an arbitrary location and tell bap where to search for it using --bap-primus-semantics=<dir>. In our example, we use the current folder (denoted with . in Unix systems) where we create a file named extra-thumb2.lisp (the name of the file doesn’t really matter for the purposes of example, as long as it has the .lisp extension).

Now, we can create the stub definition of the instruction,

$ cat extra-thumb2.lisp
(defun llvm-thumb:t2STMDB_UPD (_ _ _ _ _ _ _ _ _ _ _ _)
  (msg "yay, it works"))

and let’s check that bap properly dispatches the semantics,

$ bap mc --arch=thumb --primus-lisp-semantics=. --show-bil -- 2d e9 f8 43
yay, it works
{

}

Learning the instruction semantics

So what does this series of _ _ _ _ ... mean? We see that bap applies t2STMDB_UPD as

bap mc --arch=thumb --show-knowledge -- 2d e9 f8 43 | grep bap:insn
   (bap:insn ((t2STMDB_UPD SP SP 0xe Nil R3 R4 R5 R6 R7 R8 R9 LR)))

So it has 12 operands. We haven’t yet learned their semantics so we just ignored them. In Primus Lisp, like in OCaml, you can use the wildcard character as the argument name, if you’re not using it. Now it is time to figure out what do they mean. The encoding of the llvm machine instruction is defined in the LLVM target description (*.td) files, which we can find in the LLVLM GitHub repository, namely, we can learn,

// PUSH/POP aliases for STM/LDM
def : t2InstAlias<"push${p}.w $regs", (t2STMDB_UPD SP, pred:$p, reglist:$regs)>;

So now we know that push.w regs is an alias (syntactic sugar) to stmdb sp!, {r3, r4, r5, r6, r7, r8, r9, lr} (or even stmdb sp!, {r3-r9, lr}. Let’s check that our gues is correct using llvm-mc,

$ echo 'stmdb sp!, {r3-r9, lr}' | llvm-mc -triple=thumb -mattr=thumb2 -show-encoding
    .text
    push.w    {r3, r4, r5, r6, r7, r8, r9, lr} @ encoding: [0x2d,0xe9,0xf8,0x43]

Indeed, the encoding is the same. So now it is time to download an instruction reference and look for the stm instruction. It is on page 357 of the pdf version, in section c2.141, and it says that instruction stores multiple registers. The db suffix is the addressing mode that instructs us to Decrement the address Before each store operation. And the ! suffix (encoded as _UPD in llvm) prescribes us to store the final address back to the destination register. This is a high-level reference intended for asm programmers, so if we need more details with the precisely described semantics we can also look into the ARM Architecture Reference Manual (the pdf file). Here is the semantics obtained from this reference, which is described in the ARM pseudocode,

(I had to screen-shot it, as the indentation matters in their pseudo-code but it could not be copied properly from the pdf file).

Learning Primus Lisp

So we’re now ready to write some lisp code. You may have already skimmed through the Primus Lisp documentation that describes the syntax and semantics of the language. If you didn’t don’t worry (but it is still advised to do this eventually). Primus Lisp is a Lisp dialect that looks much like Common Lisp. Since it is Lisp, it has a very simple syntax - everything is an s-expression, i.e., either an atom, e.g., 1, ‘hello, r0, or an application of a name to a list of arguments, e.g., (malloc 100) or (malloc (sizeof ptr_t)). The semantics of Primus Lisp is very simple as well. Basically, Primus Lisp is a universal syntax for assemblers. There is no other data type than bitvectors. From the perspective of the type system we distinguish bitvectors by their sizes, e.g., 0x1:1 is a one-bitvector with the only bit set to 1 and 0x1:16 is a 16-bitvector which has different size and different value and is a 16-bit word with the lower bit set to 1. All operations evaluate to words and accept words. Now what operations you can use? We can use BAP to get the up-to-date documentation for Primus Lisp. For this we will use the primus-lisp-documentation command (we need to pass the –semantics option as we have two interperters for Primus Lips, one for dynamic execution and another for static execution, which have different libraries and slightly different set of primitives),

bap primus-lisp-documentation --semantics > semantics.org

This command will generate the API documentation in the emacs org-mode. If you don’t want to use Emacs to read this file then you can use pandoc to transform it to any format you like, e.g.,

pandoc semantics.org -o semantics.pdf

That will generate the following document.

Encoding semantics (the first attempt)

Now, after we skimmed through the documentation, let’s make our first ugly (and may be incorrect) attempt to describe the semantics of the instruction,

(defun llvm-thumb:t2STMDB_UPD (dst base _pred _? r1 r2 r3 r4 r5 r6 r7 r8)
  (let ((len 8)
        (stride (sizeof word-width))
        (ptr (- base (* len stride))))
    (store-word (+ ptr (* stride 0)) r1)
    (store-word (+ ptr (* stride 1)) r2)
    (store-word (+ ptr (* stride 2)) r3)
    (store-word (+ ptr (* stride 3)) r4)
    (store-word (+ ptr (* stride 4)) r5)
    (store-word (+ ptr (* stride 5)) r6)
    (store-word (+ ptr (* stride 6)) r7)
    (store-word (+ ptr (* stride 7)) r8)
    (set$ dst ptr)))

and before getting into the details, let’s see how bap translates this semantics to BIL,

$ bap mc --arch=thumb --primus-lisp-semantics=. --show-bil -- 2d e9 f8 43
{
  #3 := SP - 0x20
  mem := mem with [#3, el]:u32 <- R3
  mem := mem with [#3 + 4, el]:u32 <- R4
  mem := mem with [#3 + 8, el]:u32 <- R5
  mem := mem with [#3 + 0xC, el]:u32 <- R6
  mem := mem with [#3 + 0x10, el]:u32 <- R7
  mem := mem with [#3 + 0x14, el]:u32 <- R8
  mem := mem with [#3 + 0x18, el]:u32 <- R9
  mem := mem with [#3 + 0x1C, el]:u32 <- LR
  SP := #3
}

The result looks good and, maybe even correct, but let’s explore the lisp code.

The first thing to notice is that instead of using _ _ ... placeholders we gave each parameter a name. The first parameter is the destination register (it is the llvm rule that all functions that update a register have this register as the first parameter), then we have the base register (in our working example the destination and the base register are the same). Next, we have the _pred which we currently ignore, but will return later. We use the _ prefix to indicate that it is unused. Then there is an operand of unknown purpose, which we denoted as _? (I usually just blank them with _, but this is to show that lisp allows you to use any non-whitespace characters in the identifier). Finally, we have r1 til r8, which binds the passed registers. Here, r1 denotes not the register r1 of ARM but the first register passed to the function, i.e., r3 in our example. (It is to show that you can pick any names for your parameters and that they can even shadow the globally defined target-specific register names, which is probably a bad idea from the readability perspective, choosing something like xN would be probably a better idea).

Now it is time to look into the body of the function. First of all, we used (let (<bindings>) <body>) construct to bind several variables. Each binding has the form (<varN> <exprN>) and it evaluates <exprN> and binds its value to <varN> and make it available for expressions <exprN+1> and in the <body>, e.g., in the full form,

(let ((<var1> <expr1>)
      (<var2> <expr2>)
      ....
      (<varN> <exprN>))
   <body>)

we can use <var1> in <expr2>, ..., <exprN>, and in the <body>, i.e., this is the lexical scope of <var1>. Once the let expression is evaluated <var1> becomes unbound (or bound to whatever it was bound before. In other words, normal lexical scoping, which is totally equivalent to the OCaml,

let var1 = expr1 in
let var2 = expr2 in
...
let varN = exprN in
body

The let-bound variables are reified into temporary variables in BIL. You may also notice that Primus Lisp Semantic compiler is clever enough and removed the constants. Let’s go through the variables.

The first variable is len is a constant equal to 8, which is the number of registers passed to the function. Unfortunately, llvm encodes the 15 registers not with a bitset (as it is actually represented in the instruction) but as a list of operands. So instead of writing one function, we will need to write 15 overloads per each number of registers in the list.

The next variable is also a constant, but we use a complex expression to describe it, (sizeof word-width). The Primus Lisp Semantics Compiler is a meta-interpreter and it computes all values that are known in the static time. As a result, we don’t see this constant in the reified code.

Unleashing the macro power

Now let’s deal with the body. It goes without saying that it is ugly. We have to manually unfold the loop since we don’t have a program variable that denotes the set of registers that we have to store, but instead, llvm represents this instruction as a set of 15 overloads. BAP Primus Lisp supports overloads as well, but we definitely won’t like to write 15 overloads with repetitive code, it is tedious, error-prone, and insults us as programmers.

Here comes the macro system. Primus Lisp employs a very simple macro system based on term rewriting. For each application (foo x y z) the parser looks for a definition of macro foo that has the matching number of parameters. If an exact match is found then (foo x y z) is rewritten with the body of the match. Otherwise, a match with the largest number of parameters is selected and the excess arguments are bound to the last parameter, e.g., if we have

(defmacro foo (x xs) ...)

then it will be called with x bound to a and xs bound to (b c) if applied as (foo a b c), where a, b, c could be arbitrary s-expressions.

The body of the macro may reference itself, i.e., it could be recursive. To illustrate it let’s write the simplest recursive macro that will compute the length of the list,

(defmacro length (r) 1)
(defmacro length (_ rs) (+ 1 (length rs)))

and now we can check that it works by adding

(msg "we have $0 registers" (length r1 r2 r3 r4 r5 r6 r7 r8))

to the body of our definition. It will print,

we have 0x8 registers

Note that macros do not perform any compuations on bitvectors, unlike the Primus meta compiler. The macro engine operates on s-expressions, and (length r1 r2 r3 r4 r5 r6 r7 r8) is rewritten (+ 1 (+ 1 (+ 1 (+ 1 (+ 1 (+ 1 (+ 1 1))))))) on the syntatic level and is later reduced by the Primus Meta Compiler into 8.

To solidify the knowledge and to move forward let’s write a macro store-registers that will take an arbitrary number of registers, e.g., (store-registers base stride off r1 r2 r3 .... rm) which will unfold to a sequence of stores, where each register rN is stored as (store-word (+ base (* stride N)) rN).

First, let’s define the base case of our recursion,

(defmacro store-registers (base stride off reg)
  (store-word (+ base (* stride off)) reg))

and now the recursive case,

(defmacro store-registers (base stride off reg regs)
  (prog
     (store-registers base stride off reg)
     (store-registers base stride (+ off 1) regs)))

Notice, that the body of the macro must be a single s-expression. There is no so-called implicit body and if we need to chain two expressions we have to explicitly use the prog construct. This construct has a very simple semantics, (prog s1 s2 ... sN) means evaluate s1, then s2, and so on until sN and make the result of the whole form equal to the result of the evaluation of the last expression.

And a better representation of the body will be,

(defun llvm-thumb:t2STMDB_UPD (dst base _pred _? r1 r2 r3 r4 r5 r6 r7 r8)
  (let ((len 8)
        (stride (sizeof word-width))
        (ptr (- base (* len stride))))
    (store-registers ptr stride 0 r1 r2 r3 r4 r5 r6 r7 r8)
    (set$ dst ptr)))

And let’s double-check that we still have the same reification to BIL with

$ bap mc --arch=thumb --primus-lisp-semantics=. --show-bil -- 2d e9 f8 43
{
  #3 := SP - 0x20
  mem := mem with [#3, el]:u32 <- R3
  mem := mem with [#3 + 4, el]:u32 <- R4
  mem := mem with [#3 + 8, el]:u32 <- R5
  mem := mem with [#3 + 0xC, el]:u32 <- R6
  mem := mem with [#3 + 0x10, el]:u32 <- R7
  mem := mem with [#3 + 0x14, el]:u32 <- R8
  mem := mem with [#3 + 0x18, el]:u32 <- R9
  mem := mem with [#3 + 0x1C, el]:u32 <- LR
  SP := #3
}

In fact, we can also put into a macro the body of our function, and our length macro will be of use here, e.g.,

(defun llvm-thumb:t2STMDB_UPD (dst base _pred _? r1 r2 r3 r4 r5 r6 r7 r8)
  (stmdb_upd dst base r1 r2 r3 r4 r5 r6 r7 r8))

(defmacro stmdb_upd (dst base regs)
  (let ((len (length regs))
        (stride (sizeof word-width))
        (ptr (- base (* len stride))))
    (store-registers ptr stride 0 regs)
    (set$ dst ptr)))

A side-note on the macro resolution mechanism

A small notice on the macro resolution process. Another way of describing it is that the rewriting engine picks the most specific definition. In fact, this is true for the definitions also. The resolution mechanism collects all definitions for the given name that matches the given context and selects the most specific. The context is defined using the Primus Lisp type class mechanism. The only difference between macros and functions (beyond that macros operate on the syntax tree level) is that macros add the number of arguments (arity) to the context so that the definition with the highest arity is ordered after (have precedence over) all other definitions. This is described more thoroughly in the reference documentation. Another important takeaway for us is that when we write a lifter we end up referencing target-specific functions and registers. So we would like to limit the context of the applicability of our functions to the specified target. (Otherwise, our functions will be loaded and type-checked for all architectures, e.g., when an x86 binary is parsed, and we don’t want this). Therefore, we should start our lifter with the context declaration that will be automatically attached to each definition in the file,

(declare (context (target arm)))

Taming names with namespaces and packages

Now, let’s look into the packages. A package is the Common Lisp (and Lisp in general) name for a namespace. A namespace is just a set of names, and each name in Primus Lisp has a package. In fact, when we write (defmacro stmdb_upd () ...), i.e., without specifying the namespace, the name is automatically prefixed with the current namespace/package. By default, it is the user package. So our definition was read by the Lisp reader as (defmacro user:stmdb_upd ...). It is not only the macro or function names that are packaged. Any identifier is properly namespaced. So that (store-registers ptr stride 0 regs) is read as (user:store-registers user:ptr user:stride 0 user:regs).

The namespaces are also thoroughly described in the documentation but the rendered documentation is outdated because our bot is broken (I really need to fix it), so right now I can only refer you to the sources of the documentation in the mli file. And if you have bap installed in opam, then you can also install odig and odoc and use odig doc bap to generate and read the documentation on your machine. (It will spill an error but just repeat and it will show the correctly rendered documentation, it is a bug upstream that we have reported but… well I have diverged).

what we will do now, we will define the thumb package and llvm-thumb package. The first package will be our working package where we will put all our definitions. And the second package will be specific to llvm,

(defpackage thumb (:use core target arm))
(defpackage llvm-thumb (:use thumb))

(in-package thumb)

The (:use core target arm) stanza means that all definitions in these packages are imported (read “copied”) into the thumb package. And the same for llvm-thumb, basically, (defpackage llvm-thumb (:use thumb)) means copy (export) all definitions from thumb to llvm-thumb.

Both thumb and llvm-thumb packages are already defined in BAP, so we can just say

(in-package thumb)

Unlike Common Lisp, in Primus Lisp, the same package could be defined multiple times and even used before defined. The packages may even mutually import each other. The package system resolves it and finds the least fixed point, but it is probably too much for such a simple tutorial. For us, the main takeaway is that we don’t need to write llvm-thumb and are no longer polluting the namespace with our definitions thanks to packaging and contexts.

The final step, writing overloads

Unfortunately, there is no macro mechanism that will operate on the definition level and generate definitions from some pattern. We probably will develop something for that, but right now for each overload we still need to write a corresponding function, e.g.,

(defun t2STMDB_UPD (dst base _pred _?
                      r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11 r12 r13 r14 r15)
  (stmdb_upd dst base r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11 r12 r13 r14 r15))
....
(defun t2STMDB_UPD (dst base _pred _?
                      r1)
  (stmdb_upd dst base r1))

So that the final version of our code will look like this (see also the gist version)

(declare (context (target arm)))

(in-package thumb)

(defun t2STMDB_UPD (dst base _pred _?
                      r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11 r12 r13 r14 r15)
  (stmdb_upd dst base r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11 r12 r13 r14 r15))

(defun t2STMDB_UPD (dst base _pred _?
                      r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11 r12 r13 r14)
  (stmdb_upd dst base r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11 r12 r13 r14))

(defun t2STMDB_UPD (dst base _pred _?
                      r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11 r12 r13)
  (stmdb_upd dst base r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11 r12 r13))

(defun t2STMDB_UPD (dst base _pred _?
                      r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11 r12)
  (stmdb_upd dst base r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11 r12))

(defun t2STMDB_UPD (dst base _pred _?
                      r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11)
  (stmdb_upd dst base r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11))

(defun t2STMDB_UPD (dst base _pred _?
                      r1 r2 r3 r4 r5 r6 r7 r8 r9 r10)
  (stmdb_upd dst base r1 r2 r3 r4 r5 r6 r7 r8 r9 r10))

(defun t2STMDB_UPD (dst base _pred _?
                      r1 r2 r3 r4 r5 r6 r7 r8 r9)
  (stmdb_upd dst base r1 r2 r3 r4 r5 r6 r7 r8 r9))

(defun t2STMDB_UPD (dst base _pred _?
                      r1 r2 r3 r4 r5 r6 r7 r8)
  (stmdb_upd dst base r1 r2 r3 r4 r5 r6 r7 r8))

(defun t2STMDB_UPD (dst base _pred _?
                      r1 r2 r3 r4 r5 r6 r7)
  (stmdb_upd dst base r1 r2 r3 r4 r5 r6 r7))

(defun t2STMDB_UPD (dst base _pred _?
                      r1 r2 r3 r4 r5 r6)
  (stmdb_upd dst base r1 r2 r3 r4 r5 r6))

(defun t2STMDB_UPD (dst base _pred _?
                      r1 r2 r3 r4 r5)
  (stmdb_upd dst base r1 r2 r3 r4 r5))

(defun t2STMDB_UPD (dst base _pred _?
                      r1 r2 r3 r4)
  (stmdb_upd dst base r1 r2 r3 r4))

(defun t2STMDB_UPD (dst base _pred _?
                      r1 r2 r3)
  (stmdb_upd dst base r1 r2 r3))

(defun t2STMDB_UPD (dst base _pred _?
                      r1 r2)
  (stmdb_upd dst base r1 r2))

(defun t2STMDB_UPD (dst base _pred _?
                      r1)
  (stmdb_upd dst base r1))

(defmacro stmdb_upd (dst base regs)
  (let ((len (length regs))
        (stride (sizeof word-width))
        (ptr (- base (* len stride))))
    (store-registers ptr stride 0 regs)
    (set$ dst ptr)))

(defmacro store-registers (base stride off reg)
  (store-word (+ base (* stride off)) reg))

(defmacro store-registers (base stride off reg regs)
  (prog
     (store-registers base stride off reg)
     (store-registers base stride (+ off 1) regs)))

(defmacro length (r) 1)
(defmacro length (_ rs) (+ 1 (length rs)))

I think on this we can conclude our tutorial. I will later polish it, but it covers most of the features of Primus Lisp and writing lifters.

Except for the last step - which is making a PR to BAP with the file. Please, once you wrote a semantics for a machine instruction do not hesitate and PR it to the main repository. It should go to plugins//semantics/, or you can just add the lisp code to an existing lisp file in this folder if you think it would be easier to maintain. This will be highly appreciated.

Happy lifting!

The Bonus Track - Recitation

And as the final accord and recitation, let’s check how we can lift push {r1-r12,r14}.

1) We will use llvm-mc to obtain the binary encoding,

$ echo 'push {r1-r12,r14}' | llvm-mc -triple=thumb -mattr=thumb2 -show-encoding
    .text
    push.w    {r1, r2, r3, r4, r5, r6, r7, r8, r9, r10, r11, r12, lr} @ encoding: [0x2d,0xe9,0xfe,0x5f]

2) next we check that our overload is correctly picked up,

$ bap mc --arch=thumb --primus-lisp-semantics=. --show-bil -- 0x2d,0xe9,0xfe,0x5f
{
  #3 := SP - 0x34
  mem := mem with [#3, el]:u32 <- R1
  mem := mem with [#3 + 4, el]:u32 <- R2
  mem := mem with [#3 + 8, el]:u32 <- R3
  mem := mem with [#3 + 0xC, el]:u32 <- R4
  mem := mem with [#3 + 0x10, el]:u32 <- R5
  mem := mem with [#3 + 0x14, el]:u32 <- R6
  mem := mem with [#3 + 0x18, el]:u32 <- R7
  mem := mem with [#3 + 0x1C, el]:u32 <- R8
  mem := mem with [#3 + 0x20, el]:u32 <- R9
  mem := mem with [#3 + 0x24, el]:u32 <- R10
  mem := mem with [#3 + 0x28, el]:u32 <- R11
  mem := mem with [#3 + 0x2C, el]:u32 <- R12
  mem := mem with [#3 + 0x30, el]:u32 <- LR
  SP := #3
}

So, yay, it works! :)

Last year, Tarides had the honour of winning the “Coup de Coeur” Startup Award at the International Cybersecurity Forum (FIC). It’s the leading cybersecurity event in the EU. It’s both a forum, to present and discuss innovations and reflect on the state of the European cybersecurity ecosystem, and a trade fair, where cybersecurity and other tech professionals can meet and network.

This year, Tarides returns to FIC with their own booth! Our representatives, including founder and CEO of Ta…

Last year, Tarides had the honour of winning the “Coup de Coeur” Startup Award at the International Cybersecurity Forum (FIC). It’s the leading cybersecurity event in the EU. It’s both a forum, to present and discuss innovations and reflect on the state of the European cybersecurity ecosystem, and a trade fair, where cybersecurity and other tech professionals can meet and network.

This year, Tarides returns to FIC with their own booth! Our representatives, including founder and CEO of Tarides, Thomas Gazagnaire, look forward to making new connections, looking for collaborators, and catching up with colleagues.

The FIC theme for 2021 centres on encouraging “a collective and collaborative cybersecurity.” From the FIC Website:

”Collective, because each stakeholder is responsible not only for its own security but also for the security of every other stakeholder, and therefore of the whole. Collaborative, because cooperation and information sharing are essential to compensate the asymmetry between the «attacker» and the «defender»…FIC 2021 will focus on the major operational, industrial, technological, and strategic challenges of cooperation”

Solutions for most cybersecurity issues already exist, and we at Tarides are happy to consult on our many solutions that can make your systems more secure. From secure emails to runtime protection to secure IoT protocols, our representatives can help!

So stop by the Tarides booth at FIC to chat about your options. We’ll have snacks and other goodies, too!

This year marks the 25th anniversary of the OCaml Language! It's an exciting time for OCaml programmers and enthusiasts. A fun and informative way to celebrate OCaml's birthday is to attend the 26th Annual International Conference on Functional Programming (ICFP), held online this year due to ongoing Covid restrictions. While this is disappointing news for so many, it's beneficial to those of you outside France because now you can hear professionals talk about cutting edge technology from the co…

This year marks the 25th anniversary of the OCaml Language! It's an exciting time for OCaml programmers and enthusiasts. A fun and informative way to celebrate OCaml's birthday is to attend the 26th Annual International Conference on Functional Programming (ICFP), held online this year due to ongoing Covid restrictions. While this is disappointing news for so many, it's beneficial to those of you outside France because now you can hear professionals talk about cutting edge technology from the comfort of your own home.

Tarides engineers, as well as our colleagues at OCaml Labs Consultancy and Segfault Systems, have some exciting presentations at this year's ICFP! Listen to talks on running OCaml on multiple cores, generating fuzzing suites, benchmarking, and the experimental OCaml effects.

You can search the [complete ICFP Timetable](https://icfp21.sigplan.org/program/program-icfp-2021/?past=Show upcoming events only&date=Fri 27 Aug 2021) for other topics of interest and read below about our engineers' projects and presentations. Times are listed both in London (GMT +1) and Paris (GMT +2) for ease of planning. The following talks are scheduled for Friday, 27 August 2021.

Grab a cup of coffee for our first morning talk at 9am London / 10am Paris and learn about Adapting the OCaml Ecosystem for Multicore OCaml. With the soon-to-be released OCaml 5.0, there will be support for Shared-Memory Parallelism. There’s increasing interest in the community to port existing libraries to Multicore, so this talk will cover the arrival of Multicore and what that means to the OCaml ecosystem. Our engineers will highlight existing tools and provide methods for a smooth transition, so viewers can benefit from Multicore parallelism. They'll also share some insights from their experience porting existing libraries to Multicore OCaml.

Read more about this topic on todays' post at Segfault Systems, written by one of tomorrow's presenters, Sudha Parimala of Segfault Systems. Joining Sudha for the presentation are Enguerrand Decorne (Tarides), Sadiq Jaffer (Opsian and OCaml Labs Consultancy), Tom Kelly (OCaml Labs Consultancy), and KC Sivaramakrishnan of IIT Madras.

Next up is Leveraging Formal Specifications to Generate Fuzzing Suites at 11:10 London / 12:10 Paris, presented by Tarides's own Nicolas Osborne and Clément Pascutto. They'll discuss how developers typically first have to capture the semantics they want when checking a library and then write the code implementing these tests and find relevant test cases that expose possible misbehaviours. Through their work, they'll present a tool that automatically takes care of those last two steps by automatically generating fuzz testing suites from OCaml interfaces annotated with formal behavioural specifications. They'll also show some ongoing experiments on fuzzing capabilities and limitations applied to real-world libraries.

Next up is our talk on Continuous Benchmarking for OCaml Projects at 12:30 London / 13:30 Paris. Regular CI systems are optimised for workloads that do not require stable performance over time, which makes them unsuitable for running performance benchmarks. Tarides engineers Gargi Sharma, Rizo Isrof, and Magnus Skjegstad will discuss how current-bench provides a predictable environment for performance benchmarks and a UI for analysing results over time. Similar to a CI system it runs on pull requests and branches allowing performance to be analysed and compared, and it can currently be enabled on as an app on GitHub repositories with zero configuration. Several public repositories already run current-bench, including Irmin and Dune, and they plan to enable it on more projects in the future. Read Gargi's recent blog post for more information on benchmarking.

In this presentation, they will give a technical overview of current-bench, showing how results are collected and analysed, requirements for using it, and how they built the infrastructure for stable benchmarks. They'll also cover some future work that will allow more OCaml projects to run current-bench.

Immediately after the Benchmarking talk, catch A Multiverse of Glorious Documentation scheduled at 12:50 London / 13:50 Paris. Lucas Pluvinage of Tarides and Jonathan Ludlam of OCaml Labs Consultancy will discuss the process of generating documentation for every version of every package that can be built from the Opam repository and present it as a single coherent website that's continuously updated as new packages are released and old packages are updated. They will address the challenges of caching, handling different compiler versions, and incompatible libraries. The process has been implemented as an OCurrent pipeline named ocaml-docs-ci and is already available on Github. It has been used to produce the documentation of more than 10,000 package versions, generating 2.5M HTML pages. That's 38GB of artifacts!

After a relaxing lunch, come back for Experiences with Effects at 15:30 London / 16:30 Paris. Join OCaml Labs and Tarides engineers Thomas Leonard, Craig Ferguson, Patrick Ferris, Sadiq Jaffer, Tom Kelly, KC Sivaramakrishnan, and Anil Madhavapeddy as they talk about an exciting, experimental branch of Multicore OCaml that adds support for effect handlers. In this presentation, they'll discuss their experiences with effects, both from converting existing code and from writing new code. They discovered that converting the Angstrom parser from a callback style to effects greatly simplified the code while also improving performance and reducing allocations. Their experimental Eio library uses effects that allows writing concurrent code in direct style, without the need for monads (as found in Lwt or Async).

Enjoy a full day of OCaml innovation and get to know some of our talented engineers better by joining Tarides, OCaml Labs, and Segfault Systems at ICFP on Friday, 27 August 2021. See you there!

Regular CI systems are optimised for workloads that do not require stable performance over time. This makes them unsuitable for running performance benchmarks.

current-bench provides a predictable environment for performance benchmarks and a UI for analysing results over time. Similar to a CI system, it runs on pull requests and branches which allows performance to be analysed and compared. It can currently be enabled as an app on GitHub repositories with zero configuration. Several public repo…

Regular CI systems are optimised for workloads that do not require stable performance over time. This makes them unsuitable for running performance benchmarks.

current-bench provides a predictable environment for performance benchmarks and a UI for analysing results over time. Similar to a CI system, it runs on pull requests and branches which allows performance to be analysed and compared. It can currently be enabled as an app on GitHub repositories with zero configuration. Several public repositories are runningcurrent-bench, including Irmin and Dune. We plan to enable it on more projects in the future.

In this article, we give a technical overview of current-bench, showing how results are collected and analysed, requirements for using it and how we built the infrastructure for stable benchmarks. We also describe future work that would allow more OCaml projects to run current-bench.

Introduction

For performance critical software, we must run benchmarks to ensure that there's no regression. Running benchmarks before the user submits their pull request is tedious, and since every user might have a different machine, you can't be sure if the benchmarks performed actually improved or regressed performance.

Our current-bench aims to solve this problem by providing a stable benchmarking platform that runs every time the user submits a pull request and compares the result to the benchmarks on the main branch. As current-bench is zero-configuration, users can enroll their repository to run benchmarks with ease. This current-bench has helped projects ensure that regression doesn't happen, so you can merge code with more confidence.

Architecture

Benchmarking Pipeline

As shown in Figure 1 (above), the benchmarking infrastructure uses ocurrent¹, an embedded Domain Specific Language to write a pipeline. The ocurrent command computes the build incrementally and helps with static analysis. Whenever a pull request is opened on a repository monitored by current-bench, a POST request is sent to the server running the pipeline. The pipeline fetches the head commit on the pull request and uses Docker to compile the code, and then it runs the make bench command inside the generated Docker image.

The pipeline runs on a single node, and the process is pinned to a single core to ensure there's no contention of resources when running the benchmarks. Once finished, the raw JSON result is stored in a Postgres database, which the frontend can query using a GraphQL API, as shown in Figure 2 below.

The frontend supports historical navigation and provides comparison with the default branch. It allows users to select a pull request of which they want to see the graphs. The graphs display the individual result of the head commit and the comparison with the commits on the default branch. The frontend permits users to select the historical interval when they want to compare benchmarks, and it also shows the standard deviation. Once the benchmarks have run successfully, the pipeline sets the pull request status to the frontend URL. Then the user can look at the graphs.

Hardware Optimisation

Our current-bench uses the hardware optimisations developed for OCaml multicore compiler benchmarks (presented at ICFP OCaml Workshop 2019) with a few modifications to allow the benchmarks to run inside Docker containers. To get stable performance, we configured the kernel to isolate some of the CPU cores. Linux then avoids scheduling other user processes automatically. We also disabled IRQ handling and power saving.

The container that runs the benchmark is pinned to one of the isolated cores. Since I/O operations can make the benchmarks less stable, we use an in-memory tmpfs partition in /dev/shm for all storage. For NUMA enabled systems, we configure this partition to be allocated on the NUMA node of the isolated core. The pipeline disables ASLR inside the container automatically, which is normally blocked by the default Docker seccomp profile, so we have modified the profile to allow the personality(2) syscall.

Enrolling a repository

To enroll a repository, you need to ensure the following:

• Enable the ocaml-benchmarks GitHub app for your repository.
• The repository needs a bench Makefile target. This is triggered from the current-bench pipeline.
• The output of the make bench target is JSON, which can be parsed by the pipeline and displayed by the frontend.

Future work

Anyone who wants to roll out a continuous, zero-configured benchmarking infrastructure can set up the current-bench infrastructure. In the future, we want to scale current-bench by isolating cores on multiple machines and adding a scheduler to ensure that benchmarks use only one core at a time per machine. We plan to add support for different benchmarking libraries that repositories can use—for example, we currently support repositories using bechamel. We also aim to make the adoption of current-bench easier by adding a conversion library that can convert any benchmark output into output parseable by current-bench. We intend to add support for quick and slow benchmarks, which would allow users to have faster feedback loops on pull requests while ensuring they can still run more extensive, time consuming benchmarks to see the performance.

Thank you for reading! You can check out the implementation for current-bench here!

Feedback on this post is welcomed on Discuss!

We are happy to announce the release of opam 2.1.0.

Many new features made it in (see the pre-release changelogs or release notes for the details), but here are a few highlights.

What's new in opam 2.1?

• Integration of system dependencies (formerly the opam-depext plugin), increasing their reliability as it integrates the solving step
• Creation of lock files for reproducible installations (formerly the opam-lock plugin)
• Switch invariants, replacing …

Feedback on this post is welcomed on Discuss!

We are happy to announce the release of opam 2.1.0.

Many new features made it in (see the pre-release changelogs or release notes for the details), but here are a few highlights.

What's new in opam 2.1?

• Integration of system dependencies (formerly the opam-depext plugin), increasing their reliability as it integrates the solving step
• Creation of lock files for reproducible installations (formerly the opam-lock plugin)
• Switch invariants, replacing the "base packages" in opam 2.0 and allowing for easier compiler upgrades
• Improved options configuration (see the new option and expanded var sub-commands)
• CLI versioning, allowing cleaner deprecations for opam now and also improvements to semantics in future without breaking backwards-compatibility
• opam root readability by newer and older versions, even if the format changed
• Performance improvements to opam-update, conflict messages, and many other areas

Seamless integration of System dependencies handling (a.k.a. "depexts")

opam has long included the ability to install system dependencies automatically via the depext plugin. This plugin has been promoted to a native feature of opam 2.1.0 onwards, giving the following benefits:

• You no longer have to remember to run opam depext, opam always checks depexts (there are options to disable this or automate it for CI use). Installation of an opam package in a CI system is now as easy as opam install ., without having to do the dance of opam pin add -n/depext/install. Just one command now for the common case!
• The solver is only called once, which both saves time and also stabilises the behaviour of opam in cases where the solver result is not stable. It was possible to get one package solution for the opam depext stage and a different solution for the opam install stage, resulting in some depexts missing.
• opam now has full knowledge of depexts, which means that packages can be automatically selected based on whether a system package is already installed. For example, if you have neither MariaDB nor MySQL dev libraries installed, opam install mysql will offer to install conf-mysql and mysql, but if you have the MariaDB dev libraries installed, opam will offer to install conf-mariadb and mysql.

Hint: You can set OPAMCONFIRMLEVEL=unsafe-yes or --confirm-level=unsafe-yes to launch non interactive system package commands.

opam lock files and reproducibility

When opam was first released, it had the mission of gathering together scattered OCaml source code to build a community repository. As time marches on, the size of the opam repository has grown tremendously, to over 3000 unique packages with over 19500 unique versions. opam looks at all these packages and is designed to solve for the best constraints for a given package, so that your project can keep up with releases of your dependencies.

While this works well for libraries, we need a different strategy for projects that need to test and ship using a fixed set of dependencies. To satisfy this use-case, opam 2.0.0 shipped with support for using project.opam.locked files. These are normal opam files but with exact versions of dependencies. The lock file can be used as simply as opam install . --locked to have a reproducible package installation.

With opam 2.1.0, the creation of lock files is also now integrated into the client:

• opam lock will create a .locked file for your current switch and project, that you can check into the repository.
• opam switch create . --locked can be used by users to reproduce your dependencies in a fresh switch.

This lets a project simultaneously keep up with the latest dependencies (without lock files) while providing a stricter set for projects that need it (with lock files).

Hint: You can export the full configuration of a switch with opam switch export new options, --full to have all packages metadata included, and --freeze to freeze all VCS to their current commit.

Switch invariants

In opam 2.0, when a switch is created the packages selected are put into the “base” of the switch. These packages are not normally considered for upgrade, in order to ease pressure on opam's solver. This was a much bigger concern early on in opam 2.0's development, but is less of a problem with the default mccs solver.

However, it's a problem for system compilers. opam would detect that your system compiler version had changed, but be unable to upgrade the ocaml-system package unless you went through a slightly convoluted process with --unlock-base.

In opam 2.1, base packages have been replaced by switch invariants. The switch invariant is a package formula which must be satisfied on every upgrade and install. All existing switches' base packages could just be expressed as package1 & package2 & package3 etc. but opam 2.1 recognises many existing patterns and simplifies them, so in most cases the invariant will be "ocaml-base-compiler" {= "4.11.1"}, etc. This means that opam switch create my_switch ocaml-system now creates a switch invariant of "ocaml-system" rather than a specific version of the ocaml-system package. If your system OCaml package is updated, opam upgrade will seamlessly switch to the new package.

This also allows you to have switches which automatically install new point releases of OCaml. For example:

opam switch create ocaml-4.11 --formula='"ocaml-base-compiler" {>= "4.11.0" & < "4.12.0~"}' --repos=old=git+https://github.com/ocaml/opam-repository#a11299d81591
opam install utop

Creates a switch with OCaml 4.11.0 (the --repos= was just to select a version of opam-repository from before 4.11.1 was released). Now issue:

opam repo set-url old git+https://github.com/ocaml/opam-repository
opam upgrade

and opam 2.1 will automatically offer to upgrade OCaml 4.11.1 along with a rebuild of the switch. There's not yet a clean CLI for specifying the formula, but we intend to iterate further on this with future opam releases so that there is an easier way of saying “install OCaml 4.11.x”.

Hint: You can set up a default invariant that will apply for all new switches, via a specific opamrc. The default one is ocaml >= 4.05.0

Configuring opam from the command-line

Configuring opam is not a simple task: you need to use an opamrc at init stage, or hack global/switch config file, or use opam config var for additional variables. To ease that step, and permit a more consistent opam config tweaking, a new command was added : opam option.

For example:

• opam option download-jobs gives the global download-jobs value (as it exists only in global configuration)
• opam option jobs=6 --global will set the number of parallel build jobs opam is allowed to run (along with the associated jobs variable)
• opam option depext-run-commands=false disables the use of sudo for handling system dependencies; it will be replaced by a prompt to run the installation commands
• opam option depext-bypass=m4 --global bypass m4 system package check globally, while opam option depext-bypass=m4 --switch myswitch will only bypass it in the selected switch

The command opam var is extended with the same format, acting on switch and global variables.

Hint: to revert your changes use opam option <field>=, it will take its default value.

CLI Versioning

A new --cli switch was added to the first beta release, but it's only now that it's being widely used. opam is a complex enough system that sometimes bug fixes need to change the semantics of some commands. For example:

• opam show --file needed to change behaviour
• The addition of new controls for setting global variables means that the opam config was becoming cluttered and some things want to move to opam var
• opam switch install 4.11.1 still works in opam 2.0, but it's really an OPAM 1.2.2 syntax.

Changing the CLI is exceptionally painful since it can break scripts and tools which themselves need to drive opam. CLI versioning is our attempt to solve this. The feature is inspired by the (lang dune ...) stanza in dune-project files which has allowed the Dune project to rename variables and alter semantics without requiring every single package using Dune to upgrade their dune files on each release.

Now you can specify which version of opam you expected the command to be run against. In day-to-day use of opam at the terminal, you wouldn't specify it, and you'll get the latest version of the CLI. For example: opam var --global is the same as opam var --cli=2.1 --global. However, if you issue opam var --cli=2.0 --global, you will told that --global was added in 2.1 and so is not available to you. You can see similar things with the renaming of opam upgrade --unlock-base to opam upgrade --update-invariant.

The intention is that --cli should be used in scripts, user guides (e.g. blog posts), and in software which calls opam. The only decision you have to take is the oldest version of opam which you need to support. If your script is using a new opam 2.1 feature (for example opam switch create --formula=) then you simply don't support opam 2.0. If you need to support opam 2.0, then you can't use --formula and should use --packages instead. opam 2.0 does not have the --cli option, so for opam 2.0 instead of --cli=2.0 you should set the environment variable OPAMCLI to 2.0. As with all opam command line switches, OPAMCLI is simply the equivalent of --cli which opam 2.1 will pick-up but opam 2.0 will quietly ignore (and, as with other options, the command line takes precedence over the environment).

Note that opam 2.1 sets OPAMCLI=2.0 when building packages, so on the rare instances where you need to use the opam command in a package build: command (or in your build system), you must specify --cli=2.1 if you're using new features.

Since 2.1.0~rc2, CLI versioning applies to opam environment variables. The previous behavior was to ignore unknown or wrongly set environment variable, while now you will have a warning to let you know that the environment variable won't be handled by this version of opam.

To ensure not breaking compatibility of some widely used deprecated options, a default CLI is introduced: when no CLI is specified, those deprecated options are accepted. It concerns opam exec and opam var subcommands.

There's even more detail on this feature in our wiki. We're hoping that this feature will make it much easier in future releases for opam to make required changes and improvements to the CLI without breaking existing set-ups and tools.

Note: For opam libraries users, since 2.1 environment variable are no more loaded by the libraries, only by opam client. You need to load them explicitly.

opam root portability

opam root format changes during opam life-cycle, new field are added or removed, new files are added ; an older opam version sometimes can no longer read an upgraded or newly created opam root. opam root format has been updated to allow new versions of opam to indicate that the root may still be read by older versions of the opam libraries. A plugin compiled against the 2.0.9 opam libraries will therefore be able to read information about an opam 2.1 root (plugins and tools compiled against 2.0.8 are unable to load opam 2.1.0 roots). It is a read-only best effort access, any attempt to modify the opam root fails.

Hint: for opam libraries users, you can safely load states with OpamStateConfig load functions.

Tremendous thanks to all involved people, who've developed, tested & retested, helped with issue reports, comments, feedback...

Try it!

In case you plan a possible rollback, you may want to first backup your ~/.opam directory.

The upgrade instructions are unchanged:

1. Either from binaries: run

   bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.1.0"

   or download manually from the Github "Releases" page to your PATH.

2. Or from source, manually: see the instructions in the README.

You should then run:

opam init --reinit -ni

Feedback on this post is welcomed on Discuss!

We are pleased to announce the minor release of opam 2.0.9.

This new version contains some back-ported fixes.

New features

• Back-ported ability to load upgraded roots read-only; allows applications compiled with opam-state 2.0.9 to load a root which has been upgraded to opam 2.1 [#4636]
• macOS sandbox now supports OPAM_USER_PATH_RO for adding a custom read-only directory to the sandbox [#4589, #4609]
• OPAMROOT and OPAMSWITCH now reflect the --root and --…

Feedback on this post is welcomed on Discuss!

We are pleased to announce the minor release of opam 2.0.9.

This new version contains some back-ported fixes.

New features

• Back-ported ability to load upgraded roots read-only; allows applications compiled with opam-state 2.0.9 to load a root which has been upgraded to opam 2.1 [#4636]
• macOS sandbox now supports OPAM_USER_PATH_RO for adding a custom read-only directory to the sandbox [#4589, #4609]
• OPAMROOT and OPAMSWITCH now reflect the --root and --switch parameters in the package build [#4668]
• When built with opam-file-format 2.1.3+, opam-format 2.0.x displays better errors for newer opam files [#4394]

Bug fixes

• Linux sandbox now mounts host $TMPDIR read-only, then sets the sandbox $TMPDIR to a new separate tmpfs. Hardcoded /tmp access no longer works if TMPDIR points to another directory [#4589]
• Stop clobbering DUNE_CACHE in the sandbox script [#4535, fixing ocaml/dune#4166]
• Ctrl-C now correctly terminates builds with bubblewrap; sandbox now requires bubblewrap 0.1.8 or later [#4400]
• Linux sandbox script no longer makes PWD read-write on remove actions [#4589]
• Lint W59 and E60 no longer trigger for packages flagged conf [#4549]
• Reduce the length of temporary file names for pin caching to ease pressure on Windows [#4590]
• Security: correct quoting of arguments when removing switches [#4707]
• Stop advertising the removed option --compiler when creating local switches [#4718]
• Pinning no longer fails if the archive's opam file is malformed [#4580]
• Fish: stop using deprecated ^ syntax to fix support for Fish 3.3.0+ [#4736]

1. From binaries: run

   bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.0.9"

   or download manually from the Github "Releases" page to your PATH. In this case, don't forget to run opam init --reinit -ni to enable sandboxing if you had version 2.0.0~rc manually installed or to update you sandbox script.

2. From source, using opam:

   opam update; opam install opam-devel

   (then copy the opam binary to your PATH as explained, and don't forget to run opam init --reinit -ni to enable sandboxing if you had version 2.0.0~rc manually installed or to update your sandbox script)

3. From source, manually: see the instructions in the README.

We hope you enjoy this new minor version, and remain open to bug reports and suggestions.

Tarides takes great pride in a diverse workforce and strives to continue bringing talented people to its team from around the globe. This is why Sonja Heinze, a Tarides software engineer, and the Head of HR, Héloïse Lutton, will attend WomenHack, an online event dedicated to recruiting more women into the tech world. They're participating not only to present Tarides to the Women In Tech community, but to also network and possibly find new talented programmers to join our growing team.

The Wome…

Tarides takes great pride in a diverse workforce and strives to continue bringing talented people to its team from around the globe. This is why Sonja Heinze, a Tarides software engineer, and the Head of HR, Héloïse Lutton, will attend WomenHack, an online event dedicated to recruiting more women into the tech world. They're participating not only to present Tarides to the Women In Tech community, but to also network and possibly find new talented programmers to join our growing team.

The WomenHack event has a unique setup similar to 'speed dating,' but for prospective jobs. Each candidate is paired with a company, and they have 5 minutes to chat before moving on to the next company. This "rapid interview" is both fun and efficient for all involved, and it ensures each company will meet several talented candidates from diverse backgrounds, which increases their chance of finding that perfect fit for an open position!

From their website:

WomenHack is a community that empowers women in tech through events, jobs, and reviews. We aim to create a more inclusive and diverse workplace for all. Our diversity recruiting events target some of the most talented women in tech which include software developers, designers, and product talent. 

Join us tomorrow, July 21st, at WomenHack! You can get your ticket here.