OCaml Changelog

OBuilder workers for FreeBSD have been updated to FreeBSD 14.2. See ocurrent/freebsd-infra

From today, ocaml-ci and opam-repo-ci will test against FreeBSD 14.2 rather than 14.1.

Following a post on discuss.ocaml.org, there has been a new release of ocurrent/ocaml-version that moves the minimum version of OCaml, considered as recent, from 4.02 to 4.08.

let recent = [ v4_08; v4_09; v4_10; v4_11; v4_12; v4_13; v4_14; v5_0; v5_1; v5_2; v5_3 ]

This change has far reaching side effects as OCaml-CI, opam-repo-ci, Docker base image builder among other things, use this to determine the set of versions of OCaml to test against. As these services are updated, testing on the old releases will be removed.

Tarides' OCaml Infrastructure team has been hard at work over the last few months, focusing on improving the stability, reliability, and user experience of the OCaml ecosystem's core infrastructure. We are preparing for more substantive changes to come. We will have more fruit from this work to share soon, but updates on our efforts are overdue and we wanted to get you all caught up.

Platform Support

We've expanded and updated our platform support to keep pace with the evolving OCaml ecosystem:

• As part of the compiler release readiness, we've added support for OCaml 5.2 and the upcoming 5.3, currently in alpha.
• We're now publishing Windows Server 2022 Docker images to the OCaml Dockerhub. See for instance windows-server-msvc-ltsc2022-ocaml-4.14
• Dropped support for the post-LTS distro versions Debian 10 and 11, focusing our resources on the OCaml compiler's Tier 1 platform.

These updates ensure that our systems continue to support OCaml developers on a broad range of systems, in particular by testing package publication to the opam repository and publishing Docker images.

Opam Repository Publishing Experience

We've made improvements to enhance the experience of publishing packages to the opam repository:

• Implemented a linting check for package maintainer email addresses, ensuring package maintainers can be reliably contacted when needed.
• Mitigated solver timeout errors in the opam repo CI system, reducing frustrating delays in the package publishing process.
• Fixed an uncaught exception that sometimes occurred during linting checks in the opam repo CI.
• Improved error reporting and added instructions for running lint checks locally, making it easier for package authors to identify and resolve issues before submitting to the repository.
• We are generating documentation of the platforms under test in opam CI, allowing package authors a clear overview of the extensive build matrix we are providing.

We hope these changes will make package publication experience smoother. We're committed to improving the experience for newcomers and existing repository contributors alike, stay tuned for more improvements!

Security and Reliability

Maintaining the security and reliability of the OCaml infrastructure remains a top priority:

• Updated our systems to address CVE-2024-6387.
• We maintained 99.9% availability for the opam repo CI over the last two months, providing a stable and reliable service for the OCaml community.

Forging Ahead

Much has been done but much more is in the works! Expect updates more regularly going forward, and don't hesitate to reach out on ocaml/infrastructure/issues or discuss.ocaml.org if you have questions, requests, or ideas!

About six months ago, opam-repo-ci (opam.ci.ocaml.org) suffered from a lack of system memory (issue 220) which caused it to be moved to the machine hosting ocaml-ci (ocaml.ci.dev).

Subsequently, that machine suffered from BTRFS volume corruption (issue 51). Therefore, we moved both services to a larger new server. The data was efficiently migrated using BTRFS tools: btrfs send | btrfs receive.

Since the move, we have seen issues with BTRFS metadata. Plus, we have suffered from a build-up of subvolumes, as reported by other users: Docker gradually exhausts disk space on BTRFS.

Unfortunately, both services went down on Friday evening (issue 85). Analysis showed over 500 BTRFS subvolumes, a shortage of metadata space, and insufficient space to perform a BTRFS rebalance.

Returning to the original configuration of splitting the ci.dev and OCaml.org services, they have been moved onto new and separate hardware. The underlying filesystem is now a RAID1-backed ext4, formatted with -i 8192 in order to ensure the availability of sufficient inodes. Docker uses Overlayfs. RSYNC was used to copy the databases and logs from the old server. This change should add resilience and has doubled the capacity for storing history logs.

OBuilder workers for both Intel and Apple Silicon have been updated from macOS Ventura to macOS Sonoma 14.1.1.

From today, ocaml-ci and opam-repo-ci will test against Sonoma rather than Ventura.

Previously the infrastructure team had made FreeBSD available for opam-repo-ci. Now we can announce that the same support has been added to ocaml-ci, giving coverage for both OCaml 4.14 and the new OCaml 5.1 release. opam-repo-ci has also been upgraded to support OCaml 5.1. We aim to support both 4.14 as the Long Term Support release and the latest 5.* release.

Additionally an opam-health-check instance has been setup to provide continuous checking of opam repository packages against FreeBSD 13.2 x86_64 for both the 4.14 and 5.1 releases of OCaml. This will allow the community to check whether packages work on FreeBSD and provide fixes to opam-repository that will then get tested on FreeBSD. Closing the loop and giving the community the tools to support OCaml on FreeBSD effectively.

We hope the community finds the FreeBSD support useful.

OBuilder now supports FreeBSD which has allowed opam-repo-ci to be extended to test against it. Currently, we are testing Opam 2.1.5 with OCaml 4.14.1 and OCaml 5.0 with FreeBSD 13.2 (AMD64).

The OCaml infrastructure team is going to move to Debian 12 as the main distribution from Debian 11. We will continue to provide Debian 11 and 10 images while they are supported, dropping Debian 10 when it reaches end of life in 2024-06-30. In addition to these changes we are deprecating Ubuntu 18.04, Alpine 3.16/17, OL7, OpenSuse 15.2 distributions as the have reached end of life. We strongly recommend updating to a newer version if you are still using them.

Please get in touch on https://github.com/ocaml/infrastructure/issues if you have questions or requests for additional support.

The server toxis where Opam-Repo-CI and OCaml-CI were deployed suffered hardware difficulties yesterday, resulting in BTRFS filesystem corruption and memory issues. These issues are tracked on ocaml/infrastructure#51. Services were restored temporarily using a spare spinning disk, but we continued to see ECC memory issues.

All services have now been redeployed on new ARM64 hardware. We retained the databases for Prometheus, OCaml-CI and Opam-Repo-CI, but unfortunately, older job logs have been lost.

The external URLs for these services are unchanged.

The Grafana dashboards at status.ci3.ocamllabs.io and status.ci.ocamllabs.io have been merged into a single dashboard at status.ci.ocamllabs.io. HTTP redirects are in place for the old addresses.

Due to operational difficulties, opam.ci.ocaml.org has temporarily moved to toxis.caelum.ci.dev. An HTTP redirect is in place. Please do not update your bookmarks.

This issue is being tracked on GitHub ocurrent/opam-repo-ci#220.

check.ocamllabs.io has moved to check.ci.ocaml.org. Furthermore, HTTP connections are now upgraded to HTTPS.

There is an HTTP 301 permanent redirect in place for the old address.

New Ubuntu 23.04 and Fedora 38 distributions have been added to docker base image builder. Following their respective releases:

• Ubuntu 23.04 was released 20 April 2023
• Fedora 38 was released 18 April 2023 These images started building from Apr 25th 2023 and have been pushed to https://hub.docker.com/r/ocaml/opam with a full range of OCaml versions and variants. As always the status of the images can be viewed on images.ci.ocaml.org.

Alongside the Linux updates base images containing OCaml 5.1 have also been published for supported Linux platforms. Following the OCaml 5.1 Alpha release announcement on discuss.ocaml.org. Enjoy and please report any issues on ocurrent/docker-base-images/issues.

Due to maintenance operations on Tuesday 11th April between 7:30am and 11:00am (UTC+1), the build system will be running at approximately 50% capacity. You may experience some build delays in Opam Repo CI and OCaml CI.

Thank you for your patience and understanding during this time.

The intention is to retire the ocamllabs.io domain. Therefore any services using the domain will be redirected. From today, the Web UI for Opam Repo CI is available on both opam-ci.ci3.ocamllabs.io and opam.ci.ocaml.org with the service available at both opam-repo-ci.ci3.ocamllabs.io and opam-repo.ci.ocaml.org. In time, the ocamllabs.io sites will issue HTTP 301 permanent redirect messages.

Previously, opam.ci.ocaml.org targetted a web server which issued an HTTP 302 redirect to opam.ci3.ocamllabs.io. This redirection has been removed. opam.ci.ocaml.org points to the actual site.

As previously announced, "opam-repository-mingw" is no longer receiving updates.

We're actively working on getting the Windows compiler packages into ocaml/opam-repository. There are two issues which are taking (me) a little while to finish solving, but more on that further below.

In the gap - of hopefully only a month or so - for this being ready, there's is an issue that new releases are of course not available when opam-repository-mingw is being used with ocaml/setup-ocaml@v2 GitHub actions workflows. I'm hoping here to set out what's happening, and what steps you may need to take to keep your GitHub Actions Windows workflows running smoothly over the next few months.

What's happening right now?

We've updated setup-ocaml to use ocaml-opam/opam-repository-mingw instead of fdopen/opam-repository-mingw (see ocaml/setup-ocaml#651). This clone has been augmented with:

• OCaml 4.14.1 packages, in the same style as the 4.14.0 forked packages (the "pre-compiled" package variants exist, but they're not pre-compiled)
• Changes to the constraints for existing packages only

If you're using setup-ocaml in its default configuration, you should notice no change except that 4.14.x builds should now use 4.14.1 and the initial build will be a little slower as it builds from sources (GitHub Actions caching will then take over for subsequent runs).

For new releases of packages, it's necessary to add opam-repository to the repositories selections for the switches. It's important that opam-repository is at a lower priority than opam-repository-mingw for existing packages, so it's better to use these lines in your ocaml/setup-ocaml@v2 step than to issue opam repo add --rank=1000 later:

uses: ocaml/setup-ocaml@v2
with:
  opam-repositories: |
    opam-repository-mingw: https://github.com/ocaml-opam/opam-repository-mingw.git#sunset
    default: https://github.com/ocaml/opam-repository.git

What do I do when things are broken?

There's an issue tracker on ocaml-opam/opam-repository-mingw, and this is a very good place to start.

If a version of a package isn't building, there are three possible remedies:

• Previous versions of the package may have carried non-upstreamed patches in opam-repository-mingw. opam-repository's policy is not to carry such patches. In this case, the package actually doesn't work on Windows.
  • opam-repository should be updated to have os != "win32" added to the available field for the package
  • An issue on the package's upstream repo should be opened highlighting the need to upstream patches (or even a pull request with them!)
  • The patches in opam-repository-mingw make changes which may not necessarily be accepted/acceptable upstream in their current form, so the issue may be a better starting point than simply taking a patch and opening a pull request for it (for example, the utop package contains patches which may require further work and review)
• The package relies on environment changes in "OCaml for Windows". For example, the Zarith package works in "OCaml for Windows" because the compiler packages unconditionally set the CC environment variable. This change is both not particularly desirable change to upstream (it is very confusing, for example, when working on the compiler itself) and also extremely difficult to upstream, so the fix here is instead to change the package's availability with (os != "win32" | os-distribution = "cygwinports") and constrain away OCaml 5 on Windows ("ocaml" {< "5.0" | os != "win32"})
• Package constraints on existing packages need updating in ocaml-opam/opam-repository-mingw. For example, the release of ppxlib 0.29 required some existing packages to have upperbounds added.

What about OCaml 5.0.0?

OCaml 5.0.0 was released with support for the mingw-w64 port only, however, there's a quite major bug which wasn't caught by OCaml's testsuite, but is relatively easily triggered by opam packages. I've previously announced how to add OCaml 5 to a workflow. For the time being, the packages for OCaml 5 aren't automatically made available.

What's next?

The ultimate goal is to be using an upstream build of opam.exe with ocaml/opam-repository, just as on Unix. Once opam 2.2 is generally available (we're aiming for an alpha release at the end of March) and the compiler packages in opam-repository support the Windows builds, we will recommend stopping use of opam-repository-mingw completely. The default in setup-ocaml won't change straight away, since that risks breaking existing workflows.

With upstream compiler support, we'll be able to extend some of the existing bulk build support already being done for Linux to Windows and start to close the gap of patches in opam-repository-mingw.

Windows compiler packages

I mentioned earlier the problems with moving the compiler packages into opam-repository, and just for general interest this elaborates on them.

The first issue affects the use of the Visual Studio port ("MSVC") and is a consequence of the somewhat strange way that the C compiler is added to the environment when using the Visual Studio C compiler. "OCaml for Windows" (as well as Diskuv) use a wrapper command (it's ocaml-env in "OCaml for Windows" and with-dkml in Diskuv). Those commands are Windows-specific, which is an issue for upstream opam. There's an alternate way which sets the environment variables in a more opam-like way. Doing it that way, though, requires an improvement to opam's environment handling which is in opam 2.2, otherwise there's an easy risk of "blowing" the environment.

The second issue is selecting the C compiler. On Unix, this is easy with ocaml-base-compiler because there is only one "system" C compiler. Windows has two ports of OCaml, and the configuration requires it to be explicitly selected. That requires input from the user on switch creation for a Windows switch.

"OCaml for Windows" solves this by packaging the Windows compilers with the variant name appended, just as opam-repository used to, so ocaml-variants.4.14.1+mingw64 selects the the mingw-w64 port and ocaml-variants.4.14.1+msvc64 selects the MSVC64 port. The problem, as we already had in opam-repository, is that this adds 4 packages for each release of OCaml in ocaml-variants, and leads to a combinatorial explosion when we start considering flambda and other relevant compiler options.

opam-repository switched to using the ocaml-option- packages to solve the combinatorial explosion which was already present in opam-repository. The demonstration repo for OCaml 5 on Windows is already using an adapted version of this so that ocaml-option-mingw selects the mingw-w64 port (by default 64-bit, with ocaml-option-32bit then selecting the 32-bit port).

This work is all in progress and being tested alongside changes in opam 2.2 to support the depext experience on Windows. The only reason that's not being upstreamed piecemeal is that changes to the compiler packages in opam-repository trigger switch rebuilds all over the world, so we don't want to that until we're sure that the packages are correct. The intention is to do this around the time of the alpha release of opam 2.2, once the work in opam itself has settled down.

Thanks for getting to the end, and happy Windows CI testing!

Discuss this post further on the forums.

Following @avsm post on discuss.ocaml.org, we are pleased to announce that DNS names have now been switched over.

We are moving the opam.ocaml.org 2 servers between hosting providers, and wanted to give everyone clear notice that this happening. Over the next 24-48 hours, if you notice any unexpected changes in the way your opam archives work (for example, in your CI or packaging systems), then please do let us know immediately, either here or in ocaml/infrastructure#19 2.

The reason for the move is to take advantage of Scaleway’s generous sponsorship of ocaml.org, and to use their energy efficient renewable infrastructure 4 for our machines.

This also marks a move to building the opam website via the ocurrent 1 infrastructure, which leads to faster and more reliable updates to the hosted package archives (see here for the service graph and build logs 4). There are also now multiple machines behind the opam.ocaml.org 2 DNS (via round-robin DNS), and this makes it easier for us to publish the archives to a global CDN in the future.

But in the very short term, if something explodes unexpectedly, please do let us know.

watch.ocaml.org has been updated to run as a Docker service stack rather than via docker-compose. This change allowed an OCurrent pipeline to monitor the Docker repository and update the image via docker service update when a new version is available.

We have several other services updated via OCurrent: deploy.ci.ocaml.org

In OCurrent, we can create a schedule node that triggers every seven days and invokes a docker pull, yielding the current image SHA. If this has changed, run docker service update with the new image.

  let peertube =
     let weekly = Current_cache.Schedule.v ~valid_for:(Duration.of_day 7) () in
     let image = Cluster.Watch_docker.pull ~schedule:weekly "chocobozzz/peertube:production-bullseye" in
     Cluster.Watch_docker.service ~name:"infra_peertube" ~image ()

The deployment is achieved through an Ansible Playbook. Further details are available here.

The second part of the update was to improve the visibility of the backups for watch.ocaml.org. As noted previously, these use Tarsnap running monthly via CRON.

For this, a new plugin was added to OCurrent called ocurrent_ssh. This plugin allows arbitrary SSH commands to be executed as part of an OCurrent pipeline.

Again using a schedule node, the Current_ssh.run node will be triggered on a 30-day schedule, and the logs for each run will be available on deploy.ci.ocaml.org.

  let monthly = Current_cache.Schedule.v ~valid_for:(Duration.of_day 30) () in
   let tarsnap = Current_ssh.run ~schedule:monthly "watch.ocaml.org" ~key:"tarsnap" (Current.return ["./tarsnap-backup.sh"])

Some of our ocaml.org services such as <watch.ocaml.org> involve storing user uploaded content. We need a way to make sure these are backed up in case of information loss, and to date this has been adhoc (involving rsyncing to another machine).

We now have a Tarsnap account as tarsnap@ocaml.org, and it is first being used to store backups of the videos uploaded to the <watch.ocaml.org> service. We'll expand its use to other infrastructures that also have precious data.

Other suggestions for backup services are welcome. In general, we're looking for solutions that do not involve a lot of key management, and a reasonable amount of redundancy (but backing up across 2-3 other machines in different datacentres is probably sufficient).

Equinix is closing their data centre in Amsterdam where we currently host www.ocaml.org and staging.ocaml.org on an m1.xlarge.x86. These websites will be moved to virtual machines Scaleway cluster.