Feedback on this post is welcomed on Discuss!
On behalf of the opam team, it gives me great pleasure to announce the third beta release of opam 2.1. Don’t worry, you didn’t miss beta3 - we had an issue with a configure script that caused beta2 to report as beta3 in some instances, so we skipped to beta4 to avoid any further confusion!
We encourage you to try out this new beta release: there are instructions for doing so in our wiki. The instructions include taking a backup of your ~/.opam root as part of the process, which can be restored in order to wind back. Please note that local switches which are written to by opam 2.1 are upgraded and will need to be rebuilt if you go back to opam 2.0. This can either be done by removing _opam and repeating whatever you use in your build process to create the switch, or you can use opam switch export switch.export to backup the switch to a file before installing new packages. Note that opam 2.1 shouldn’t upgrade a local switch unless you upgrade the base packages (i.e. the compiler).
What’s new in opam 2.1?
- Switch invariants
- Improved options configuration (see the new
option and expanded var sub-commands)
- Integration of system dependencies (formerly the opam-depext plugin), increasing their reliability as it integrates the solving step
- Creation of lock files for reproducible installations (formerly the opam-lock plugin)
- CLI versioning, allowing cleaner deprecations for opam now and also improvements to semantics in future without breaking backwards-compatibility
- Performance improvements to opam-update, conflict messages, and many other areas
- New plugins: opam-compiler and opam-monorepo
Switch invariants
In opam 2.0, when a switch is created the packages selected are put into the “base” of the switch. These packages are not normally considered for upgrade, in order to ease pressure on opam’s solver. This was a much bigger concern early on in opam 2.0’s development, but is less of a problem with the default mccs solver.
However, it’s a problem for system compilers. opam would detect that your system compiler version had changed, but be unable to upgrade the ocaml-system package unless you went through a slightly convoluted process with --unlock-base.
In opam 2.1, base packages have been replaced by switch invariants. The switch invariant is a package formula which must be satisfied on every upgrade and install. All existing switches’ base packages could just be expressed as package1 & package2 & package3 etc. but opam 2.1 recognises many existing patterns and simplifies them, so in most cases the invariant will be "ocaml-base-compiler" {= 4.11.1}, etc. This means that opam switch create my_switch ocaml-system now creates a switch invariant of "ocaml-system" rather than a specific version of the ocaml-system package. If your system OCaml package is updated, opam upgrade will seamlessly switch to the new package.
This also allows you to have switches which automatically install new point releases of OCaml. For example:
opam switch create ocaml-4.11 --formula='"ocaml-base-compiler" {>= "4.11.0" & < "4.12.0~"}' --repos=old=git+https://github.com/ocaml/opam-repository#a11299d81591
opam install utop
Creates a switch with OCaml 4.11.0 (the --repos= was just to select a version of opam-repository from before 4.11.1 was released). Now issue:
opam repo set-url old git+https://github.com/ocaml/opam-repository
opam upgrade
and opam 2.1 will automatically offer to upgrade OCaml 4.11.1 along with a rebuild of the switch. There’s not yet a clean CLI for specifying the formula, but we intend to iterate further on this with future opam releases so that there is an easier way of saying “install OCaml 4.11.x”.
opam depext integration
opam has long included the ability to install system dependencies automatically via the depext plugin. This plugin has been promoted to a native feature of opam 2.1.0 onwards, giving the following benefits:
- You no longer have to remember to run
opam depext, opam always checks depexts (there are options to disable this or automate it for CI use). Installation of an opam package in a CI system is now as easy as opam install ., without having to do the dance of opam pin add -n/depext/install. Just one command now for the common case!
- The solver is only called once, which both saves time and also stabilises the behaviour of opam in cases where the solver result is not stable. It was possible to get one package solution for the
opam depext stage and a different solution for the opam install stage, resulting in some depexts missing.
- opam now has full knowledge of depexts, which means that packages can be automatically selected based on whether a system package is already installed. For example, if you have neither MariaDB nor MySQL dev libraries installed,
opam install mysql will offer to install conf-mysql and mysql, but if you have the MariaDB dev libraries installed, opam will offer to install conf-mariadb and mysql.
opam lock files and reproducibility
When opam was first released, it had the mission of gathering together scattered OCaml source code to build a community repository. As time marches on, the size of the opam repository has grown tremendously, to over 3000 unique packages with over 18000 unique versions. opam looks at all these packages and is designed to solve for the best constraints for a given package, so that your project can keep up with releases of your dependencies.
While this works well for libraries, we need a different strategy for projects that need to test and ship using a fixed set of dependencies. To satisfy this use-case, opam 2.0.0 shipped with support for using project.opam.locked files. These are normal opam files but with exact versions of dependencies. The lock file can be used as simply as opam install . --locked to have a reproducible package installation.
With opam 2.1.0, the creation of lock files is also now integrated into the client:
opam lock will create a .locked file for your current switch and project, that you can check into the repository.opam switch create . --locked can be used by users to reproduce your dependencies in a fresh switch.
This lets a project simultaneously keep up with the latest dependencies (without lock files) while providing a stricter set for projects that need it (with lock files).
CLI Versioning
A new --cli switch was added to the first beta release, but it’s only now that it’s being widely used. opam is a complex enough system that sometimes bug fixes need to change the semantics of some commands. For example:
opam show --file needed to change behaviour- The addition of new controls for setting global variables means that the
opam config was becoming cluttered and some things want to move to opam var
opam switch install 4.11.1 still works in opam 2.0, but it’s really an OPAM 1.2.2 syntax.
Changing the CLI is exceptionally painful since it can break scripts and tools which themselves need to drive opam. CLI versioning is our attempt to solve this. The feature is inspired by the (lang dune ...) stanza in dune-project files which has allowed the Dune project to rename variables and alter semantics without requiring every single package using Dune to upgrade their dune files on each release.
Now you can specify which version of opam you expected the command to be run against. In day-to-day use of opam at the terminal, you wouldn’t specify it, and you’ll get the latest version of the CLI. For example: opam var --global is the same as opam var --cli=2.1 --global. However, if you issue opam var --cli=2.0 --global, you will told that --global was added in 2.1 and so is not available to you. You can see similar things with the renaming of opam upgrade --unlock-base to opam upgrade --update-invariant.
The intention is that --cli should be used in scripts, user guides (e.g. blog posts), and in software which calls opam. The only decision you have to take is the oldest version of opam which you need to support. If your script is using a new opam 2.1 feature (for example opam switch create --formula=) then you simply don’t support opam 2.0. If you need to support opam 2.0, then you can’t use --formula and should use --packages instead. opam 2.0 does not have the --cli option, so for opam 2.0 instead of --cli=2.0 you should set the environment variable OPAMCLI to 2.0. As with all opam command line switches, OPAMCLI is simply the equivalent of --cli which opam 2.1 will pick-up but opam 2.0 will quietly ignore (and, as with other options, the command line takes precedence over the environment).
Note that opam 2.1 sets OPAMCLI=2.0 when building packages, so on the rare instances where you need to use the opam command in a package build: command (or in your build system), you must specify --cli=2.1 if you’re using new features.
There’s even more detail on this feature in our wiki. We’re still finalising some details on exactly how opam behaves when --cli is not given, but we’re hoping that this feature will make it much easier in future releases for opam to make required changes and improvements to the CLI without breaking existing set-ups and tools.
What’s new since the last beta?
- opam now uses CLI versioning (#4385)
- opam now exits with code 31 if all failures were during fetch operations (#4214)
opam install now has a --download-only flag (#4036), allowing opam’s caches to be primedopam init now advises the correct shell-specific command for eval $(opam env) (#4427)post-install hooks are now allowed to modify or remove installed files (#4388)- New package variable
opamfile-loc with the location of the installed package opam file (#4402)
opam update now has --depexts flag (#4355), allowing the system package manager to update too- depext support NetBSD and DragonFlyBSD added (#4396)
- The format-preserving opam file printer has been overhauled (#3993, #4298 and #4302)
- pins are now fetched in parallel (#4315)
os-family=ubuntu is now treated as os-family=debian (#4441)opam lint now checks that strings in filtered package formulae are booleans or variables (#4439)
and many other bug fixes as listed on the release page.
New Plugins
Several features that were formerly plugins have been integrated into opam 2.1.0. We have also developed some new plugins that satisfy emerging workflows from the community and the core OCaml team. They are available for use with the opam 2.1 beta as well, and feedback on them should be directed to the respective GitHub trackers for those plugins.
opam compiler
The opam compiler plugin can be used to create switches from various sources such as the main opam repository, the ocaml-multicore fork, or a local development directory. It can use Git tag names, branch names, or PR numbers to specify what to install.
Once installed, these are normal opam switches, and one can install packages in them. To iterate on a compiler feature and try opam packages at the same time, it supports two ways to reinstall the compiler: either a safe and slow technique that will reinstall all packages, or a quick way that will just overwrite the compiler in place.
opam monorepo
The opam monorepo plugin lets you assemble standalone dune workspaces with your projects and all of their opam dependencies, letting you build it all from scratch using only Dune and OCaml. This satisfies the “monorepo” workflow which is commonly requested by large projects that need all of their dependencies in one place. It is also being used by projects that need global cross-compilation for all aspects of a codebase (including C stubs in packages), such as the MirageOS unikernel framework.
Next Steps
This is anticipated to be the final beta in the 2.1 series, and we will be moving to release candidate status after this. We could really use your help with testing this release in your infrastructure and projects and let us know if you run into any blockers. If you have feature requests, please also report them on our issue tracker -- we will be planning the next release cycle once we ship opam 2.1.0 shortly.
See full changelog
- (*) Implemented CLI version compatibility layer [#4385 @rjbou]
- (*) Return code 31 (
Sync_error) instead of code 40
(Package_operation_error) when all failures happend during fetching
[#4416 @rjbou - fix #4214]
- (+) Add
--download-only flag [#4071 @Armael @rjbou - fix #4036]
- (+) Provide
opam update --depexts to request an update of the system package manager databases [#4379 @AltGr - fix #4355]
- Set OPAMCLI=2.0 during package action commands [#4492 @kit-ty-kate]
- Fix sandbox check on first
opam init [#4370 @rjbou - fix #4368]
- Print shell-appropriate eval command on
opam init [#4427 @freevoid]
- Fix
init script check in csh [#4482 @gahr]
- The
stdout of pre- and post-session hooks is now propagated to the user [#4382 @AltGr - fix #4359]
post-install hooks are now allowed to modify or remove installed files [#4388 @lefessan]- Add support for switch-specific pre/post sessions hooks [#4476 @rjbou - fix #4472]
- Ensure we don't advertise upgrades to hidden versions [#4477 @AltGr - fix #4432]
- Fix
opam remove --autoremove <PKG> to not autoremove unrelated packages [#4369 @AltGr - fix #4250 #4332]
- Fix cases where
opam remove -a could trigger conflicts in the presence of orphan packages [#4369 @AltGr - fix #4250 #4332]
- Fix
--update-invariant when removing or changing package name [#4360 @AltGr - fix #4353]
- Fix updates of the invariant with
--update-invariant [#4431 @AltGr]
- Fix cleanup of build dirs for version pinned packages [#4436 @rjbou - fix #4255]
- Fix
opamfile format upgrade on pinning [#4366 @rjbou - fix #4365]
- Fix
pin --show actually pinning [#4367 @rjbou - fix #4348]
- When several pins are needed, do their fetching in parallel [#4399 @rjbou - fix #4315]
- Don't cleanup VCS pin source directories [#4399 @rjbou]
- Fix
--working-dir with local switches [#4433 @rjbou]
- Add package variable
opamfile-loc, containing the location of installed package opam file [#4402 @rjbou]
- Fix
arch detection when using 32bit mode on ARM64 [#4462 @kit-ty-kate]
- Fix
arch detection of i486 [#4462 @kit-ty-kate]
- Skip loading the switch state for variable lookup when possible [#4428 @rjbou]
- Fix package variables display when no config file is found [#4428 @rjbou]
- Fix
opam option depext-bypass-=["XXX"] [#4428 @rjbou]
- Lint: add a check that strings in filtered package formula are Booleans or variables [#443 @rjbou - fix #4439]
- Fix handling of filename-encoded pkgname in opam files [#4401 @AltGr - fix ocaml-opam/opam-publish#107]
- Don't recompile when modifying the package flags [#4477 @AltGr]
- Add depext support for NetBSD and DragonFlyBSD [#4396 @kit-ty-kate]
- Fix depexts on OpenBSD, FreeBSD, and Gentoo: Allow short names and full name paths for ports-based systems [#4396 @kit-ty-kate]
- Handle the case where
os-family=ubuntu as os-family=debian [#4441 @alan-j-hu]
- Update opam's
opam files to 2.0 [#4371 @AltGr]
- Makefile: Add rule
custom-libinstall for opam-custom-install use [#4401 @AltGr]
- Use the archive caches when running
opam admin cache [#4384 @AltGr - fix #4352]
- Fix explosion of
opam admin check --cycles on repositories with huge cliques [#4392 @AltGr]
- Much improved format-preserving printer [#4298 #4302 @rjbou - fix #3993]
- Fix missing conflict message when trying to remove required packages [#4362 @AltGr]
- Fix the Z3 backend for upgrades [#4393 @AltGr]
- Fix cases where opam would wrongly complain about action cycles [#4358 @AltGr - fix #4357]
- Fix permission denied fallback for
openssl [#4449 @Blaisorblade - fix #4448]
- Add debug & verbose log for patch &
subst applications [#4464 @rjbou - fix #4453]
- Be more robust w.r.t. new caches updates when
--read-only is not used [#4467 @AltGr - fix #4354]
- Improved and extended tests [#4375 #4395 #4428 #4385 #4467 #4475 #4483 @emillon @rjbou @AltGr @freevoid @dra27]
- Switched to GitHub actions [#4463 @rjbou]