Feedback on this post is welcomed on Discuss!
We are happy to announce the release of opam 2.1.0.
Many new features made it in (see the pre-release
changelogs or release
notes for the details),
but here are a few highlights.
What's New in opam 2.1?
- Integration of system dependencies (formerly the
opam-depext plugin),
increasing their reliability as it integrates the solving step
- Creation of lock files for reproducible installations (formerly the
opam-lock
plugin)
- Switch invariants, replacing the "base packages" in opam 2.0 and allowing for
easier compiler upgrades
- Improved options configuration (see the new
option and expanded var subcommands)
- CLI versioning, allowing cleaner deprecations for opam now and also
improvements to semantics in future without breaking backwards-compatibility
opam root readability by newer and older versions, even if the format changed- Performance improvements to
opam-update, conflict messages, and many other
areas
Seamless Integration of System Dependencies Handling (a.k.a. "depexts")
Opam has long included the ability to install system dependencies automatically
via the depext plugin. This plugin
has been promoted to a native feature of opam 2.1.0 onwards, giving the
following benefits:
- You no longer have to remember to run
opam depext, as opam always checks
depexts (there are options to disable this or automate it for CI use).
Installation of an opam package in a CI system is now as easy as opam install ., without having to do the dance of opam pin add -n/depext/install. Just
one command now for the common case!
- The solver is only called once, which both saves time and also stabilises the
behaviour of opam in cases where the solver result is not stable. It was
possible to get one package solution for the
opam depext stage and a
different solution for the opam install stage, resulting in some depexts
missing.
- Opam now has full knowledge of depexts, which means that packages can be
automatically selected based on whether a system package is already installed.
For example, if you have neither MariaDB nor MySQL dev libraries installed,
opam install mysql will offer to install conf-mysql and mysql, but if you
have the MariaDB dev libraries installed, opam will offer to install
conf-mariadb and mysql.
Hint: You can set OPAMCONFIRMLEVEL=unsafe-yes or
--confirm-level=unsafe-yes to launch non interactive system package commands.
opam Lock Files and Reproducibility
When opam was first released, it had the mission of gathering together
scattered OCaml source code to build a community
repository. As time marches on, the
size of the opam repository has grown tremendously to over 3000 unique
packages with over 19500 unique versions. Opam looks at all these packages and
is designed to solve for the best constraints for a given package, so your
project can keep up with releases of your dependencies.
While this works well for libraries, we need a different strategy for projects
that need to test and ship using a fixed set of dependencies. To satisfy this
use case, opam 2.0.0 shipped with support for using project.opam.locked
files. These are normal opam files but with exact versions of dependencies. The
lock file can be used as simply as opam install . --locked to have a
reproducible package installation.
With opam 2.1.0, the creation of lock files is also now integrated into the
client:
opam lock will create a .locked file for your current switch and project,
that you can check into the repository.opam switch create . --locked can be used by users to reproduce your
dependencies in a fresh switch.
This lets a project simultaneously keep up with the latest dependencies
(without lock files) while providing a stricter set for projects that need it
(with lock files).
Hint: You can export the full configuration of a switch with opam switch export new options, --full to have all packages metadata included, and
--freeze to freeze all VCS to their current commit.
Switch Invariants
In opam 2.0, when a switch is created the packages selected are put into the
“base” of the switch. These packages are not normally considered for upgrade,
in order to ease pressure on opam's solver. This was a much bigger concern
early on in opam 2.0's development, but it is less of a problem with the default
mccs solver.
However, it's a problem for system compilers. opam would detect that your
system compiler version had changed, but it's unable to upgrade the ocaml-system
package, unless you went through a slightly convoluted process with
--unlock-base.
In opam 2.1, base packages have been replaced by switch invariants. The switch
invariant is a package formula which must be satisfied on every upgrade and
install. All existing switches' base packages could just be expressed as
package1 & package2 & package3, etc., but opam 2.1 recognises many existing
patterns and simplifies them. Therefore, in most cases, the invariant will be
"ocaml-base-compiler" {= "4.11.1"}, etc. This means that opam switch create my_switch ocaml-system now creates a switch invariant of "ocaml-system"
rather than a specific version of the ocaml-system package. If your system
OCaml package is updated, opam upgrade will seamlessly switch to the new
package.
This also allows you to have switches which automatically install new point
releases of OCaml. For example:
opam switch create ocaml-4.11 --formula='"ocaml-base-compiler" {>= "4.11.0" & < "4.12.0~"}' --repos=old=git+https://github.com/ocaml/opam-repository#a11299d81591
opam install utop
Creates a switch with OCaml 4.11.0 (the --repos= was just to select a version
of opam-repository from before 4.11.1 was released). Now issue:
opam repo set-url old git+https://github.com/ocaml/opam-repository
opam upgrade
Opam 2.1 will automatically offer to upgrade OCaml 4.11.1 along with a
rebuild of the switch. There's not yet a clean CLI for specifying the formula,
but we intend to iterate further on this with future opam releases so that
there is an easier way of saying “install OCaml 4.11.x."
Hint: You can set up a default invariant that will apply for all new switches,
via a specific opamrc. The default one is ocaml >= 4.05.0
Configuring opam From the Command Line
Configuring opam is not a simple task: you need to use an opamrc at init
stage, hack global/switch config file, or use opam config var for
additional variables. To ease that step, and permit a more consistent opam
config tweaking, a new command was added : opam option.
For example:
opam option download-jobs gives the global download-jobs value (as it
exists only in global configuration)opam option jobs=6 --global will set the number of parallel build
jobs opam is allowed to run (along with the associated jobs variable)opam option depext-run-commands=false disables the use of sudo for
handling system dependencies. It will be replaced by a prompt to run the
installation commandsopam option depext-bypass=m4 --global bypasses m4 system package check
globally, while opam option depext-bypass=m4 --switch myswitch will only
bypass it in the selected switch.
The command opam var is extended with the same format, acting on switch and
global variables.
Hint: to revert your changes use opam option <field>=, it will take its
default value.
CLI Versioning
A new --cli switch was added to the first beta release, but it's only now
that it's being widely used. Opam is a complex enough system that sometimes bug
fixes need to change the semantics of some commands. For example:
opam show --file needed to change behaviour- The addition of new controls for setting global variables means that the
opam config was becoming cluttered and some things want to move to opam var
opam switch install 4.11.1 still works in opam 2.0, but it's really an opam
1.2.2 syntax.
Changing the CLI is exceptionally painful since it can break scripts and tools
which themselves need to drive opam. CLI versioning is our attempt to solve
this. The feature is inspired by the (lang dune ...) stanza in dune-project
files, which has allowed the Dune project to rename variables and alter
semantics without requiring every single package using Dune to upgrade their
dune files on each release.
Now you can specify which version of opam you expected the command to be run
against. In day-to-day use of opam at the terminal, you wouldn't specify it,
so you'll get the latest version of the CLI. For example: opam var --global
is the same as opam var --cli=2.1 --global. However, if you issue opam var --cli=2.0 --global, you will have told it that --global was added in 2.1, so it's
not available to you. You can see similar things with the renaming of opam upgrade --unlock-base to opam upgrade --update-invariant.
The intention is that --cli should be used in scripts, user guides (e.g., blog
posts), and in software which calls opam. The only decision you have to take is
the oldest version of opam which you need to support. If your script is using
a new opam 2.1 feature (for example, opam switch create --formula=) then you
simply don't support opam 2.0. If you need to support opam 2.0, then you can't
use --formula and should use --packages instead. Opam 2.0 does not have the
--cli option, so for opam 2.0 instead of --cli=2.0 you should set the
environment variable OPAMCLI to 2.0. As with all opam command line
switches, OPAMCLI is simply the equivalent of --cli, which opam 2.1 will
pick-up but opam 2.0 will quietly ignore (and, as with other options, the
command line takes precedence over the environment).
Note that opam 2.1 sets OPAMCLI=2.0 when building packages, so on the rare
instances where you need to use the opam command in a package build:
command (or in your build system), you must specify --cli=2.1 if you're
using new features.
Since 2.1.0~rc2, CLI versioning applies to opam environment variables. The
previous behaviour was to ignore unknown or wrongly set environment variables.
Now you will have a warning to let you know that the environment variable
won't be handled by this version of opam.
To ensure not breaking compatibility of some widely used deprecated options,
a default CLI is introduced: when no CLI is specified, those deprecated
options are accepted. It concerns opam exec and opam var subcommands.
There's even more detail on this feature in our
wiki. We're
hoping that this feature will make it much easier in future releases for opam
to make required changes and improvements to the CLI without breaking existing
setups and tools.
Note: For opam libraries users, since 2.1 environment variable are no more
loaded by the libraries, only by opam client. You need to load them explicitly.
opam root Portability
opam root format changes during opam's life cycle. New fields are added or
removed, and new files are added. An older opam version sometimes can no longer
read an upgraded or newly created opam root. opam root format has been updated
to allow new versions of opam to indicate that the root may still be read by
older versions of the opam libraries. A plugin compiled against the 2.0.9 opam
libraries will therefore be able to read information about an opam 2.1 root
(plugins and tools compiled against 2.0.8 are unable to load opam 2.1.0 roots).
It is a read-only best effort access, any attempt to modify the opam root
fails.
Hint: for opam libraries users, you can safely load states with
OpamStateConfig
load functions.
Tremendous thanks to all involved people, who've developed, tested & retested,
helped with issue reports, comments, feedback, etc...
Try It!
In case you plan a possible rollback, you may want to first backup your
~/.opam directory.
The upgrade instructions are unchanged:
-
Either from binaries: run
bash -c "sh <(curl -fsSL https://opam.ocaml.org/install.sh) --version 2.1.0"
or download manually from the GitHub "Releases" page to your PATH.
-
Or from source, manually: see the instructions in the README.
You should then run:
opam init --reinit -ni
See full changelog
- Set DEBIAN_FRONTEND=noninteractive for unsafe-yes confirmation level
[#4735 @dra27 - partially fix #4731]
- Fix 2.1~alpha2 to 2.1 format upgrade with reinit [#4750 #4756 @rjbou - fix #4748]
- Fix bypass-check handling on reinit [#4750 @rjbou]
- fish: fix deprecated redirection syntax
^ [#4736 @vzaliva]
- Bump
src_exts and fix build compat with Dune 2.9.0 [#4754 @dra27]
- Fix depext alpine tagged repositories handling [#4758 @rjbou]