Platform Newsletter: September 2023
Welcome to the sixth edition of the OCaml Platform newsletter!
Dive into the latest updates from September and discover how the OCaml Platform is evolving. Just like in previous newsletters, it spotlights the recent developments and enhancements to the OCaml development workflows.
In addition to the updates on the Platform team's progress highlighted below, don't hesitate to share your feedback on the upcoming Platform roadmap. We've just updated it based on the most recent feedback and are aiming to adopt it in the coming weeks, barring new concerns from the community.
- Building Packages
- Generating Documentation
- Editing and Refactoring Code
Here are all the new versions of Platform tools that were released this month:
For detailed release notes and announcements, explore the OCaml Changelog.
[Dune] Exploring Package Management in Dune
Contributors: @rgrinberg (Tarides), @Leonidas-from-XIV (Tarides), @gridbugs (Tarides), @kit-ty-kate (Tarides), @Alizter
The biggest highlight from September is that the work to expose the compiler and libraries from packages installed by Dune to the rest of the project is now complete! This means that there is now a prototype of Dune package management that can be used to build projects that depend on (simple) opam packages! This is still an early prototype that's not ready to be tested outside of the core team, but still a significant milestone: :tada:!
In addition to this, work in September focussed on three areas:
- Increasing coverage of opam features to support more opam packages from the
opam-repository. This month, the Dune team added support for new fields, including
subst, and they also added support for patching.
- Designing and implementing a string manipulation DSL for Dune configurations. This will allow users to express the same amount of dynamism found in opam filters in Dune package lockfiles, which is necessary for converting opam
installcommands into Dune expressions.
- Started working on support for custom opam repositories by making the
dune-workspace. The next step is to experiment on how opam repositories are stored and accessed. One idea is that all opam repositories would be stored in one revision storage that would supply all the data. This has the advantage that incremental updates are small, which work like pulling via Git. The repo doesn't need to be uncompressed, thus less storage and inodes used.
- Move packages to private context -- ocaml/dune#8467
build-envfrom opam file into lock dir -- ocaml/dune#8701
setenvfrom opam file into Dune lock dir -- ocaml/dune#8708
substsfield of opam file into build action -- ocaml/dune#8669
- Add patching support to Dune pkg -- ocaml/dune#8654
- Copy files from opam repository to lock dir -- ocaml/dune#8648
dune.lockis ignored in
- Opam repositories from
dune pkg outdatedcommand for showing outdated packages -- ocaml/dune#8773
- Experimental string list language -- ocaml/dune#8596
[opam] Native Support for Windows in opam 2.2
Contributors: @rjbou (OCamlPro), @kit-ty-kate (Tarides), @dra27 (Tarides), @AltGr (OCamlPro)
In preparation for the upcoming release of opam 2.2~alpha3, the work has focussed on better handling of path rewriting for the
build-env opam fields.
The proposed change will allow users to specify, in the opam file, the path separator and format they want for each environment variable in
build-env. This ensures the environment variables are correctly set and usable on Windows.
The PR is in review and not quite ready to be merged, but this is the last issue scoped for opam 2.2~alpha3.
- Path rewriting for
[Dune] Dune Terminal User Interface
Contributors: @Alizter, @rgrinberg (Tarides)
Following the merge of the PR to port Dune TUI to Nottui in August, and the addition of a few features, @Alizter continued the work on building a full-on Terminal User Interface for Dune with two pull requests, namely the addition of a Jobs tab in
tui mode, and support for multiline status lines.
[Dune] Support on Niche Platforms
Dune now builds on both Haiku and Android (using Termux)! This means it is now possible to build and install both OCaml and Dune on these platforms, which should pave the way for more native OCaml development.
For reference, here is a table of Dune's platform support (with
? indicating that further testing is needed):
| Platform | Support | Watch | TUI | Cache | Sandboxing | |------------------|---------|-------|-----|-------|------------| | Linux | Full | Yes | Yes | Yes | Yes | | MacOS | Full | Yes | Yes | Yes | Yes | | Windows (DKML) | Full | Yes | No* | Yes | Copy only | | Windows (MinGW) | Limited | Yes | Yes | Yes | Yes | | Windows (Cygwin) | Limited | Yes | Yes | Yes | Yes | | Linux (Android) | Limited | Yes | Yes | ? | ? | | FreeBSD | Limited | Yes | Yes | ? | ? | | NetBSD | Limited | Yes | Yes | ? | ? | | OpenBSD | Limited | ? | ? | ? | ? | | Haiku | Limited | Yes | Yes | ? | ? |
If you're working on one of these platforms, don't hesitate to open issues on Dune's bug tracker if you encounter any problem!
- Add Haiku support -- ocaml/dune#8795
odoc] Add Search Capabilities to
Contributors: @panglesd (Tarides), @EmileTrotignon (Tarides), @julow (Tarides), @jonludlam (Tarides)
Work continues on adding search capabilities to
odoc in order to improve the documentation browsing experience.
In September, the
odoc team continued reviewing the different pull requests started in August. Peer-reviews suggested several improvements to to the CLI and the library API.
They also worked on client-side performance improvements by loading the search script only when the user clicks on the search bar, and they made quite a lot of progress on the UI overall.
odoc] Syntax for Images and Assets in
Contributors: @panglesd (Tarides)
The effort to add support for images and assets to
odoc and bring images to the OCaml.org package documentation continues!
This month, @panglesd opened a PR with an implementation for asset references.
The exact syntax for medias went through several designs, in particular whether a media is a block, a nestable block, or an inline element. At the end of the month, @panglesd created a PR that builds on the asset references PR in order to add support for medias.
For some time, there has been no official convention on how documentation for opam-installed packages should be built. With the added complexity of having assets, it was a good time to solve this. A documentation PR was opened for this. Warm thank you to @dbuenzli for the thourough review and participating in establishing these conventions!
- Asset References -- ocaml/odoc#1002
- Medias in
- Document parent-child convention for installed packages -- ocaml/odoc#1011
[Dune] Generate Dependencies Documentation with Dune
Contributors: @jonludlam (Tarides)
Currently, Dune only knows how to build the documentation for the packages in your Dune workspace, meaning that you can only read the documentation of your dependencies from the OCaml.org package site. Alternative
odoc drivers, like
odig, build documentation for all the packages in your switch and have been the recommended solution for users who prefer to read the dependencies' documentation locally.
In an effort to improve the documentation generation experience with Dune, @jonludlam worked on a new version of Dune rules to generate the documentation. With these rules, Dune will gain the additional ability to build the combination of the two: a coherent set of docs that cover both switch-installed libraries and local libraries.
The PR is in review and is set to be merged in the coming weeks.
Future plans for the new rules include better integration with the rest of the platform, improvements in capabilities to cover the use cases that
dune build @doc covers, integration of source rendering, and integration of search (once it lands in
odocrule -- ocaml/dune#8803
Editing and Refactoring Code
[Merlin] Support for Project-Wide References in Merlin
Contributors: @voodoos (Tarides), @trefis (Tarides), @Ekdohibs (OCamlPro), @gasche (INRIA)
In August, the Merlin team opened the PR on the compiler that adds the necessary information in the Shapes to implement project-wide references.
The PR received reviews, so the team worked on taking the feedback into account while also continuing work on the rest of the stack (build system rules, the indexer and new locate, and occurrences backends for Merlin).
They also consolidated a release plan and timeline. The plan is to first release an experimental 4.14-based variant of the compiler in order to gather feedback on this eagerly awaited feature before the end of the year. The current aim is to provide official project-wide occurrences support in OCaml 5.2.
- Add support for project-wide occurrences to the compiler -- ocaml/ocaml#12508
- Use new compile information in CMT files to build and aggregate indexes -- voodoos/ocaml-uideps#5
- Dune orchestrates index generation -- voodoos/dune#1
- Use new CMT info to provide buffer occurrences and indexes for project-wide occurrences -- voodoos/merlin#7
- Support project-wide occurrences in
[Merlin] Improving Merlin's Performance
Contributed by: @pitag-ha (Tarides), @3Rafal (Tarides), @voodoos (Tarides), @let-def (Tarides)
The Merlin team continues work on improving Merlin performance.
Before diving into specific performance optimisation, they worked on a benchmarking CI to catch performance regressions and measure improvements. While at it, they've also worked on a fuzzy testing CI to catch behaviour regressions.
In September, following the Proof of Concept (PoC) of the fuzzy-testing CI (from the work in July), the team continued their work on addressing the limitations of the current CI implementation that would prevent it from being merged in Merlin. Specifically, they focussed on finding a solution to store the fuzzy testing results in a way that wouldn't bloat the Merlin repository. The current approach is to store the data in a separate Git repository and pull it when running the fuzzy-testing CI. They've created a GitHub action workflow that implements this behaviour.
Next, the plan is to complete the work on the Merlin CI before gradually shifting focus to performance optimisations.
- PoC for the Behavior CI – ocaml/merlin#1657