package owl-rllib
Build not yet checkedInstall
dune-project
Dependency
Authors
Maintainers
Sources
sha256=7f21880386e8dfad71fa7be32819e437b0e555db6ef4fb457f6824ec516422d8
sha512=f49617d1f6faee0b3e32cef9565879503836b46a6eb130513f63cc598d0669fbaa8c535b43070be3cc281abbe19047e6a547fec713058f02194a519818623347
doc/README.html
owl-rllib
Reinforcement learning in OCaml, built on Owl.
owl-rllib aims to be small, readable and reproducible rather than large: one file per algorithm, explicit random number generators, and every environment and loss checked against a reference implementation (Gymnasium, PyTorch). DQN and PPO match CleanRL's published results on its benchmark environments (see bench/).
Contents
- MDPs and tabular methods: finite MDPs with explicit transition tables; policy evaluation, policy iteration and value iteration; Monte Carlo, TD(0), SARSA and Q-learning.
- Environments: GridWorld, FrozenLake, CliffWalking, Taxi, CartPole, MountainCar, Acrobot and Pendulum, written in OCaml and checked step by step against Gymnasium 0.29.1. Synchronous vector environments, frame stacking, observation and reward normalisation.
- Deep RL: linear function approximation, REINFORCE (with an optional baseline), DQN, A2C and PPO, with categorical and Gaussian policies, on Owl's neural networks (MLPs and small CNNs).
- RL for language models (
owl-rllib.llm): a toy autoregressive model on a verifiable task (two-digit addition), with REINFORCE, RLOO, token-level PPO, GRPO, and the GRPO variants Dr. GRPO, DAPO, GSPO and CISPO as configuration switches. - Python bridge (
owl-rllib.bridges, optional): any Gymnasium environment (e.g. LunarLander, MuJoCo) and VizDoom scenarios, run in a Python child process and exposed as ordinary environments.
Installation
owl-rllib needs Owl 1.3 or later, which needs OpenBLAS and LAPACKE (e.g. libopenblas-dev liblapacke-dev on Debian and Ubuntu, openblas on Homebrew).
opam
opam install owl-rllibDocker
The image matrixanger/owl-rllib (tag latest) has Owl's main branch and owl-rllib built, tested and installed, together with the Python packages for the Gymnasium / MuJoCo / VizDoom bridge. It is rebuilt whenever this repository or the Owl image (matrixanger/owl:main) changes.
docker run -it matrixanger/owl-rllib
cd /root/owl-rllib
dune exec examples/classic/a7_actor_critic.exeOther projects in the container can use (libraries owl-rllib) directly.
From source
git clone https://github.com/owlbarn/owl-rllib.git
cd owl-rllib
opam install . --deps-only --with-test # Owl and, for the tests, alcotest
dune build
dune testThe API documentation is generated with dune build @doc.
Python, for the bridge
owl-rllib.bridges runs Gymnasium or VizDoom in a Python child process, so it needs Python with those packages; nothing else does. The interpreter is $OWL_RLLIB_PYTHON, by default python3, and tests that need the packages are skipped when they are missing.
The Docker image already has them: Python 3.11 in /opt/venv, with OWL_RLLIB_PYTHON set, so the bridge examples and the MuJoCo benchmarks run as they are.
Elsewhere, install the versions in python-requirements.txt, the ones the tests' reference data was recorded with. VizDoom 1.2.3 needs Python 3.12 or older; with uv:
uv venv --python 3.11 ~/.venvs/owl-rllib
uv pip install --python ~/.venvs/owl-rllib/bin/python -r python-requirements.txt
export OWL_RLLIB_PYTHON=~/.venvs/owl-rllib/bin/python
dune test # now also runs the bridge testsBox2D is needed for LunarLander, MuJoCo and imageio for Hopper and the other MuJoCo tasks, and VizDoom for examples/classic/a8_vizdoom.ml; install only what you use.
Quick start
The package has three libraries: owl-rllib (module Owl_rllib), owl-rllib.llm (Owl_rllib_llm) and owl-rllib.bridges (Owl_rllib_bridges). With dune:
(executable (name main) (libraries owl-rllib))open Owl_rllib
module N = Neural.Make (Owl.Neural.S.Graph)
let () =
let rngs = Rng.create 0 in
let policy = N.mlp ~seed:rngs.network_seed ~shape:[| 4 |] ~hidden:[ 64; 64 ] ~outputs:2 in
let critic = N.mlp ~seed:(rngs.network_seed + 1) ~shape:[| 4 |] ~hidden:[ 64; 64 ] ~outputs:1 in
let env = Vec_env.create (Array.init 8 (fun _ -> Cartpole.make ~rng:rngs.env ())) in
let returns = Ppo.S.train ~updates:100 ~rngs ~policy ~critic ~distribution:Policy.S.categorical env in
Printf.printf "last episode: %.0f\n" returns.(Array.length returns - 1)OPENBLAS_NUM_THREADS=1 dune exec examples/classic/a7_actor_critic.exeSet OPENBLAS_NUM_THREADS=1: the networks here are small, and OpenBLAS's default thread count slows them down.
Examples
examples/ holds runnable programs in two tracks, each building on the last:
examples/classic: from MDPs to pixels. a1 MDPs, a2 dynamic programming, a3 tabular methods, a4 function approximation, a5 DQN, a6 REINFORCE, a7 actor-critic and PPO (CartPole, Pendulum, LunarLander), a8 PPO from pixels on VizDoom.examples/llm: RL for language models, following Cameron Wolfe's Reinforcement Learning for LLMs: b0 supervised fine-tuning, b1 generation as an MDP, b2 REINFORCE, b3 RLOO, b4 KL penalties, b5 PPO, b6 GRPO, b7a–b7d Dr. GRPO, DAPO, GSPO and CISPO.
Each file's header gives the command to run it and what it shows.
Reproducibility
Every random choice goes through an explicit Random.State.t (see Rng): the same seed on the same machine gives bit-identical results (another machine or BLAS build, including the Docker image, gives different but equally valid ones). One exception: training a convolutional network in Owl occasionally differs by one unit in the last place between runs, so the VizDoom example (a8) is not bit-for-bit reproducible.
License
MIT, see LICENSE.