package valkey
Install
dune-project
Dependency
Authors
Maintainers
Sources
md5=2946546c5a0e587fbfa4452e6d95a7d6
sha512=e3b213449b3ad70ea541c01a5e77802e07a7222df67ce467c0015f11df9055b00edbe24a431958990e0f5402518a1badd7576c1a9be2bff7951f1d356ef05de5
doc/CHANGELOG.html
Changelog
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
[0.3.1] — 2026-05-01
Fixed
runtestin the opam sandbox failed because the fivemtls config (pure)tests read PEM bytes fromtls/, which is populated at build time byscripts/gen-tls-certs.sh(run by our GHA, not by opam). The tests are pure parse coverage ofTls_config.with_client_cert, so they should not depend on a shell script + openssl at runtest time.Committed self-signed fixtures under
test/fixtures/mtls/(CA + client cert/key, ~5 KB) and declared them as a(source_tree fixtures)dep on the unit_tests stanza so dune copies them next to the test binary. The live mTLS integration test (test_mtls_integration, separate executable) keeps reading fromtls/at runtime — the script remains the source of truth for live-server certs.Reported by @jmid on opam-repository#29825.
Internal
dune-project(depends): no functional change.valkey.opamis now regenerated cleanly fromdune-projectrather than carrying drift from the v0.3.0 cut.
[0.3.0] — 2026-04-30
Added — Phase 10: IAM authentication + mTLS
First-class AWS ElastiCache IAM-token authentication and mutual-TLS client certificates, with the same "configure once, rotations happen in the background" posture as the rest of the library.
Connection.Auth.provider— abstract auth-provider type replacing(string * string) option. Every handshake (initial + reconnect) pulls fresh credentials from the closure, so short-lived secrets flow in without client-code awareness. Constructors:Auth.static ~user ~password(shim for the existing static-password case) andAuth.custom ~name closure(escape hatch for Vault / env-var / user-supplied backends).Connection.refresh_auth t ~user ~password— sendsAUTH user passwordon a live socket. On success the connection stays up; on any wire or server error the connection isinterrupted so the supervisor reconnects via the currently configuredAuth.provider(which returns whatever fresh credentials the caller has produced since). No retry-at-refresh layer — the reconnect path is the single recovery path.Iam_credentials— AWS access-key / secret-key / optional session-token record.Iam_credentials.of_env ()readsAWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY, andAWS_SESSION_TOKEN; explicitmakefor Vault / IRSA / config-file sources.Iam_sigv4— pure-OCaml AWS Signature Version 4 signer for ElastiCacheconnecttokens. Byte-exact against AWS's published signing-key test vector (c4afb1cc5771d871763a393e44b703571b55cc28424d1a5e86da6ed3c154a4b9); usesDigestif.SHA256.hmac_string(already an opam dep) and no AWS SDK. Internals (canonical_query_string,percent_encode,derive_signing_key,hex_sha256) exposed for unit-testing.Iam_provider— stateful IAM provider.createsigns an initial token eagerly and spawns a daemon refresh fiber on the caller's switch. Everyrefresh_interval(default 600s, matching the 10-minute point recommended by the AWS clock-skew window) the fiber re-signs and pushesAUTHto every registeredConnection.tviaConnection.refresh_auth. Dead connections are pruned on each tick, so a forgottenunregister_connectionis bounded-cost.force_refreshexposes manual on-demand rotation.Client.connect_with_iam— convenience wrapper that installs the provider'sauth_provideron the handshake, opens the connection bundle, and registers every live conn with the provider;Eio.Switch.on_releaseunregisters on teardown.Tls_config.with_client_cert ~ca_pem ~client_cert_pem ~client_key_pem— mTLS constructor. Plumbs through toTls.Config.client ?certificatesviawrap_with_tls. PEM inputs are string contents; error messages are redacted (no cert bytes — just the stage name). Empty-cert-list inputs are rejected. The pre-existinginsecure/with_ca_cert/with_system_casconstructors all leave the client-cert slot empty, so server-auth-only TLS works unchanged.- Observability:
Observability.connect_spangains avalkey.auth.modeattribute ("none"/"static"/"iam"/ the user-supplied name fromAuth.custom). NewObservability.record_auth_refresh_failureemits a span event under the active span on refresh-AUTH failure — stateless, no counter added. - Docs:
docs/security.md§IAM walks through the full opt-in with a TLS +connect_with_iamexample;docs/tls.mdmTLS section replaces the "not yet wired" stub with a concretewith_client_certrecipe. - Tests: pure-unit coverage of SigV4 against AWS's signing-key vector + primitive coverage (percent-encode, canonical-query sort, token-shape invariants);
Iam_providerrefresh-fiber rotation (sleeps 1.5s, asserts the cached token changed);Tls_config.with_client_certPEM parse + garbage-input rejection with redacted errors (caught a real bug: the X509 decoder returnsOk []on malformed input, which the first cut would have passed through as valid). Integration tests extendtest/test_connection.mlwithrefresh_authcases: same password is a clean no-op; bad password forces recovery; rotated password installs in place with no socket disturbance. scripts/gen-tls-certs.shnow also mints aclient.crt/client.keypair under the existing test CA for mTLS fixtures.
Dropped / revised from the original scope.
- The original scope promised per-backend first-party providers for env-vars and HashiCorp Vault. Those collapse into
Auth.custom— a three-line closure is all the integration requires; no per-backend shim ships this pass. - The "hardcoded-secret audit" flagged a
"pass"literal inscripts/cluster-hosts-setup.shperdocs/security.mdand ROADMAP §Phase 10. Verified from git history — the script never contained such a literal; the docs claim was stale. A grep sweep ofscripts/,lib/,test/, andbin/(grep -riE '(password|token|secret|api_?key)\s*[:=]\s*"[^"]+"') returns zero hits. Docs rewritten to describe the actual posture. - Live-server mTLS integration test (with a custom
docker-compose.mtls.ymlprofile requiringtls-auth-clients yes) is deferred as follow-up infrastructure — the pure-unit coverage proves the wiring, and setting up the Valkey server fortls-auth-clients yesis a separate deployment concern.
Added — Phase 9: blocking pool
Per-node lease pool that lets one Client.t serve regular multiplexed traffic and BLPOP-class blocking commands from the same process without freezing the multiplex FIFO.
Blocking_poolmodule — narrow lease pool keyed bynode_id. Each blocking call leases one exclusive connection for its duration; clean return re-idles it, exception or cancellation closes it (blocking-command wire state is opaque). Config knobs:max_per_node(default0= feature off),min_idle_per_node,borrow_timeout,on_exhaustion(`Block/`Fail_fast),max_idle_age.- Typed errors:
Pool_not_configured,Pool_exhausted,Borrow_timeout,Node_gone,Connect_failed. Surface viaClient.blocking_erroralongside the existingExec _/No_primary_for_slot _/Cross_slot _/Wait_needs_dedicated_conn _variants. - Client wiring —
Client.blpop/brpop/blmove/blmpop/bzpopmin/bzpopmax/xread_block/xreadgroup_blockroute through the pool when configured; returnError (Pool Pool_not_configured)otherwise. - Dedicated-conn helpers —
Client.with_dedicated_connscopes a one-shot client;Client.wait_replicas_on/Client.wait_aof_onrunWAIT/WAITAOFcorrectly on it.Client.wait/waitaofon the multiplexed client now surfaceError (Wait_needs_dedicated_conn _)by design. - Topology hooks —
Cluster_router.Config.topology_hookswithon_node_removed/on_node_refreshedcallbacks;Client.topology_hooks_for_pool_refbuilds the record the router expects so the refresh fiber drains removed buckets and bumps generation on re-roled nodes. - Observability bridge —
Observability.observe_blocking_pool_metricsmirrors the shippedobserve_cache_metrics. Eight metrics undervalkey.blocking_pool.*:in_use/idle/waitersemitted as non-monotonic sums (UpDownCounter), fivetotal_*counters as cumulative monotonic sums. - Integration tests —
test/test_blocking_pool_integration.mlcovers happy-path, timeout,Pool_not_configured,Pool_exhausted,Borrow_timeout,Node_gonevia synthetic drain,Execerror conn-closure,Wait_needs_dedicated_conn,wait_replicas_onvia dedicated conn,xread_block, cluster cross-slot rejection, same-hashtag allowance, and liveCLUSTER FAILOVER FORCErouting (15 tests total). - Stress test —
test/test_blocking_pool_stress.ml, gated behindSTRESS=1: 1000 concurrentBLPOPcallers withmax_per_node=100, asserts bounded wait, no leaks, all stats settle correctly. Closes the ROADMAP success criterion for Phase 9. - Guide —
docs/blocking-pool.md;docs/performance.mdand the README docs index point at it.
Dropped-scope note. Phase 9's original scope also included a general-purpose Client_pool.t with pick strategies for throughput scaling. An exploratory round with a prototype did not beat single-client multiplex + connections_per_node on the bench rig, so that half has been dropped. The connections_per_node knob on Node_pool remains the supported throughput lever. See ROADMAP.md §Phase 9 for the decision record.
Added — Phase 8: client-side caching (CSC)
Full server-invalidated client-side caching, on standalone and cluster, in all three tracking modes Valkey supports. Configured via the new Client_cache.mode variant — type-level mutual exclusion replaces the old optin : bool + mode = Default | Bcast shape so OPTIN/BCAST mismatches stop compiling.
- Bounded LRU cache primitive (
Valkey.Cache) — byte budget, per-entry overhead accounting, optional TTL safety net, atomic-counter metrics (hits,misses,evicts_budget,evicts_ttl,invalidations,puts). CLIENT TRACKINGhandshake on every (re)connect, fail-closed if the server rejects the mode (no silent drop to unconfigured cache).- Invalidation parser + invalidator fiber — RESP3
>2 ["invalidate", [keys...]]push frames are routed onto a dedicated stream so the invalidator drains them without racing pubsub consumers. - Read-path coverage:
Client.get,Client.mget(scatter- gather over hit/batch/joining groups),Client.hgetall,Client.smembers.Null(missing-key) responses are intentionally not cached. - Single-flight + invalidation-race safety —
Inflighttable dedups concurrent fetches; an invalidation that arrives during the fetch flips a dirty flag so the post-fetch put is skipped. - Cluster integration — one shared
Cache.tacross every shard connection; per-shard tracking happens automatically; flush on every per-connection reconnect AND on topology refresh. mode = Bcast { prefixes }— server-side prefix-broadcast tracking.TRACKINGINFOflag visibility, in-prefix evict, and out-of-prefix isolation all verified end-to-end.mode = Optin— pipelined per-read tracking via the new internalConnection.request_pair(two-frame indivisible submit) andRouter.pairdispatch closure. On cluster, MOVED on the read frame triggers a redirect-aware retry that re- submits the whole pair on the new owner soCACHING YESstays adjacent to the read across the redirect. ASK on the read frame goes throughConnection.request_triple, which sends[CACHING YES; ASKING; read]as one wire-adjacent submit on the importing primary. Frame ordering is load-bearing:ASKINGis consumed by the very next command on the connection regardless of what it is, so it must sit immediately before the slot-keyed read; puttingASKINGfirst would letCACHING YESeat the flag and the read would bounce as MOVED. Verified against a liveCLUSTER SETSLOT MIGRATING/IMPORTINGwindow intest_csc_optin_migration.ml. Empirically validated against Valkey 9.0.3: the OPTIN flag is consumed by exactly the next single command on the wire (so a pipelinedCACHING YES + GET k1 + GET k2tracksk1only); a write before the read consumes the flag too; MULTI/EXEC counts as one logical command for CACHING purposes.
Fixed — cluster routing under topology change
Batch.run_atomic/run_with_guardno longer leak EXECABORT or MOVED on slot-move. A slot ownership change between WATCH/MULTI and EXEC was surfacing to callers asServer_error EXECABORT(orMOVEDdirectly on the EXEC frame). Both paths now map toOk None— the same WATCH-abort outcome the caller's retry loop already handles. Only topology-induced redirects map this way; bad-arity / WRONGTYPE EXECABORTs still flow through the per-command-result array unchanged.Cluster_routerMOVED triggers topology refresh + eager cache clear when the cached slot owner disagrees. Previouslyhandle_retriesonly firedtrigger_refreshon unknown redirect targets; a MOVED to a known node (the common case during failover) silently retried with no refresh, leaving the CSC cache stale until the next periodic refresh (15 s). Now refreshes whenever the cached topology disagrees with the redirect's destination, andtrigger_refreshsynchronously clears the CSC cache (closes the window between MOVED and async refresh-completion).- Pool-race fix on MOVED. When the redirect target is in the topology but not yet in
Node_pool(race between refresh completing and pool-diff applying),handle_retriespreviously short-circuited and returned the original error without triggering refresh. Now triggers refresh so the next attempt finds the connection. - Topology stale-ref bug in retry loops.
topology_refwas only re-synced fromtopology_atomicat the outer call boundary; CLUSTERDOWN/TRYAGAIN backoffs (100–1600 ms) could let the refresh fiber commit a fresh topology while the retry loop kept dispatching against the stale value. Bothhandle_retriesandmake_pairnow accept an optional?sync_refcallback that runs before each (re-)dispatch. handle_retriesASK arm: atomic[ASKING; cmd]submit. Previously sentASKINGand the actual command as two sequentialConnection.requestcalls. The Valkey server consumes the one-shotASKINGflag from the very next command on the connection regardless of which fiber sent it, so under concurrent fibers sharing a node connection, another fiber's frame could interleave betweenASKINGand the retry on the pool's connection — eating the flag and bouncing the actual command as MOVED. Both wires are now pipelined throughConnection.request_pairas one indivisible submit. The matching CSC OPTIN variant (make_pairASK arm) is fixed in the same shape viaConnection.request_triple. Surfaced by a new live-migration concurrent stress test.
Added — OpenTelemetry tracing
- Library emits OpenTelemetry spans for the bounded operations worth knowing latency and failure-mode of:
valkey.connect(TCP + TLS +HELLO/SELECT, one per (re)connect),valkey.cluster.discover,valkey.cluster.refresh. Outcomes (agreed/agreed_fallback/no_agreement) recorded as span attributes. With no exporter configured the cost is near-zero. Per-command spans intentionally not emitted (~150 k req/s would dominate trace volume — apps that need them wrap their own call sites). Seedocs/observability.mdfor setup, attribute schema, and the redaction invariants enforced inlib/observability.ml(no auth credentials, no command keys, no command values, no server error message bodies inside spans). - New dependency:
opentelemetry >= 0.90.
Security audit pass (Phase 2.5)
Reviewed against threat model: trusted Valkey cluster (operators own all members), TLS chain validated when enabled. Three items fixed in this pass; the rest were either out-of-scope under that model or features Valkey itself doesn't support (e.g. per-node SNI). Items addressed:
- Narrowed
Connection.wrap_with_tlscatch. Previously any exception insideTls_eio.client_of_flowwas wrapped asTls_failed, masking internal bugs. Now only the actual handshake-time exceptions map to that variant (Tls_eio.Tls_alert,Tls_eio.Tls_failure,End_of_file,Eio.Io _); anything else propagates. - Non-leaking error payloads.
Tcp_refused/Dns_failed/Tls_failedno longer carryPrintexc.to_stringof the underlying exception (which prints constructor args, internal paths, raw cert bytes). Replaced with a small classifier producing stable short kinds (peer_closed,tls_alert,tls_failure,io_error, errno text viaUnix.error_message). Programmatic handling stays on theError.tvariant. - Observability gap closed (see "Added" above). Auth events, redirect outcomes, and refresh results are now visible in traces.
[0.2.0] — 2026-04-21
Changed — opam packaging
- Split test suite into pure-unit and integration targets. Pure-unit tests (resp3, resp3 round-trip, retry state machine, byte_reader, valkey_error, slot, topology, discovery, redirect, command_spec) run under
dune build @runtestand ship green through opam CI. Integration suites (everything that talks to a live Valkey or cluster) are a plainexecutablestanza now — invoke withdune exec test/run_tests.exelocally. Fixes the opam-repository CI failure on valkey.0.1.0 where @runtest tried to contact a server that does not exist in the opam sandbox.
Added — Batch.pfcount_cluster
Valkey.Batch.pfcount_cluster— HLL union cardinality across keys that may live on different cluster slots. Single-slot inputs go straight to server-side multi-keyPFCOUNT. Cross-slot inputs are materialised under a hashtag-controlled slot viaDUMP+RESTORE,PFMERGEd into a destination HLL,PFCOUNTed, and cleaned up before returning. Missing input keys are treated as empty HLLs (no contribution). Closes the long-standing "pfcount_cluster is intentionally missing" note in the CHANGELOG.
Changed — Transaction folded into Batch
Valkey.Transactionis now a thin wrapper over atomic {!Batch}:begin_with~watch:opens aBatch.guard(so the watched primary's atomic mutex is held across the user's code);queueappends to a bufferedBatch.t ~atomic:true;execruns the whole block viaBatch.run_with_guard(orBatch.runwhen there's no watch). One primitive, one mental model.- Behaviour shift: bad-arity / unknown-command errors surface inside the per-command replies returned by
exec(asResp3.Simple_error _) or via EXECABORT, not atqueuetime. Fan-out commands are still rejected atqueuewith aTerminalerror. - Concurrent transactions on the same primary now serialise at
exectime (via the router's per-primary atomic mutex). Localqueuecalls on different handles never contend. Non-atomic traffic continues to multiplex as before. Batch.validate_same_slotnow compares connections, not slot numbers: standalone (one conn for every slot) and co-located keys within one primary both pass through, only true cross-primary batches fail CROSSSLOT.
Added — Batch.watch (read-modify-write CAS)
Valkey.Batch.with_watch— scoped WATCH guard for the classic optimistic-concurrency pattern. SendsWATCHimmediately, holds the watched primary's atomic mutex across the closure, and guaranteesUNWATCH+ mutex release on any exit (commit, abort, or exception).Valkey.Batch.watch/run_with_guard/release_guard— lower-level pieces for callers that need explicit lifetime control.run_with_guardissuesMULTI/queued/EXECon the guard's connection, returningOk (Some _)on commit,Ok Noneif a watched key was modified,Error _on transport failure. Empty batch under guard sendsUNWATCHand returnsOk (Some [||])— clean abort path for "no write needed" decisions.- Watched keys must hash to one slot (client-side CROSSSLOT validation before any I/O). Pass
~hint_keyto override. - Closes the long-standing limitation noted in the previous entry ("
Batch.create ~watch:is effectively useless"). Legacy~watch:onBatch.createstill exists but is documented as paranoia padding; real CAS goes through the guard API.
Added — Phase 7 (batch + cluster-aware commands)
Valkey.Batch— scatter-gather batch primitive. One module, two modes selected by~atomic:boolatcreatetime:- Non-atomic (default): commands bucketed by slot, each slot's bucket runs as a parallel pipeline on that slot's connection, results merged back into input order. Per-command results; partial success is the norm.
- Atomic (
~atomic:true): all keys must hash to one slot (client-side CROSSSLOT validation). Sends a singleWATCH/MULTI/ commands /EXECburst on the slot's primary. ReturnsOk (Some results)on commit,Ok Noneon WATCH abort.
Per-entry result variant:
One of (Resp3.t, Error.t) result— single-target commands.Many of (node_id * result) list— fan-out commands (SCRIPT LOAD, FLUSHALL, CLUSTER NODES, …), matchingClient.exec_multi.
- Fan-out commands are rejected at
Batch.queuetime in atomic mode (Batch.Fan_out_in_atomic_batchstructured error). They route throughexec_multiin non-atomic mode. - Wall-clock
?timeoutapplies to the whole batch; commands that don't complete in the window come back asOne (Error Timeout); completed ones keep their real reply. Typed helpers collapse any timeout into a whole-call error. Typed cluster helpers under
Valkey.Batch:mget_cluster—(key, value option) listin input order.mset_cluster— per-slot atomic, cross-slot interleaved.del_cluster/unlink_cluster— sum of per-slot removal counts.exists_cluster— sum of existence hits (duplicates count separately, matching server semantics).touch_cluster— sum of keys whose last-access was bumped.
examples/10-batch/— three runnable programs:bulk.ml(1000-key mset/mget/del + perf comparison with per-key loop),scatter.ml(heterogeneous non-atomic batch),atomic_counters.ml(SET NX + INCR + INCR + GET pinned via hashtag in atomic mode).docs/batch.md— concept, atomic vs non-atomic semantics, timeout, ordering, typed helpers, WATCH caveat.
Changed — router
Router.tgainedatomic_lock_for_slot : int -> Eio.Mutex.t. Serialises concurrent atomic operations (bothBatch ~atomic:trueandTransaction) on the same primary connection so their MULTI/EXEC blocks no longer interleave. Non-atomic traffic bypasses the lock and continues to multiplex at full speed. Per-primary (not per-slot): ops on different primaries run in parallel; ops on slots sharing a primary queue behind each other. Standalone uses a single mutex across all slots.Transaction.begin_now acquires the mutex beforeWATCH/MULTI;exec/discardrelease it. Leaking a transaction withoutexec/discardnow also leaks the lock — same failure-mode class as the pre-existing leak of server-side MULTI state, so no user-visible regression for correctly-written callers.
[0.1.0] — 2026-04-20
First public release. The repo's been incubating for a few weeks across Phases 0–5; this tag freezes a working surface and ships it to opam under the 0.x umbrella (API may evolve through 0.x; 1.0 waits for the deeper audit in ROADMAP Phase 12).
Highlights of what's in 0.1.0 (full per-phase detail below):
- Connection layer: auto-reconnect, byte-budget backpressure, circuit breaker, keepalive, TLS, optional cross-domain split.
- RESP3 parser + writer; all 14 wire types.
- ~140 typed commands across strings / counters / TTL / hashes (incl. field TTL on Valkey 9+) / sets / lists / sorted sets / streams (incl. consumer groups + XAUTOCLAIM) / scripting / Functions / pub/sub / blocking / bitmap / HLL / geo / generic keyspace / CLIENT admin / CLUSTER introspection / LATENCY / MEMORY.
- Cluster router: quorum-based topology discovery, periodic refresh, MOVED/ASK/CLUSTERDOWN/TRYAGAIN/Interrupted retry, Read_from with 3-tier AZ-affinity fallback, fan-out helpers.
- Standalone uses the same router behind a synthetic single-shard topology.
- Transaction (MULTI/EXEC/WATCH with hint_key slot pinning).
- Pub/sub: standalone with auto-resubscribe; cluster-aware with failover-watchdog re-pinning.
- 9 runnable examples under
examples/. - 9 hand-written guides under
docs/+ odoc HTML. - 5 GitHub Actions workflows: ci, coverage (60% floor), bench (10% regression gate), nightly fuzz, docs (gh-pages).
- 221 tests; 10M strict parser-fuzz iterations; bin/soak/ for long-running stability with GC + fd slope detection.
Added — typed sorted-set wrappers
The leaderboard example surfaced this gap; the wrappers below replace the Client.custom calls that example was using. docs-first audit done against valkey.io for every command — each .mli quotes the upstream syntax + version + reply shape.
zadd—?mode:zadd_modeis a single sum encoding every server-permitted combination of NX/XX/GT/LT (server rejects NX+GT, NX+LT, NX+XX, GT+LT — those are now type-impossible). Six legal modes:Z_only_add,Z_only_update,Z_only_update_if_greater,Z_only_update_if_less,Z_add_or_update_if_greater,Z_add_or_update_if_less.?chtoggles the CH modifier.zadd_incr— separate function so the return type tracks the reply. Single score-member pair enforced by signature (server rejects multiple under INCR). Returnsfloat option: the new score on success,Nonewhen~modeprevented the write.- Score formatting uses
Printf.sprintf "%.17g"for full IEEE-754 round-trip precision (was%g, which truncated to 6 significant digits and silently lost data on real scores). zincrby— atomic increment, returns the new score asfloat.zrem— returns count of members actually removed.zrank/zrevrank—int option,Nonewhen missing.zrank_with_score/zrevrank_with_score— Valkey 7.2+ WITHSCORE variant, returns(int * float) option.zscore—float option,Nonewhen missing.zmscore— multi-member; returnsfloat option listparallel to the input list.zcount— count members in a score range.zrange_with_scores/zrangebyscore_with_scores— return(string * float) listinstead of just members.zpopmin/zpopmax— atomic pop with optional count; returns(string * float) list.
Score parsing handles both RESP3 Double and Bulk_string shapes (different commands and server versions emit different encodings).
examples/09-leaderboard/ is now typed end-to-end (no Client.custom left).
Added — Phase 5 (examples)
- Initial set of 9 runnable example programs under
examples/: 01-hello (strings/counters/hashes/streams/consumer groups), 02-cluster (Read_from modes + TLS template), 03-pubsub, 04-transaction (WATCH retry), 05-cache-aside (hash field TTL), 06-distributed-lock (SET NX EX + CAD release), 07-task-queue (streams + consumer groups + XAUTOCLAIM), 08-blocking-commands (BRPOP), 09-leaderboard (sorted-set). examples/README.mddocuments the set and the convention.CONTRIBUTING.mdupdated: new significant features land with the example that exercises them.
Changed
Cluster_router.pick_node_by_read_fromnow picks a random replica forPrefer_replicaandAz_affinitymodes (was always the first replica, which pinned all reads from one client to a single replica). Spreads readonly traffic across the replica set.
Known gaps surfaced by examples
- Sorted-set commands
ZADD,ZINCRBY,ZRANK,ZSCORE,ZREVRANGEare not yet typed wrappers — the leaderboard example usesClient.customfor them. Wrapping them is on the pre-1.0 list.
Added — Phase 4 (documentation)
docs/— 9 hand-written guides covering getting-started, cluster, transactions, pub/sub, TLS, performance, troubleshooting, security, and migration fromocaml-redis.CONTRIBUTING.mdat repo root — build/test/fuzz/bench/coverage workflow, style rules, PR checklist.CHANGELOG.mdrewritten to explicitly cover Phases 0 → 3 (was stale, reported 82 tests)..github/workflows/docs.yml— builds odoc HTML on every push/PR, stages guides under/guides/, deploys togh-pageson main.dune build @docis warning-clean on all 20 modules; the previously-hidden constructor warning onConnection.Error.twas resolved during this pass.
Added — Phase 3 (CI/CD + coverage)
GitHub Actions workflows:
ci.yml— Ubuntu × OCaml {5.3, 5.4} integration (docker standalone + cluster, full tests, 100k parser fuzz strict, 30s standalone + 30s cluster stability fuzz). macOS × OCaml {5.3, 5.4} portability subset (docker-free tests + 50k parser fuzz).coverage.yml— bisect_ppx instrumentation, HTML artifact, 60 % floor (baseline 63 %), gh-pages deploy of the report on main.fuzz-nightly.yml— scheduled 02:00 UTC. 200M parser fuzz strict + 15 min cluster stability with docker-restart chaos. Auto-opens an issue on non-zero exit.bench.yml— per-PR delta table vsmainwith a 10% regression gate; pushes tomainstash the baseline on thebench-historybranch.
bin/bench_compare/— zero-dep bench-JSON diff tool producing GitHub-flavoured markdown tables.bin/bench/gained--json PATHoutput.bisect_ppxis a newwith-dev-setupdependency;lib/dunedeclares it as the instrumentation backend.
Added — Phase 2 (testing rigour + audit)
test/test_resp3_roundtrip.ml— randomised round-trip proptest, 10k random leaves + 10k nested trees + targeted edge cases (empty aggregates, bulk-with-CRLF, int64 extremes, inf/NaN, exotic map keys). Hand-rolled generator, no new dep.test/test_command_spec_property.ml— three cluster-level properties: 500 random-keyed round-trips, every-slot endpoint coverage, Read_from.Prefer_replica actually reaches replicas.test/test_retry_state.ml— nine focused tests over the retry state machine viaCluster_router.For_testing: ok / non-retryable / TRYAGAIN / CLUSTERDOWN (×2, incl. exponential schedule + budget exhaustion) / Interrupted / Closed / mixed / backoff schedule.- Parser fuzzer upgrades: tree-level structural mutation (swap / duplicate / reverse sublist / recursively mutate), length-field poisoning on every declared aggregate/bulk header, and a delta-debug shrinker that prints the minimal reproducer on failure. 10 M strict clean at ~145k inputs/s on the new six-strategy mix.
bin/soak/— long-running stability soak. Steady SET/GET/DEL workload with a sampler recordingGc.quick_statheap + top + live,/proc/self/fdcount, total ops every N seconds. OLS slope detection flags heap or fd leaks;--strictexits 1 on threshold breach.docker-compose.toxiproxy.yml+scripts/chaos/chaos.sh— TCP chaos via toxiproxy. Subcommands: setup / latency / loss / bandwidth / reset / close / clear / teardown. Point bin/fuzz, bin/soak, or bin/bench at the proxy ports for chaos runs.AUDIT.md— inventory of everyObj.magic,try _ with _ -> (),ignore (_ : _ result),mutablefield (+ lock discipline), andAtomic.*site, each with a disposition.Cluster_router.For_testing— exposeshandle_retriesand the backoff constants so the retry loop can be driven from unit tests without a real pool.
Changed — Phase 2
- Deleted the dead
watchdogfunction incluster_pubsub.ml— it contained anObj.magic ()landmine (never executed, but dangerous to leave around); the correct implementation was already inlined increate. Connection.closeandCluster_router.closenow guard withAtomic.exchange closing true, making repeat calls a true no-op instead of relying on each inner step being re-invokable.- Added a locking-discipline comment at the top of
cluster_pubsub.mldocumenting theshards_mutex/subs_mutexcontract. - Narrowed all 14 drain-path
try ... with _ -> ()sites to the specific exceptions that legitimately arise on teardown (Eio.Io _ | End_of_file | Invalid_argument _ | Unix.Unix_error _for close paths;Invalid_argument _forEio.Promise.resolve). Anything else now surfaces instead of being silently swallowed.
Added — Phase 1 (command surface completion)
- Bitmap:
BITCOUNT,BITPOS,BITOP,SETBIT,GETBIT,BITFIELD,BITFIELD_RO.bit_rangeis a sum type (From,From_to,From_to_unit) matching Valkey 8.0+ semantics, andBITOP NOTis encoded with arity-at-type-level (Bitop_not of stringtakes exactly one source). - HyperLogLog:
PFADD,PFCOUNT,PFMERGE. - Generic keyspace:
COPY,DUMP,RESTORE,TOUCH,RANDOMKEY,OBJECT ENCODING|REFCOUNT|IDLETIME|FREQ. - Geo:
GEOADD,GEODIST,GEOPOS,GEOHASH,GEOSEARCH,GEOSEARCHSTOREwith typedgeo_from,geo_shape,geo_search_result. - CLIENT admin:
CLIENT ID|GETNAME|SETNAME|INFO|LIST|PAUSE| UNPAUSE|NO-EVICT|NO-TOUCH|KILL|TRACKING, with closed-sum filters (client_kill_filter) and typed tracking options (client_tracking_on+ REDIRECT / PREFIX / BCAST / OPTIN / OPTOUT / NOLOOP). - Functions + FCALL:
FUNCTION LOAD|DELETE|FLUSH|LIST,FCALL,FCALL_RO(fan-out to every primary for LOAD-class commands viafan_primaries_unanimous). - Cluster introspection:
CLUSTER KEYSLOT,CLUSTER INFO. - Observability:
LATENCY DOCTOR|RESET,MEMORY USAGE,MEMORY PURGE. Named_commands— register command and transaction templates with$Nplaceholders, run by name later. Thread-safe; shares the same routing asClient.custom.Pubsub— standalone client-level pub/sub with auto-resubscribe after reconnect viaConnection.on_connected.Cluster_pubsub— regular + sharded pub/sub on one handle, with a watchdog fiber re-pinning sharded connections on failover (integration test forces all 3 primaries to restart and asserts delivery resumes).Transaction—MULTI/EXEC/WATCH/DISCARDwith ahint_keyto pin to a slot, and awith_transactionscope helper.Command_spec— ~230 entries covering all typed wrappers, plus a test that cross-checks every entry against liveCOMMAND INFOmetadata.- Send-path optimisation — new
Resp3_writer.command_to_cstructproduces a single allocation of the exact wire size with one blit per argument. SET 16 KiB went from 47 % to 91 % of the C reference. - Parser hardening —
Resp3_parsernow rejects negative bulk and aggregate lengths (regression found by the parser fuzzer on its first run).
Added — Phase 0 (core)
- Connection layer: auto-reconnect with jittered backoff, byte-budget backpressure, circuit breaker (always-on generous default), app-level keepalive fiber, TLS (self-signed + system CAs), optional cross-domain split (
?domain_mgr) moving socket I/O to a dedicated OS thread. - RESP3 parser + writer covering all 14 wire types. Streamed aggregates raise explicitly (not silently mis-decoded).
- Client layer: abstract
Client.twith typed commands covering strings, counters, TTL, hashes, hash field TTL (Valkey 9+), sets, lists, sorted sets, scripting withScript.tand transparentNOSCRIPTfallback, iteration, streams (non-blocking + consumer groups), blocking commands. - Typed variants for every wire-level keyword set (
set_cond,set_ttl,hexpire_cond,hgetex_ttl,hsetex_ttl,score_bound,value_type, …) and every per-field status code (field_ttl_set,field_persist,expiry_state). - Routing interface (
Read_from,Target) — surfaces the API shape the cluster router plugs into without changing callers.