irmin-unix

Unix backends for Irmin
Legend:
Library
Module
Module type
Parameter
Class
Class type
Manual

Fixed

  • Add a consistency check for the files on disk when opening store
    (#2004, #2007, #@icristescu)

  • Fix irmin-tezos CLI and ./tezos-node storage to work with
    v2 inodes (#1903, @icristescu, @Ngoguey42)

3.3.1 (2022-06-22)

Fixed

  • irmin-pack

    • Fix topology irregularities on disk which may lead to post-gc crashes.
      (#1925, @Ngoguey42, @icristescu)

3.3.0 (2022-06-20)

Added

  • irmin

    • Add Metrics module to describe metric gathering in irmin.
      (#1817, @maiste)

    • Add Repo.config to access config used to create repo
      (#1886, @zshipko)

  • irmin-unix

    • Add --plugin flag to load Dynlink plugins that can register new
      contents, hash and store types (#1808, @zshipko)

  • irmin-pack

    • Add use_fsync, dict_auto_flush_threshold and

    • suffix_auto_flush_threshold in store configuration. (#1865, @Ngoguey42)

    • Add no_migrate in store configuration. (#1893, @zshipko)

Changed

  • irmin-pack

    • Move Irmin_pack.Pack_store.Indexing_strategy to
      Irmin_pack.Indexing_strategy and the rest of Pack_store
      to Irmin_pack_unix (#1833, @Ngoguey42)

    • Different repos opened using the same store module no longer share caches
      and file descriptors (#1852, @Ngoguey42)

    • Snapshot.Import.close requires a repo as additional argument (#1872,
      @icristescu)

    • Upgraded on-disk format to version 3 to support better synchronisation
      mechanism between readwrite and readonly instances. This change is not
      backwards-compatible with existing stores using irmin-pack.x.x < 3.3.0
      versions. A migration done by the readwrite instance is necessary to open
      older stores with irmin-pack.3.3.0. It is not forwards compatible. (#1865)

    • Rename Store.sync to Store.reload (#1900, @Ngoguey42).

    • Add Pack_error exception that centralizes many error cases alongside
      RO_not_allowed and Unix.Unix_error (#1899, @Ngoguey42)

Fixed

  • irmin-pack

    • Allow snapshot export to work on indexed root nodes (#1845, @icristescu)

  • irmin

    • Fix Tree.export for nodes exported twice using different repos. (#1795,
      @Ngoguey42)

3.2.1 (2022-04-07)

  • Support all version of cmdliner (#1803, @samoht)

3.2.0 (2022-03-28)

Added

  • irmin-pack

    • Add forbid_empty_dir_persistence in store configuration. (#1789,
      @ngoguey42)

    • Add Store.Snapshot to expose the inodes for tezos snapshots (#1757,
      @icristescu).

Changed

  • irmin

    • Add error types in the API or proof verifiers. (#1791, @icristescu)

    • Reduced the memory footprint of Tree.fold ~uniq:`True by a factor of 2.
      (#1770, @CraigFe)

    • Remove clear from all content addresssable stores. (#1794, @icristescu)

3.1.0 (2022-02-25)

Fixed

  • irmin-pack

    • Drop unnecessary runtime dependency on ppx_irmin. (#1782, @hhugo)

    • Split the unix part of irmin-pack into irmin-pack.unix (#1783, @hhugo)

  • irmin-unix

    • Fix conflicting command line arguments for push, pull, fetch and
      clone (#1776, @zshipko)

    • Fix issues with Sync functions by provided a better default Mimic.ctx. A
      side-effect of this update is that the remote function now returns an Lwt
      promise. (#1778, @zshipko)

Added

  • libirmin

    • Create libirmin package providing a C interface to the irmin API
      (#1713, @zshipko)

Changed

  • irmin-bench

    • Make trace replay API public and simpler (#1781, @Ngoguey42)

3.0.0 (2022-02-11)

Fixed

  • irmin

    • Fix the implementation of comparison on Irmin.Tree objects to use the
      comparison defined on hashes. The previous implementation was unstable.
      (#1519, @CraigFe)

    • Default implementation for contents, nodes and commits can no longer trigger
      pre_hash collisions. This changes their hash. (#1715, @Ngoguey42,
      @icristescu)

  • irmin-pack

    • Improve the performance of Index encode and decode operations by
      eliminating intermediate allocations. (#1577, @CraigFe)

  • irmin-unix

    • Fix terms that can be manipulated at runtime by delaying computation
      (#1645, @zshipko)

Added

  • irmin

    • Add Read_only.S and Read_only.Maker module types (#1343, @samoht)

    • Append-only and content-addressable backend implementations have to
      provide close and batch functions (#1345, @samoht)

    • Atomic-write backend implementations have to provide a close function
      (#1345, @samoht)

    • Add a function Store.Tree.singleton for building trees with a single
      contents binding. (#1567, @CraigFe)

    • Add with_handler and head to Store.Backend.Node and
      Store.Backend.Node_portableto work with recursive node structures from
      irmin core. (#1712, #1746 @Ngoguey42). Forward
      port of #1692 and #1670.

    • Add proof, to_proof and of_proof to Store.Backend.Node_portable
      (#1716, @Ngoguey42). Forward port from #1583.

    • Add hash_exn to Store.Backend.Node.Val and Store.Backend.Node_portable
      (#TODO, @Ngoguey42) Forward ported from #1741.

    • Add a Store.Tree.kinded_hash function. (#1767, @Ngoguey) Forward ported
      from #1625.

    • Add Contents.String_v2, Node.Generic_key.Make_v2 and
      Commit.Generic_key.Make_v2 for backward compatibility with older stores.
      (#1715, @icristescu)

  • irmin-bench

    • Many improvements to the actions trace replay:

      • Support for the layered store (#1293, @Ngoguey42)

      • Fix replay for the first ~650k commits (was ~13k) (#1314, @Ngoguey42)

      • Can change inode configuration prior to replay (#1326, @Ngoguey42)

      • Check hash of commits (#1328, @icristescu)

      • Fix the path flattening technique (#1357, @Ngoguey42)

      • Introduce a new actions trace that can support replaying up to ~1300k
        commits. (#1358, @Ngoguey42)

      • Improve the stats collection and stats report (#1367, #1384, #1403,
        #1404, #1416, #1429, #1438, #1501, #1616, @Ngoguey42, @maiste)

      • Enable replay in CI (#1430, @Ngoguey42)

      • Enable replay in CB (#1441, @Ngoguey42)

  • irmin-mem

    • Add Irmin_mem.Content_addressable (#1369, @samoht)

  • irmin-pack

    • Add a stat-store command to irmin-fsck to output stats on the tree
      under a specified commit (#1391, @icristescu, @Ngoguey42, @CraigFe).

    • Add new counters in Stats (#1570, @Ngoguey42).

    • Add an option to configure the index function and pick the relevant bits
      in a cryptographic hash by default (#1677 #1699, @samoht)

    • Verify inode depth invariants (#1711, @Ngoguey42). Forward port of #1665.

  • irmin-unix

    • Update irmin CLI to raise an exception when an invalid/non-existent
      config file is specified (#1413, @zshipko)

    • Add --commit flag to CLI to load a store from a specific commit hash
      (#1721, @zshipko)

  • irmin-tezos

    • Add a new package to mirror Tezos tezos-context.encoding library.
      That'll simplify building benchmarks and custom tools (#1579, @samoht)

Changed

  • irmin

    • Irmin.Sync is now a namespace: use Irmin.Sync.Make(S) instead of
      Irmin.Sync(S) (#1338, @samoht)

    • Irmin.Private is now Irmin.Backend (#1530, @CraigFe)

    • Store.master is now Store.main. The existing Store.master function is
      deprecated and will be removed in a future release. (#1564, @CraigFe)

    • Store.Private is now Store.Backend (#1530, @CraigFe)

    • Store.Private.Sync is now Store.Backend.Remote (#1338, @samoht)

    • Irmin.Branch.S.master is now Irmin.Branch.S.main (#1564, @CraigFe)

    • Irmin.Private.{Commit,Node} are now Irmin.{Node,Commit}. (#1471,
      @CraigFe)

    • All module types are now using snake-case and are not capitalized anymore.
      (#1341, @samoht)

    • Move signatures for backend stores into their own modules. All the
      X_STORE sigs have moved to X.S:

      • APPEND_ONLY_STORE is now Append_only.S

      • CONTENT_ADDRESSABLE_STORE is now Content_addressable.S

      • ATOMIC_WRITE_STORE is now Irmin.Atomic_write.S
        And all the X_STORE_MAKER have moved to X.Maker:

      • APPEND_ONLY_STORE_MAKER is now Append_only.Maker

      • CONTENT_ADDRESSABLE_STORE_MAKER is now Content_addressable.Maker

      • ATOMIC_WRITE_STORE_MAKER is now Atomic_write.Maker
        This gives some space to move convenient functors closer to where they
        belong:

      • Content_addressable is now Content_addressable.Make

      • New Content_adddressable.Check_closed and Atomic_write.Check_closed
        (#1342, @samoht)

    • Rename Irmin.Make into Irmin.Maker ; stage its result to return
      Make functor once provided with a content-addressable and an
      atomic-writes stores (#1369, @samoht)

    • Rename Irmin.Make_ext into Irmin.Maker_ext ; stage its result to
      return Make functor once provided with a content-addressable and an
      atomic-writes stores, as well as node and commit makers (#1369, @samoht)

    • Require at least lwt.5.3.0 to use Lwt.Syntax in the codebase
      (#1401, @samoht)

    • Info implementations are not part of store: use S.Info.v
      instead of Irmin.Info.v (#1400, @samoht)

    • Rename Commit.V1 to Commit.V1.Make. This functor now takes separate
      hash and key implementations as arguments. (#1431 #1634, @CraigFe
      @icristescu)

    • Introduce a Schema module to hold all the types that users can
      define in an Irmin store. Use this as a parameter to every Maker
      functor. This is a large change which touches all the backends.
      (#1470, @samoht, @CraigFe)

    • Add Irmin.Backend.Conf.Schema for grouping configuration keys. Now
      Irmin.Backend.Conf.key takes an additional ~spec parameter.
      (#1492, @zshipko)

    • Tree.empty and Node.empty now both take a unit argument. (#1566 #1629,
      @CraigFe)

    • Rename key type to path and Key module to Path when it is in a path
      context in Tree and Store. (#1569, @maiste)

    • Move Node.default metadata default values into a Node.Metadata.default
      to give room for other metadata values (#1611, @samoht)

    • Add support for non-content-addressed ("generic key") backend stores. This
      allows Irmin to work with backends in which not all values are addressed by
      their hash. In particular, this includes:

      • New functions: Store.{Commit,Contents,Tree}.of_key.

      • Adds Irmin.{Node,Commit}.Generic_key modules.

      • Adds new types that must be provided by backends: Node.Portable and
        Commit.Portable.

      • Adds a new type of backend store: Irmin.Indexable.S.
        (#1510 #1647, @CraigFe)

    • Cache hits in several Tree functions are more frequent than before.
      (#1724, @Ngoguey42, @CraigFe)

    • Add a new Pruned_hash tag to the error case of several Store.Tree
      functions (#1744 @Ngoguey42). Forward ported from #1583.

  • irmin-containers

    • Removed Irmin_containers.Store_maker; this is now equivalent to
      Irmin.Content_addressable.S (#1369, @samoht)

    • Renamed Irmin_containers.CAS_maker to
      Irmin_containers.Content_addressable (#1369, @samoht)

  • irmin-fs

    • Renamed Irmin_fs.Make into Irmin_fs.Maker (#1369, @samoht)

    • Renamed Irmin_fs.Make_ext into Irmin_fs.Maker_ext (#1369, @samoht)

  • irmin-git

    • All of the configuration keys have moved into their own namespace:

      • Irmin_git.root is now Irmin_git.Conf.root

      • Irmin_git.head is now Irmin_git.Conf.head

      • Irmin_git.bare is now Irmin_git.Conf.bare

      • Irmin_git.level is now Irmin_git.Conf.level

      • Irmin_git.buffers is now Irmin_git.Conf.buffers

      • Irmin_git.dot_git is now Irmin_git.Conf.dot_git
        (#1347, @samoht)

    • Renamed Irmin_git.Make into Irmin_git.Maker (#1369, @samoht)

    • Require at least git.3.7.0 in the codebase (#1632, @dinosaure)

  • irmin-graphql:

    • Changed the name of the default branch node from master to main in the
      GraphQL API. (#1564, @CraigFe)

    • Updated to be compatible with generic keys.

      • The Key type is now called Path to match the new name in irmin

      • All key fields and parameters have been renamed to path
        (#1618, @zshipko)

  • irmin-mirage

    • Renamed Irmin_mirage_git.Make into Irmin_mirage_git.Maker
      (#1369, @samoht)

  • irmin-pack

    • Changed the implementation of backend store keys to use direct pointers to
      store contents (by offset in the pack file) when possible, rather than
      querying the index on each lookup. (#1659, @CraigFe @ngoguey42 @icristescu)

    • The Irmin_pack.Maker module type now no longer takes a Conf argument.
      (#1641, @CraigFe)

    • The backend configuration type Conf.S requires a new parameter
      contents_length_header that (optionally) further specifies the encoding
      format used for commits in order to improve performance. (#1644, @CraigFe)

    • Upgraded on-disk format of pack files to support more efficient lookups and
      reduce indexing overhead. This change is fully backwards-compatible with
      existing stores using irmin-pack.2.x versions, but not
      forwards compatible. (#1649 #1655, @CraigFe @Ngoguey42)

    • Added support for user-specified indexing strategies. The default strategy
      is to index all objects appended to the pack file (as before), but users may
      now choose to index fewer objects in order to improve the write performance
      of the store, at the cost of introducing potential duplicate values to the
      pack file. (#1664, #1761, @CraigFe, @maiste)

  • irmin-unix

    • Clean up command line interface. Allow config file to be specified when
      using Irmin_unix.Resolver.load_config and make command line options
      take precedence over config options.
      (#1464, #1543, #1607 @zshipko)

    • Irmin_unix.Resolver.destruct has been removed (and partially replaced by
      Resolver.spec). (#1603, @CraigFe)

    • Update irmin CLI to support empty path in list subcommand.
      (#1575, @maiste)

    • Add new commands to CLI: branches for listing available branches and
      log which is similar to git log (#1609, #1727, @zshipko)

    • Update irmin watch to take parameters to specify a command that should
      be executed when there are new changes (#1608, @zshipko)

Removed

  • irmin-pack

    • Removed the irmin-pack.layered library. Support for the layered store
      will be restored on a future release of irmin-pack. (#1651, @CraigFe)

    • Removed support for the clear operation in irmin-pack. This operation
      is incompatible with performance optimisations made in this release.
      (#1655, @CraigFe)

  • irmin-layers

    • This experimental package has been removed.

2.10.2 (2022-02-02)

Fixed

  • irmin

    • Fixed a bug causing stream proof extender nodes to have their segments be
      returned in reverse order (i.e. bottom to top, rather then top-down).
      (#1742, @CraigFe)

    • Fixed a bug that allowed the creation of overly-large stable inodes via
      stream proofs. (#1741, @Ngoguey42)

Added

  • irmin

    • Add Store.Private.Node.Val.hash_exn (#1741, @Ngoguey42)

2.10.1 (2022-01-20)

Fixed

  • irmin

    • Fix bug introduced in #1683 which causes Tree.seq and Tree.list to
      produce pruned children (#1720, @Ngoguey42)

2.10.0 (2022-01-07)

Fixed

  • irmin

    • Conversion between proofs and trees are now done in CPS (#1624, @samoht)

    • Better support for s390x to workaround https://github.com/ocaml/ocaml/issues/10857
      (#1694, @icristescu)

  • irmin-pack

    • Fix proofs for large inodes by tracking side-effects reads inside the
      inode implementation (#1670, @samoht, @Ngoguey42)

    • Flush branch store without calling Repo.close (#1707, @zshipko)

Added

  • irmin

    • Add Tree.produce_proof and Tree.verify_proof to produce and verify
      proofs from complex computations. produce_proof and verify_proof
      takes a callback over tree and instead of a static list of operations
      -- this now means that the full Tree API can now be used in proofs,
      including sub-tree operations, folds and paginated lists
      (#1625, #1663, #1683, @samoht, @Ngoguey42)

    • Add Tree.produce_stream and Tree.verify_stream to produce and
      verify stream proofs (#1684, #1692, #1691, @samoht, @Ngoguey42, @icristescu)

  • irmin-pack

    • Verify inode depth invariants (#1665, @samoht)

  • irmin-unix

    • Add tezos store type for irmin command-line (#1678, @zshipko)

Changed

  • irmin

    • Remove Tree.Proof.of_keys. Use Tree.produce_proof instead
      (#1625, @samoht)

    • Tree.empty now takes a unit argument. (#1566, @CraigFe)

    • Tree.length now takes a tree as argument (#1676, @samoht)

    • Tree.Proof.t now uses a more precise datatype to encode value
      invariants (#1688, @samoht)

  • irmin-pack

    • irmin-pack: add an option to configure the index function and pick
      the relevant bits in cryptographic a hash by default (#1677, @samoht)

  • irmin-git

    • Require at least git.3.7.0 in the codebase (#1637, @dinosaure)

2.9.1 (2022-01-10)

Fixed

  • irmin

    • Better support for s390x to workaround
      https://github.com/ocaml/ocaml/issues/10857 (#1694, @icristescu)

2.9.0 (2021-11-15)

Fixed

  • irmin-pack

    • Improved the performance of Index encode and decode operations by
      eliminating intermediate allocations (up to 5% fewer minor words
      allocated) (#1577, @CraigFe)

    • Reduce the number of backend nodes built during export
      (up to 20% fewer minor words allocated) (#1553, @Ngoguey42)

Added

  • irmin

    • Add Merkle Proofs and expose function to convert a proof to and from a tree.
      Once converted, normal tree operations can be performed on the proof, as
      long at it access values contained in the proof.
      (#1583, @samoht, @Ngoguey42, @icristescu)

Changed

  • irmin-pack

    • Limit inode depth (#1596, #samoht)

    • Adapt to index 1.5.0 (#1593, @icristescu)

2.8.0 (2021-10-15)

Fixed

  • irmin

    • Tree operati