package octez-shell-libs

  1. Overview
  2. Docs
package octez-shell-libs
Legend:
Library
Module
Module type
Parameter
Class
Class type

Persistent block store with linear history

The cemented block store is a store where blocks are stored linearly (by level) in chunks. Blocks in this store should not be reorganized anymore and are thus *cemented*. As these blocks should not be accessed regularly and especially their optionally stored metadata, the later are compressed using a zip format to save disk space. For each chunk of blocks, a dedicated file is used. Moreover, to enable easy access and to prevent too much on-disk reading, two indexed maps are used to retrieve blocks hash from their level and their level from the block hash.

The cemented block store contains a set of files updated each time a new chunk is added to the store. These files indicate which interval of blocks (w.r.t. their levels) are stored in it.

Invariants

This store is expected to respect the following invariants:

  • A key/value present in an index is present as well in the other as value/key.
  • Every block stored is correctly referenced through its associated indexes.
  • A cemented chunk of blocks that is represented by the interval i ; j (with i <= j) contains | j - i + 1 | blocks and are ordered from i to j in the file.
  • The set F of cemented chunks is always ordered by block level.
  • The cemented store does not contain holes: let F be the cemented chunks, if |F| > 1 then:
      ∀f_x=(i,j) ∈ F ∧ x < |F|, ∃f_y =(i', j'), x = y - 1 ∧ j + 1 = j'

meaning the concatenation of every chunk must be continuous.

  • A metadata zip file is indexed by the same interval as the chunks and, when it is the lowest chunk of metadata stored, is not assured to contain every block's metadata of the chunk.

Files format

The cemented block store is composed of the following files:

  • file: /<i_j>, a chunk of blocks from level i to level j. The format of this file is:

| <n> × <offset> | <n> × <block> |

where n is (j - i + 1), <offset> is a 4 bytes integer representing the absolute offset of a block where the k-th (with 0 <= k < n) offset stands for the absolute offset of the k-th block in the file and with <block>, a Block_repr.t value encoded using Block_repr.encoding (thus prefixed by the its size).

  • dir: /cemented_block_index_level, the Hash -> Level key/value index ;
  • dir: /cemented_block_index_hash, the Level -> Hash key/value index.
  • dir: /metadata, the directory containing chunks of compressed metadata (present if relevant).
  • files: /metadata/<i_j>.zip, the compressed metadata where every chunk of block's metadata is indexed by their level encoded as string (present if relevant).
module Cemented_block_level_index : Index.S with type key = Tezos_store_shared.Block_key.t and type value = Tezos_store_shared.Block_level.t

On-disk index of block's hash to level.

module Cemented_block_hash_index : Index.S with type key = Tezos_store_shared.Block_level.t and type value = Tezos_store_shared.Block_key.t

On-disk index of block's level to hash.

type t

The type of the cemented block store

type cemented_metadata_file = {
  1. start_level : int32;
  2. end_level : int32;
  3. metadata_file : [ `Cemented_blocks_metadata ] Tezos_store_shared.Naming.file;
}
type cemented_blocks_file = {
  1. start_level : int32;
  2. end_level : int32;
  3. file : [ `Cemented_blocks_file ] Tezos_store_shared.Naming.file;
}

The type for cemented block chunks file description

val init : ?log_size:int -> [< `Chain_dir ] Tezos_store_shared.Naming.directory -> readonly:bool -> t Tezos_base.TzPervasives.tzresult Lwt.t

init ?log_size ~cemented_blocks_dir ~readonly creates or loads an existing cemented block store at path cemented_blocks_dir. cemented_blocks_dir will be created if it does not exists. If readonly is true, cementing blocks will result in an error. log_size determines the index cache size.

val close : t -> unit

close cemented_store closes the cemented_store opened files: its indexes.

val cemented_blocks_files : t -> cemented_blocks_file array option

cemented_blocks_files cemented_store returns the current array of cemented blocks chunks files. The returned array is sorted in ascending order such that the first element of the array is the lowest known cycle of the store.

val cemented_metadata_files : t -> cemented_metadata_file array option Tezos_base.TzPervasives.tzresult Lwt.t

cemented_metadata_files cemented_store returns the current array of cemented metadata files. The returned array is sorted in ascending order such that the first element of the array is the lowest known cycle of the store.

val cemented_block_level_index : t -> Cemented_block_level_index.t

cemented_block_level_index block_store returns the hash to level index.

val cemented_block_hash_index : t -> Cemented_block_hash_index.t

cemented_block_hash_index block_store returns the level to hash index.

val load_table : [ `Cemented_blocks_dir ] Tezos_store_shared.Naming.directory -> cemented_blocks_file array option Tezos_base.TzPervasives.tzresult Lwt.t

load_table ~cemented_blocks_dir reads the cemented_blocks_dir directory and instantiate the cemented blocks chunks files.

val load_metadata_table : [ `Cemented_blocks_dir ] Tezos_store_shared.Naming.directory -> cemented_metadata_file array option Tezos_base.TzPervasives.tzresult Lwt.t

load_metadata_table ~cemented_blocks_dir similar to load_table, but for the cemented metadata files.

val find_block_file : t -> int32 -> cemented_blocks_file option

find_block_file cemented_store block_level lookups the cemented_store to find the cemented block chunk file that contains the block at level block_level. Returns None if the block cannot be found.

val is_cemented : t -> Tezos_base.TzPervasives.Block_hash.t -> bool

is_cemented cemented_store block_hash checks if the block_hash is stored in the cemented_store.

val get_cemented_block_level : t -> Tezos_base.TzPervasives.Block_hash.t -> int32 option

get_cemented_block_level cemented_store block_hash returns the level of the block_hash if present in cemented_store. Returns None otherwise.

val get_cemented_block_hash : t -> int32 -> Tezos_base.TzPervasives.Block_hash.t option

get_cemented_block_hash cemented_store block_level returns the hash of the block at block_level if present in cemented_store. Returns None otherwise.

val read_block_metadata : t -> int32 -> Tezos_store_shared.Block_repr.metadata option Tezos_base.TzPervasives.tzresult Lwt.t

read_block_metadata cemented_store block_level returns the metadata of the block at block_level if present in cemented_store. Returns None otherwise.

val cement_blocks_metadata : t -> Tezos_store_shared.Block_repr.t list -> unit Tezos_base.TzPervasives.tzresult Lwt.t

cement_blocks_metadata cemented_store chunk compresses and stores the metadata of blocks present in chunk. If no block of the given chunk contains metadata, nothing is done. Otherwise, for every block containing metadata, an entry is written in the dedicated .zip metadata file.

We assume that the blocks containing metadata are contiguous and if at least one block has metadata, then the blocks from that block with metadata to the last block of chunk must have metadata. However, we do not check the validity of this assumption.

val get_lowest_cemented_level : t -> int32 option

get_lowest_cemented_level cemented_store returns the lowest cemented block in cemented_store, if it exists.

val get_highest_cemented_level : t -> int32 option

get_highest_cemented_level cemented_store returns the highest cemented block in cemented_store if it exists.

val get_cemented_block_by_level : t -> read_metadata:bool -> int32 -> Tezos_store_shared.Block_repr.block option Tezos_base.TzPervasives.tzresult Lwt.t

get_cemented_block_by_level cemented_store ~read_metadata level reads the cemented block at level in cemented_store, if it exists. It also tries to retrieves the metadata depending on read_metadata but do not fail if no metadata is available.

val get_cemented_block_by_hash : read_metadata:bool -> t -> Tezos_base.TzPervasives.Block_hash.t -> Tezos_store_shared.Block_repr.block option Tezos_base.TzPervasives.tzresult Lwt.t

get_cemented_block_by_hash cemented_store hash reads the cemented block of hash in cemented_store, if it exists. It also retrieves the metadata depending on read_metadata.

type chunk_iterator = {
  1. chunk_length : int;
  2. reading_sequence : (Tezos_base.TzPervasives.Block_hash.t * int * bytes) Tezos_base.TzPervasives.tzresult Lwt.t Tezos_base.TzPervasives.Seq.t;
}

The type used to describe reading sequences used to perform buffered block cementing.

val make_chunk_iterator : Tezos_store_shared.Block_repr.block list -> chunk_iterator

make_chunk_iterator bl is an utility function that transforms a Block_repr.block list into a chunk_iterator

val cement_blocks : ?check_consistency:bool -> t -> write_metadata:bool -> chunk_iterator -> unit Tezos_base.TzPervasives.tzresult Lwt.t

cement_blocks ?check_consistency cemented_store ~write_metadata chunk_iterator iterates from the lazy-bufferized chunk_iterator to cement blocks and to write their metadata if the flag write_metadata is set. check_consistency (default is true) ensures that the cycles are contiguous.

val trigger_gc : t -> Tezos_shell_services.History_mode.t -> unit Lwt.t

trigger_gc cemented_store history_mode garbage collects metadata chunks and/or chunks from the cemented_store depending on the History_mode.t:

  • in Archive mode, nothing is done;
  • in Full offset mode, only offset chunks of metadata are kept;
  • in Rolling offset mode, only offset chunks of metadata and chunks are kept.

Important: when purging chunks of blocks, it is necessary to rewrite the index to remove garbage collected blocks. Therefore, the higher the offset is, the longest the GC phase will last.

val iter_cemented_file : (Tezos_store_shared.Block_repr.block -> unit Lwt.t) -> cemented_blocks_file -> unit Tezos_base.TzPervasives.tzresult Lwt.t

iter_cemented_file ~cemented_block_dir f block_file reads from the cemented block_file located in cemented_block_dir and applies f on every block.

Warning: in this version, exceptions are caught. Use raw_iter_cemented_file for manual exception management.

val raw_iter_cemented_file : (Tezos_store_shared.Block_repr.block -> unit Lwt.t) -> cemented_blocks_file -> unit Lwt.t

Unsafe version of iter_cemented_file where internal exceptions/errors are not caught.

val check_indexes_consistency : ?post_step:(unit -> unit Lwt.t) -> ?genesis_hash:Tezos_base.TzPervasives.Block_hash.t -> t -> unit Tezos_base.TzPervasives.tzresult Lwt.t

check_indexes_consistency ?post_step ?genesis_hash cemented_store history_mode iterates over a partially initialized cemented_store that contains both chunks of blocks and indexes then check the consistency of each block: (hashes, predecessors and levels). The hash is not checked for genesis_hash and post_step is called after each treated chunk. This is used for snapshot imports.

OCaml

Innovation. Community. Security.

GitHub Discord Twitter Peertube RSS
About
Changelog Releases Industrial Users Academic Users Why OCaml
Ecosystem
Packages Community Events OCaml Planet Jobs
Resources
Install OCaml Get Started Platform Tools Language Manual Standard Library API Books Exercises Papers OCaml Playground Logo
Policies
Carbon Footprint Governance Privacy Code of Conduct