package granary
Install
dune-project
Dependency
Authors
Maintainers
Sources
sha256=8b18780ea373be48301d9f333925860a2f9110fc0ac28684295118d72b65a67e
sha512=25ca3c9c5e2b528704a542502e0f37dc33ba003f65622d969b8c2b800778585f8ef0cf89b36e6679832e3993e8303aecddfc662742baf7044d6afe4a796b8f11
doc/granary.catalog/Granary_catalog/Catalog/index.html
Module Granary_catalog.CatalogSource
In-memory catalog backed by system trees in the Store. System tree allocation: 0=_sys_tables, 1=_sys_columns, 2=_sys_indexes, 3=_sys_meta, 4=_sys_fts_tables, 5=_sys_views, 6=_sys_triggers, 7=_sys_catalog_mirror (#174 redundant schema copy). User tables and indexes use tree_ids >= 16.
Pretty-print a summary: table/index/FTS counts and FK-enforcement flag.
Kind of pending FK check (matched against current row state at commit).
Re-verification callback for a queued FK violation. Universally polymorphic in the transaction mode so it accepts either the active write txn (deferred path inside an explicit transaction) or an RO snapshot opened after auto-commit.
type pending_fk_check = {pfk_kind : pending_fk_kind;pfk_table : string;pfk_rowid : int64;pfk_message : string;pfk_recheck : pending_fk_recheck;(*Re-run the FK check. An open transaction (the active write txn about to be committed, or a fresh RO snapshot taken AFTER an auto-commit) is supplied so the recheck observes the live state — opening a separate RO snapshot mid-transaction on the B+-tree backend would see only the pre-txn state and report spurious violations.
Returns true if STILL violated; false if the violation has been resolved (e.g. parent row now exists, child row now deleted).
*)
}A queued FK violation awaiting re-verification at commit.
type storage = | Row of {tree_id : Granary_store.Store.tree_id;next_rowid : int64;(*Next rowid to auto-allocate.
*)empty_next_rowid= never seeded.without_rowid : bool;(*WITHOUT ROWID — phase 37 #122.
*)autoincrement : bool;(*#299: sticky rowid high-water.
*)
}| Columnar of Granary_columnar.Col_store.t * Granary_store.Store.tree_id(*Columnar table: in-memory store plus the tree_id allocated for persistence. The Col_store is empty on re-open after restart (M1 behaviour); the tree_id is reserved for T2 block-region persistence.
*)
type table_meta = {name : string;storage : storage;columns : Granary_encoding.Row.column list;fk_constraints : fk_constraint list;
}Extract the Row payload. Raises if storage = Columnar.
Returns (tree_id, next_rowid, without_rowid, autoincrement).
tid_of_storage s returns the tree_id for Row or Columnar storage. Columnar tables with a legacy dummy tree_id return -1.
Sentinel next_rowid for a rowid/alias table that has never seeded its counter — conceptually max = -inf. Used to tell "never inserted" apart from a live counter (e.g. an AUTOINCREMENT table absent from sqlite_sequence until its first insert). See #250/#312.
val compute_rowid_alias_col :
Granary_encoding.Row.column list ->
without_rowid:bool ->
int option#243 (T1): index of the INTEGER PRIMARY KEY column that aliases the rowid, if the table qualifies (rowid table, single INTEGER PRIMARY KEY column). For such tables the table tree is keyed by this column's value and no separate __pk index exists. None for every other shape.
rowid_alias_col m = compute_rowid_alias_col m.columns ~without_rowid.
None for columnar tables (no rowid).
#243 (T1): record the rowid of the most recently INSERTed row (for last_insert_rowid()). Set by the executor after each successful insert.
#243 (T1): the rowid recorded by the last set_last_inserted_rowid; read by the db layer to answer last_insert_rowid().
How an index came to exist. `Implicit_pk and `Implicit_unique are auto-created from a table's PRIMARY KEY / UNIQUE constraints at create_table time; `User is an explicit CREATE INDEX. The logical dump (#264) keys on this field to decide which indexes a replayed CREATE TABLE already recreates, rather than sniffing the __pk_/__uniq_ name prefix the binder assigns (which a user index could collide with).
type index_info = {idx_name : string;idx_table : string;idx_columns : string list;idx_unique : bool;idx_tree_id : Granary_store.Store.tree_id;idx_expr_flags : bool list;idx_where_sql : string option;idx_origin : idx_origin;
}type fts_table_meta = {fts_name : string;fts_content_tree : Granary_store.Store.tree_id;fts_index_tree : Granary_store.Store.tree_id;fts_columns : string list;
}Open (or initialise) a catalog on the given store. Loads all existing table metadata and index metadata from the store.
Read the user_version from the sys_meta tree inside an already-open transaction (RO or RW). Returns 0 if not yet set.
val write_user_version_tx :
Granary_store.Store.rw Granary_store.Store.txn ->
int64 ->
unit Lwt.tWrite user_version to the sys_meta tree inside an already-open RW transaction. Caller is responsible for the commit.
val create_table :
?txn:Granary_store.Store.rw Granary_store.Store.txn ->
t ->
name:string ->
columns:Granary_encoding.Row.column list ->
without_rowid:bool ->
autoincrement:bool ->
Granary_store.Store.tree_id Lwt.tCreate a new table, returning its assigned tree_id. Raises Failure if a table with that name already exists.
?txn (#269): run the creation — tree-ID allocation, catalog rows, and cache insertion — through this already-held explicit writer transaction instead of opening (and committing) a fresh one. This lets CREATE TABLE participate in an ambient BEGIN … COMMIT (no self-deadlock) and be undone by rollback_schema_changes on ROLLBACK.
Find a table by name. Returns None if not found.
Synchronous in-memory lookup (no Lwt). Always up to date since the cache is updated on every DDL operation.
Schema fingerprint (#174) of a table: a stable 64-bit hash of its shape (columns + WITHOUT ROWID), computed from the in-memory cache via Granary_encoding.Schema_fingerprint. None if no such table. Stable across table renames; used for drift detection on open, the redundant catalog mirror, per-page fingerprint stamps, and replication schema matching (#92 / #172).
(tree_id, fingerprint) for every persistent table currently in the catalog (ephemeral CTE entries are excluded). Used to register per-tree page-header stamps with the store (#174).
(tree_id, fingerprint) recorded in the redundant catalog mirror (#174). On a healthy database this agrees with fingerprints_by_tree_id; a divergence indicates schema drift or corruption in one of the two copies.
type schema_discrepancy = | Fingerprint_mismatch of {tree_id : Granary_store.Store.tree_id;primary : int64;(*fingerprint computed from the primary catalog
*)mirror : int64;(*fingerprint recorded in the mirror
*)
}| Missing_in_mirror of Granary_store.Store.tree_id(*a table present in the primary catalog has no mirror entry
*)| Missing_in_primary of Granary_store.Store.tree_id(*the mirror records a table the primary catalog does not have
*)
A disagreement between the primary catalog and the redundant mirror (#174).
Cross-check the primary catalog against the redundant mirror and report every disagreement (#174). An empty list means the two copies agree. A non-empty list signals schema drift or corruption; open_ also logs a warning for any Fingerprint_mismatch it finds, but stays openable so recovery tooling can run.
Temporarily register an ephemeral (CTE) table entry in the in-memory cache. tree_id = -1 is the sentinel for CTE virtual tables.
Remove a previously registered ephemeral table entry from the in-memory cache.
List all known tables. Order is unspecified.
val persist_dirty_columnar_stores :
t ->
Granary_store.Store.rw Granary_store.Store.txn ->
Granary_columnar.Col_store.t list Lwt.tpersist_dirty_columnar_stores t tx iterates all columnar tables in the catalog and persists any whose Granary_columnar.Col_store.dirty flag is set, using the given write transaction. Returns the list of stores that were persisted so callers can clear their dirty flags after commit.
load_columnar_stores t store iterates all columnar tables in the catalog and loads their data from store via a fresh RO transaction. Used on DB open to restore persisted columnar data. Tables with no persisted data are reset to empty.
Allocate and return the next rowid for a table, incrementing the counter. Raises Failure if the table does not exist.
val next_rowid_in_txn :
?defer_counter:bool ->
t ->
name:string ->
Granary_store.Store.rw Granary_store.Store.txn ->
int64 Lwt.tLike next_rowid but operates within an already-held RW transaction. Does NOT commit; the caller owns the commit. Use within explicit transactions to avoid deadlocking on the store's writer mutex.
val bump_next_rowid_in_txn :
?defer_counter:bool ->
t ->
name:string ->
at_least:int64 ->
Granary_store.Store.rw Granary_store.Store.txn ->
unit Lwt.t#243 (T1): advance a table's autoincrement counter so the next allocation is at least at_least (= an explicitly-inserted INTEGER PRIMARY KEY value + 1), matching SQLite. Never lowers the counter; does NOT commit.
val set_next_rowid_in_txn :
t ->
name:string ->
requested:int64 ->
Granary_store.Store.rw Granary_store.Store.txn ->
unit Lwt.t#312.1: writable sqlite_sequence SET/INSERT. Sets table name's autoincrement counter to max(requested + 1, max(rowid) + 1) (SQLite's effective rule). Mutates through the held RW transaction (does NOT commit) and marks the counter dirty so a ROLLBACK reverts it. Fails if name is unknown or not an AUTOINCREMENT table.
val reset_next_rowid_in_txn :
t ->
name:string ->
Granary_store.Store.rw Granary_store.Store.txn ->
unit Lwt.t#312.1: writable sqlite_sequence DELETE. Resets table name's autoincrement counter to the never-seeded sentinel so the next insert recomputes from data. Mutates through the held RW transaction (does NOT commit) and marks the counter dirty so a ROLLBACK reverts it. Fails if name is unknown or not an AUTOINCREMENT table.
val reset_all_next_rowid_in_txn :
t ->
Granary_store.Store.rw Granary_store.Store.txn ->
unit Lwt.t#312.1: writable sqlite_sequence DELETE with no WHERE — reset every seeded AUTOINCREMENT counter (SQLite parity; what sqlite3 .dump emits). Mutates through the held RW transaction (does NOT commit).
val note_rowid_deleted :
t ->
name:string ->
rowid:int64 ->
Granary_store.Store.rw Granary_store.Store.txn ->
unit Lwt.t#409: note that rowid was deleted from plain rowid table name. When the deleted row was the current high-water (next_rowid - 1), recompute the cached counter to max(rowid) + 1 over the live rows (scanned within the held transaction, so it reflects the just-applied delete) so the rowid is reused on the next insert, matching SQLite. Mutates through the held RW transaction (does NOT commit) and marks the counter dirty so a ROLLBACK reverts it. A no-op for AUTOINCREMENT tables (sticky high-water), WITHOUT ROWID tables, unseeded counters, and deletes of any non-high-water row.
val flush_dirty_counters_tx :
t ->
Granary_store.Store.rw Granary_store.Store.txn ->
unit Lwt.t#347: flush all dirty rowid counters into the B-tree under tx. Call once before S.commit when inserts used ~defer_counter:true to skip per-row put_table_counter_tx writes. Clears the dirty set; the subsequent commit_schema_changes reset is a no-op.
val create_index :
?txn:Granary_store.Store.rw Granary_store.Store.txn ->
t ->
name:string ->
table:string ->
columns:string list ->
unique:bool ->
expr_flags:bool list ->
where_sql:string option ->
origin:idx_origin ->
(index_info, string) result Lwt.tCreate a new index on a single column of an existing table. The new index gets its own tree_id (separate from the table tree). The index metadata is persisted in the _sys_indexes tree. The actual index entries are NOT populated here; callers (the executor) must scan the table and emit index entries within the same logical operation.
Errors:
`Error msgif a table withtabledoes not exist.`Error msgif an index namednamealready exists.`Error msgif any column incolumnsis not a column oftable.
?txn (#269): run through this already-held explicit writer transaction instead of opening (and committing) a fresh one.
Return the list of indexes on the given table. Order is unspecified.
Look up an index by name.
val find_index_covering_cols :
t ->
table_name:string ->
col_idxs:int list ->
index_info optionfind_index_covering_cols cat ~table_name ~col_idxs returns an index whose leading key columns match col_idxs (in order), and which is safe to use for FK enforcement scans:
- all columns are plain (no expression columns);
- no partial-index WHERE clause. Returns
Noneif no such index exists.
The index may have more columns than col_idxs; only the leading prefix is required to match.
val drop_table :
t ->
Granary_store.Store.rw Granary_store.Store.txn ->
name:string ->
unit Lwt.tRemove a table and all its indexes from the catalog. Removes entries from _sys_tables, _sys_columns, and _sys_indexes. The B+-tree pages for the table and its indexes are NOT reclaimed (Phase 3).
val drop_index :
t ->
Granary_store.Store.rw Granary_store.Store.txn ->
name:string ->
unit Lwt.tRemove an index from _sys_indexes. The B+-tree pages are NOT reclaimed (Phase 3).
val add_column :
?txn:Granary_store.Store.rw Granary_store.Store.txn ->
t ->
table_name:string ->
column:Granary_encoding.Row.column ->
(unit, string) result Lwt.tAdd a new column to an existing table. Updates the catalog's persistent storage and in-memory cache. Returns Error msg if the table does not exist or the column already exists.
?txn (#282): write through this already-held explicit writer transaction instead of opening a fresh one, and register a schema-cache undo so a ROLLBACK restores the prior table shape.
val rename_table :
?txn:Granary_store.Store.rw Granary_store.Store.txn ->
t ->
old_name:string ->
new_name:string ->
(unit, string) result Lwt.tRename a table. Updates _sys_tables, re-keys all _sys_columns entries, and refreshes the in-memory cache and any index entries that reference the old table name. Returns Error msg if old_name does not exist or new_name already exists.
?txn (#282): as for add_column.
val rename_column :
?txn:Granary_store.Store.rw Granary_store.Store.txn ->
t ->
table_name:string ->
old_col:string ->
new_col:string ->
(unit, string) result Lwt.tRename a column within a table. Updates the _sys_columns entry and refreshes the in-memory cache. Returns Error msg if the table or column does not exist.
?txn (#282): as for add_column.
val drop_column :
?txn:Granary_store.Store.rw Granary_store.Store.txn ->
t ->
table_name:string ->
col_name:string ->
(unit, string) result Lwt.tRemove a column from an existing table. Re-keys all column entries with ordinal > drop_idx (shift down by 1). Does NOT migrate existing row data — caller (the executor) is responsible. Returns Error msg if the table or column does not exist.
?txn (#282): as for add_column.
val create_columnstore_table :
?txn:Granary_store.Store.rw Granary_store.Store.txn ->
t ->
name:string ->
columns:Granary_encoding.Row.column list ->
unit Lwt.tcreate_columnstore_table ?txn t ~name ~columns registers a columnar table in the catalog (sys_tables + sys_columns), allocates no B-tree. Returns unit (no tree_id — columnar tables have no B-tree).
Find an FTS table by name. Returns None if not found.
List all FTS virtual tables in the catalog. Order is unspecified.
val create_fts_table :
?txn:Granary_store.Store.rw Granary_store.Store.txn ->
t ->
name:string ->
columns:string list ->
fts_table_meta Lwt.tCreate a new FTS virtual table. Allocates two new tree IDs (content tree and inverted-index tree) and persists the entry in the _sys_fts_tables system tree. Raises Failure if a table with that name already exists.
?txn (#269): run the creation through this already-held explicit writer transaction instead of opening (and committing) a fresh one.
val next_fts_rowid_in_txn :
t ->
name:string ->
Granary_store.Store.rw Granary_store.Store.txn ->
int64 Lwt.tAllocate and return the next rowid for an FTS table within an already-held RW transaction. Does NOT commit; the caller owns the commit.
val ensure_fts_rowid_above_in_txn :
t ->
name:string ->
Granary_store.Store.rw Granary_store.Store.txn ->
int64 ->
unit Lwt.t#330: advance an FTS table's rowid high-water so the next auto-allocated rowid is past rowid (used after an explicit-rowid insert). No-op when the counter is already beyond it. Does NOT commit; the caller owns the commit.
val save_fk_constraints :
?txn:Granary_store.Store.rw Granary_store.Store.txn ->
t ->
table_name:string ->
fks:fk_constraint list ->
unit Lwt.tPersist FK constraints for a table to the sys_meta B-tree.
?txn (#269): write through this already-held explicit writer transaction instead of opening (and committing) a fresh one.
Update FK constraints in the in-memory cache for a table.
#269: register an in-memory schema-cache reversal for DDL run through an explicit transaction. The closure is invoked by rollback_schema_changes if the transaction is rolled back; it is discarded by commit_schema_changes on commit. Used by the db layer for view/trigger caches it owns.
The closure MUST be idempotent (e.g. Hashtbl.replace/remove over values captured at registration, not read-modify-write of live state). #280's savepoint_rollback_schema reverts the cache by re-running the closures registered since a savepoint; its defensive "snapshot not reached" branch can also leave an already-run closure queued for the outer ROLLBACK. Both rely on a second invocation being a no-op.
#269: discard the pending schema-undo closures — the transaction's DDL is now durable. Call at COMMIT (and savepoint-release auto-commit).
#269: run the pending schema-undo closures (most-recent-first), reverting the in-memory cache mutations of DDL whose store writes were just rolled back. Call at ROLLBACK.
#293: re-derive the cached next_rowid from the data tree (max(rowid)+1, as at open-time) for ONLY the rowid tables bumped during the rolled-back transaction, discarding the in-memory counter bump made by INSERTs run through it. This makes the next rowid allocation reuse a rolled-back rowid, matching SQLite for plain (non-AUTOINCREMENT) rowid tables. Restricting to the bumped set (usually one table) keeps rollback ~O(1) instead of scanning every table's tree. Clears the bumped set. Must be called by the db layer AFTER the store rollback (so the trees show last-committed state and the RW lock is free), and after rollback_schema_changes (which leaves the bumped set intact for this call to consume).
#280: open a schema-undo savepoint named name. Records the current undo-log position so a later savepoint_rollback_schema/ savepoint_release_schema of name can act on only the entries registered since this point. Call from the db layer's SAVEPOINT handler.
#280: ROLLBACK TO savepoint name — run (most-recent-first) and drop the schema-undo closures registered since name was opened, reverting their in-memory cache mutations to agree with the store rolled back to name. Keeps name (re-rollback-able) and drops newer savepoints. No-op if name is unknown.
#280: RELEASE savepoint name — drop the marker (and newer ones) without running any undo; the since-name entries merge into the enclosing scope so an outer ROLLBACK still unwinds them. No-op if name is unknown.
#286: mark the ambient explicit transaction uncommittable. Called when an in-txn DDL statement fails partway through — its borrowed txn is left open with partial on-disk effects, so a subsequent COMMIT must be forced to roll back instead. Cleared by commit_schema_changes/rollback_schema_changes.
#286: whether the active explicit transaction has been poisoned by a failed in-txn DDL statement. The db layer checks this at COMMIT and rolls back (returning an error) instead of persisting partial DDL effects.
Load all persisted view definitions. Returns (view_name, create_view_sql) pairs.
#322/#323: load all view definitions through a caller-supplied snapshot, so Db.dump can read view DDL under its single shared RO snapshot and keep the schema section point-in-time consistent with the row data.
val persist_view :
?txn:Granary_store.Store.rw Granary_store.Store.txn ->
Granary_store.Store.t ->
name:string ->
sql:string ->
unit Lwt.tPersist a view's SQL text to the sys_views B-tree.
?txn (#269): write through this already-held explicit writer transaction instead of opening (and committing) a fresh one.
val remove_view :
?txn:Granary_store.Store.rw Granary_store.Store.txn ->
Granary_store.Store.t ->
name:string ->
unit Lwt.tRemove a view's SQL text from the sys_views B-tree. ?txn as above (#269).
#427: load all reactive-view definitions as (name, create_sql) pairs.
val persist_reactive_view :
?txn:Granary_store.Store.rw Granary_store.Store.txn ->
Granary_store.Store.t ->
name:string ->
sql:string ->
unit Lwt.t#427: persist a reactive view's CREATE REACTIVE VIEW SQL text. ?txn as for persist_view.
val remove_reactive_view :
?txn:Granary_store.Store.rw Granary_store.Store.txn ->
Granary_store.Store.t ->
name:string ->
unit Lwt.t#427: remove a reactive view's SQL text. ?txn as above.
Load all persisted trigger definitions. Returns (trigger_name, create_trigger_sql) pairs.
#322/#323: trigger-DDL counterpart to load_all_views_in_tx.
val persist_trigger :
?txn:Granary_store.Store.rw Granary_store.Store.txn ->
Granary_store.Store.t ->
name:string ->
sql:string ->
unit Lwt.tPersist a trigger's CREATE TRIGGER SQL to the sys_triggers B-tree. Call this whenever CREATE TRIGGER is executed.
?txn (#269): write through this already-held explicit writer transaction instead of opening (and committing) a fresh one.
val remove_trigger :
?txn:Granary_store.Store.rw Granary_store.Store.txn ->
Granary_store.Store.t ->
name:string ->
unit Lwt.tRemove a trigger's SQL from the sys_triggers B-tree. Call this whenever DROP TRIGGER is executed. ?txn as above (#269).
Set the FK enforcement flag (PRAGMA foreign_keys = 0/1).
Get the current recursive-triggers flag (default true in granary — a deliberate divergence from real SQLite which defaults OFF). When this flag is false, DML executed inside a trigger body does NOT fire further triggers.
Set the recursive-triggers flag (PRAGMA recursive_triggers = 0/1).
Get the runtime PRAGMA defer_foreign_keys flag (default false). When ON every FK enforcement site behaves as DEFERRED for the duration of the transaction. SQLite resets this flag to OFF at every transaction end.
Append a pending FK check to be drained at commit time.
Return and clear the pending FK check list.
Clear the pending FK check list without raising (used on rollback).
Peek at the current pending FK check count (debugging/tests).
Underlying store handle. Exposed so subsystems (FK deferred rechecks) can open their own RO snapshots without threading the store through every function signature.