package granary

  1. Overview
  2. Docs
Pure-OCaml SQL engine

Install 

dune-project
 Dependency

github.com Edit opam file Versions (1)

Authors

  1. T
    Tej Pochiraju <foss@tej.sh>

Maintainers

  1. T
    Tej Pochiraju <foss@tej.sh>

Sources

0.0.3.tar.gz
sha256=8b18780ea373be48301d9f333925860a2f9110fc0ac28684295118d72b65a67e
sha512=25ca3c9c5e2b528704a542502e0f37dc33ba003f65622d969b8c2b800778585f8ef0cf89b36e6679832e3993e8303aecddfc662742baf7044d6afe4a796b8f11

doc/granary.sql/Granary_sql/Fts_tokenizer/index.html

Module Granary_sql.Fts_tokenizerSource

FTS word tokenizer.

Splits text on non-alphanumeric boundaries, lowercases each token, and assigns monotonically-increasing position numbers per column. High-byte UTF-8 sequences (>127) are treated as word characters so that multi-byte Unicode letters are not split on byte boundaries.

Sourcetype token = {
  1. term : string;
    (*

    lowercased word

    *)
  2. col : int;
    (*

    which input column (0-indexed)

    *)
  3. pos : int;
    (*

    word position within this column (0-indexed)

    *)
  4. start_byte : int;
    (*

    byte offset of first byte of raw token in source string

    *)
  5. end_byte : int;
    (*

    byte offset one past last byte of raw token

    *)
}
Sourceval tokenize_string : col:int -> string -> token list

tokenize_string ~col text tokenizes a single text string as column col. Tokens are numbered starting at 0 within this column.

Sourceval tokenize : (int * string) list -> token list

tokenize col_texts tokenizes a list of (col_index, text) pairs. Tokens for each column are numbered starting at 0.