Module Saga_tokenizersSource

Tokenization library for Saga.

This module provides the main tokenization API matching HuggingFace Tokenizers design. It supports multiple tokenization algorithms (BPE, WordPiece, Unigram, Word-level, Character-level), text normalization, pre-tokenization, post-processing, and decoding.

Quick Start

Load a pretrained tokenizer:

  let tokenizer = Tokenizer.from_file "tokenizer.json" |> Result.get_ok in
  let encoding = Tokenizer.encode tokenizer "Hello world!" in
  let ids = Encoding.get_ids encoding

Create a BPE tokenizer from scratch:

  let tokenizer =
    Tokenizer.bpe
      ~vocab:[("hello", 0); ("world", 1); ("[PAD]", 2)]
      ~merges:[]
      ()
  in
  let encoding = Tokenizer.encode tokenizer "hello world" in
  let text = Tokenizer.decode tokenizer [0; 1]

Train a new tokenizer:

  let texts = [ "Hello world"; "How are you?"; "Hello again" ] in
  let tokenizer =
    Tokenizer.train_bpe (`Seq (List.to_seq texts)) ~vocab_size:1000 ()
  in
  Tokenizer.save_pretrained tokenizer ~path:"./my_tokenizer"

Architecture

Tokenization proceeds through stages:

• Normalization: Clean and normalize text (lowercase, accent removal, etc.)
• Pre-tokenization: Split text into words or subwords
• Tokenization: Apply vocabulary-based encoding (BPE, WordPiece, etc.)
• Post-processing: Add special tokens, set type IDs
• Padding/Truncation: Adjust length for batching

Each stage is optional and configurable via builder methods.

Post-processing patterns are model-specific:

• BERT: Adds CLS at start, SEP at end, type IDs distinguish sequences
• GPT-2: No special tokens by default, uses BOS/EOS if configured
• RoBERTa: Uses <s> and </s> tokens similar to BERT but different format

Special Token Constructors