package kaun

  1. Overview
  2. Docs
Flax-inspired neural network library for OCaml

Install 

dune-project
 Dependency

github.com Readme Changelog Edit opam file Versions (2)

Authors

  1. Thibaut Mattio

Maintainers

  1. Thibaut Mattio

Sources

raven-1.0.0.alpha1.tbz
sha256=8e277ed56615d388bc69c4333e43d1acd112b5f2d5d352e2453aef223ff59867
sha512=369eda6df6b84b08f92c8957954d107058fb8d3d8374082e074b56f3a139351b3ae6e3a99f2d4a4a2930dd950fd609593467e502368a13ad6217b571382da28c

doc/kaun.models/Kaun_models/Bert/Tokenizer/index.html

Module Bert.TokenizerSource

Sourcetype t

BERT tokenizer instance

Sourceval create : ?vocab_file:string -> ?model_id:string -> unit -> t

Create a WordPiece tokenizer for BERT. Either provide a vocab_file path or a model_id to download from HuggingFace (defaults to bert-base-uncased)

Sourceval create_wordpiece : ?vocab_file:string -> ?model_id:string -> unit -> t

Alias for create

Sourceval encode_to_array : t -> string -> int array

Encode text to token IDs with CLS and SEP tokens

Sourceval encode : t -> string -> inputs

Encode text directly to input tensors ready for forward pass

Sourceval encode_batch : t -> ?max_length:int -> ?padding:bool -> string list -> (int32, Rune.int32_elt) Rune.t

Encode multiple texts with padding and special tokens

Sourceval decode : t -> int array -> string

Decode token IDs back to text