package saga

  1. Overview
  2. Docs
Text processing and NLP extensions for Nx

Install 

dune-project
 Dependency

github.com Readme Changelog Edit opam file Versions (1)

Authors

  1. Thibaut Mattio

Maintainers

  1. Thibaut Mattio

Sources

raven-1.0.0.alpha1.tbz
sha256=8e277ed56615d388bc69c4333e43d1acd112b5f2d5d352e2453aef223ff59867
sha512=369eda6df6b84b08f92c8957954d107058fb8d3d8374082e074b56f3a139351b3ae6e3a99f2d4a4a2930dd950fd609593467e502368a13ad6217b571382da28c

doc/saga.tokenizers/Saga_tokenizers/Added_token/index.html

Module Saga_tokenizers.Added_tokenSource

Sourcetype t

Added token with customization, matching HF Added_token.

Sourceval create : ?content:string -> ?single_word:bool -> ?lstrip:bool -> ?rstrip:bool -> ?normalized:bool -> ?special:bool -> unit -> t

Create added token with options and defaults.

Sourceval content : t -> string
Sourceval lstrip : t -> bool
Sourceval normalized : t -> bool
Sourceval rstrip : t -> bool
Sourceval single_word : t -> bool