package stem

  1. Overview
  2. Docs
A stemmer of words for few languages

Install 

dune-project
 Dependency

github.com Readme Changelog MIT License Edit opam file Versions (2)

Authors

  1. R
    Romain Calascibetta <romain.calascibetta@gmail.com>

Maintainers

  1. R
    Romain Calascibetta <romain.calascibetta@gmail.com>

Sources

stem-0.0.2.tbz
sha256=830e8693a19996caa7745fc67c6d3655b29128d253ff78088e77faba9ad7b488
sha512=d89a8246864f204fbaa159d7c09517003838913944f4a6235b63efb8e4e96fb1872baaa8563392b8e3a4e2354baf363b5e15ed2899f60d8cbf66bbcfdb6d2d85

doc/stem.tokenizer/Tokenizer/index.html

Module TokenizerSource

Sourcetype behavior =
  1. | Remove
  2. | Isolate
  3. | Merge_with_previous
  4. | Merge_with_next
Sourcetype pattern =
  1. | Whitespace
  2. | Dash
  3. | Bert
  4. | Regex of Re.re
Sourcetype action = pattern * Tokenizer__.Norm.behavior
Sourceval run : ?encoding:Snowball.encoding -> ?to_lowercase:bool -> action list -> string Seq.t -> string Seq.t

run actions str applies actions on the given str.

Sourceval run_on_bstr : ?encoding:Snowball.encoding -> ?to_lowercase:bool -> action list -> Bstr.t -> string Seq.t

run actions bstr applies actions on the given bstr.

NOTE: The advantage of using a bigstring is that you can load a file using Unix.map_file and process the file to obtain a stream of tokens.