package owl

  1. Overview
  2. Docs
package owl
Legend:
Library
Module
Module type
Parameter
Class
Class type

NLP: Vocabulary module

Type definition
type t

Type of vocabulary (or dictionary).

Query vocabulary
val get_w2i : t -> (string, int) Hashtbl.t

get_w2i v returns word -> index mapping of v.

val get_i2w : t -> (int, string) Hashtbl.t

get_i2w v returns index -> word mapping of v.

val exits_w : t -> string -> bool

exits_w v w returns true if word w exists in the vocabulary v.

val exits_i : t -> int -> bool

exits_i i w returns true if index i exists in the vocabulary v.

val word2index : t -> string -> int

word2index v w converts word w to its index using vocabulary v.

val index2word : t -> int -> string

index2word v i converts index i to its corresponding word using vocabulary v.

Obtain properties
val length : t -> int

length v returns the size of vocabulary v.

val freq_i : t -> int -> int

freq_i v i returns the frequency of word of index i.

val freq_w : t -> string -> int

freq_w v w returns the frequency of word w in the vocabulary v.

val sort_freq : ?inc:bool -> t -> (int * int) array

sort_freq v returns the vocabulary as a (index, freq) array in increasing or decreasing frequency specified by parameter inc.

val top : t -> int -> (string * int) array

top v k returns the top k words in vocabulary v.

val bottom : t -> int -> (string * int) array

bottom v k returns the bottom k words in vocabulary v.

val re_index : t -> t

re_index v re-indexes the indices of words in vocabulary v.

Core functions
val build : ?lo:float -> ?hi:float -> ?alphabet:bool -> ?stopwords:(string, 'a) Hashtbl.t -> string -> t

build ~lo ~hi ~stopwords fname builds a vocabulary from a text corpus file of name fname. If alphabet=false then tokens are the words separated by white spaces; if alphabet=true then tokens are the characters and a vocabulary of alphabets is returned.

Parameters: * lo: percentage of lower bound of word frequency. * hi: percentage of higher bound of word frequency. * alphabet : build vocabulary for alphabets or words. * fname: file name of the text corpus, each line contains a doc.

val build_from_string : ?lo:float -> ?hi:float -> ?alphabet:bool -> ?stopwords:(string, 'a) Hashtbl.t -> string -> t

build_from_string is similar to build but builds the vocabulary from an input string rather than a file.

val trim_percent : lo:float -> hi:float -> t -> t

trim_percent ~lo ~hi v remove extremely low and high frequency words based on percentage of frequency.

Parameters: * lo: the percentage of lower bound. * hi: the percentage of higher bound.

val trim_count : lo:int -> hi:int -> t -> t

trim_count ~lo ~hi v remove extremely low and high frequency words based on absolute count of words.

Parameters: * lo: the lower bound of number of occurrence. * hi: the higher bound of number of occurrence.

val remove_stopwords : ('a, 'b) Hashtbl.t -> ('a, 'c) Hashtbl.t -> unit

remove_stopwords stopwords v removes the stopwords defined in a hashtbl from vocabulary v.

val copy : t -> t

copy v makes a copy of vocabulary v.

val tokenise : t -> string -> int array

tokenise v s tokenises the string s according to the vocabulary v.

val w2i_to_tuples : t -> (string * int) list

w2w2i_to_tuples v converts vocabulary v to a list of (word, index) tuples.

val to_array : t -> (int * string) array

to_array v converts a vocabulary to a (index, word) array.

val of_array : (int * string) array -> t

of_array v converts a (index, word) array to a vocabulary.

I/O functions
val save : t -> string -> unit

save v fname serialises the vocabulary and saves it to a file of name s.

val load : string -> t

load fname loads the serialised vocabulary from a file of name fname.

val save_txt : t -> string -> unit

save_txt v fname saves the vocabulary in the text format to a file of name s.

val to_string : t -> string

to_string v returns the summary information of a vocabulary.

val pp_vocab : Format.formatter -> t -> unit

Pretty printer for vocabulary type.

OCaml

Innovation. Community. Security.

GitHub Discord Twitter Peertube RSS
About
Changelog Releases Industrial Users Academic Users Why OCaml
Ecosystem
Packages Community Events OCaml Planet Jobs
Resources
Install OCaml Get Started Platform Tools Language Manual Standard Library API Books Exercises Papers OCaml Playground Logo
Policies
Carbon Footprint Governance Privacy Code of Conduct