package nx-text

  1. Overview
  2. Docs

Module Nx_text.UnicodeSource

Unicode text processing utilities

Sourcetype normalization =
  1. | NFC
    (*

    Canonical Decomposition, followed by Canonical Composition

    *)
  2. | NFD
    (*

    Canonical Decomposition

    *)
  3. | NFKC
    (*

    Compatibility Decomposition, followed by Canonical Composition

    *)
  4. | NFKD
    (*

    Compatibility Decomposition

    *)
Sourcetype char_category =
  1. | Letter
  2. | Number
  3. | Punctuation
  4. | Symbol
  5. | Whitespace
  6. | Control
  7. | Other

Character Classification

Sourceval categorize_char : Uchar.t -> char_category

categorize_char u returns Unicode category of character

Sourceval is_whitespace : Uchar.t -> bool

is_whitespace u checks if character is whitespace

Sourceval is_punctuation : Uchar.t -> bool

is_punctuation u checks if character is punctuation

Sourceval is_word_char : Uchar.t -> bool

is_word_char u checks if character is letter or number

Sourceval is_cjk : Uchar.t -> bool

is_cjk u checks if character is Chinese/Japanese/Korean

Text Normalization

Sourceval normalize : normalization -> string -> string

normalize form text applies Unicode normalization.

Sourceval case_fold : string -> string

case_fold text performs Unicode case folding .

Sourceval strip_accents : string -> string

strip_accents text removes diacritical marks.

Sourceval clean_text : ?remove_control:bool -> ?normalize_whitespace:bool -> string -> string

clean_text ?remove_control ?normalize_whitespace text cleans text.

  • remove_control: Remove control characters
  • normalize_whitespace: Collapse whitespace

Text Processing

Sourceval split_words : string -> string list

split_words text splits on Unicode word boundaries.

Handles CJK text where each character is typically a word.

Sourceval grapheme_count : string -> int

grapheme_count text counts user-perceived characters.

Sourceval is_valid_utf8 : string -> bool

is_valid_utf8 text validates UTF-8 encoding

Sourceval remove_emoji : string -> string

remove_emoji text removes emoji and symbols.