Module Uuseg

Unicode text segmentation.

Uuseg segments Unicode text. It implements the locale independent Unicode text segmentation algorithms to detect grapheme cluster, word and sentence boundaries and the Unicode line breaking algorithm to detect line break opportunities.

The module is independent from any IO mechanism or Unicode text data structure and it can process text without a complete in-memory representation.

The supported Unicode version is determined by the unicode_version value.

Consult the basics, limitations and examples of use.

References

• The Unicode Standard. (latest version)
• UAX #29 Unicode Text Segmentation. (latest version)
• UAX #14 Unicode Line Breaking Algorithm. (latest version)
• Web based ICU break utility.

Segment

Custom segmenters

Limitations

A `Grapheme_cluster segmenter will always consume only a small bounded amount of memory on any text. Other segmenters will also do so on non-degenerate text, but it's possible to feed them with input that will make them buffer an arbitrary amount of characters.

Basics

A segmenter is a stateful filter that inputs a sequence of characters and outputs the same sequence except characters are interleaved with `Boundary values whenever the segmenter detects a boundary.

The function create returns a new segmenter for a given boundary type:

let words = Uuseg.create `Word

To add characters to the sequence to segment, call add on words with `Uchar _. To end the sequence call add on words with `End. The segmented sequence of characters is returned character by character, interleaved with `Boundary values at the appropriate places, by the successive calls to add.

The client and the segmenter must wait on each other to limit internal buffering: each time the client adds to the sequence by calling add with `Uchar or `End it must continue to call add with `Await until the segmenter returns `Await or `End. In practice this leads to the following kind of control flow:

let rec add acc v = match Uuseg.add words v with
| `Uchar u -> add (`Uchar u :: acc) `Await
| `Boundary -> add (`B :: acc) `Await
| `Await | `End -> acc

For example to segment the sequence <U+0041, U+0020, U+0042> ("a b") to a list of characters interleaved with `B values on word boundaries we can write:

let uchar = `Uchar (Uchar.of_int u)
let seq = [uchar 0x0041; uchar 0x0020; uchar 0x0042]
let seq_words = List.rev (add (List.fold_left add [] seq) `End)

Examples

utf_8_segments seg s is the list of UTF-8 encoded seg segments of the UTF-8 encoded string s.

let utf_8_segments seg s =
  let flush_segment buf acc =
    let segment = Buffer.contents buf in
    Buffer.clear buf; if segment = "" then acc else segment :: acc
  in
  let rec add buf acc segmenter v = match Uuseg.add segmenter v with
  | `Uchar u -> Buffer.add_utf_8_uchar buf u; add buf acc segmenter `Await
  | `Boundary -> add buf (flush_segment buf acc) segmenter `Await
  | `Await | `End -> acc
  in
  let rec loop buf acc s i max segmenter =
    if i > max then flush_segment buf (add buf acc segmenter `End) else
    let dec = String.get_utf_8_uchar s i in
    let acc = add buf acc segmenter (`Uchar (Uchar.utf_decode_uchar dec)) in
    loop buf acc s (i + Uchar.utf_decode_length dec) max segmenter
  in
  let buf = Buffer.create 42 in
  let segmenter = Uuseg.create seg in
  List.rev (loop buf [] s 0 (String.length s - 1) segmenter)

Note that this function can be derived directly from Uuseg_string.fold_utf_8.