Unicode text normalization.
Uunf normalizes Unicode text. It supports all Unicode normalization forms. The module is independent from any IO mechanism or Unicode text data structure and it can process text without a complete in-memory representation of the data.
The supported Unicode version is determined by the
Consult the basics, limitations and examples of use.
Unicode version 15.0.0
- The Unicode Consortium. The Unicode Standard. (latest version)
- Mark Davis. UAX #15 Unicode Normalization Forms. (latest version)
- The Unicode Consortium. Normalization charts.
The type for normalization forms.
`NFDnormalization form D, canonical decomposition.
`NFCnormalization form C, canonical decomposition followed by canonical composition (recommended for the www).
`NFKDnormalization form KD, compatibility decomposition.
`NFKCnormalization form KC, compatibility decomposition, followed by canonical composition.
add n v is:
uis the next character in the normalized sequence. The client must then call
`Awaitwhen the normalizer is ready to add a new
`Uchar u to add a new character to the sequence to normalize and
`End to signal the end of sequence. After adding one of these two values, always call
`Await is returned.
`End is added directly after an
`Uchar was returned by the normalizer or if an
`Uchar is added after
`End was added.
val reset : t -> unit
reset n resets the normalizer to a state equivalent to the state of
Uunf.create (Uunf.form n).
copy n is a copy of
n in its current state. Subsequent
n do not affect the copy.
val pp_ret : Format.formatter -> ret -> unit
pp_ret ppf v prints an unspecified representation of
These properties are used internally to implement the normalizers. They are not needed to use the module but are exposed as they may be useful to implement other algorithms.
val ccc : Uchar.t -> int
ccc u is
u's canonical combining class value.
val decomp : Uchar.t -> int array
decomp u is
u's decomposition mapping. If the empty array is returned,
u decomposes to itself.
The first number in the array contains additional information, it cannot be used as an
d_uchar on the number to get the actual character and
d_compatibility to find out if this is a compatibility decomposition. All other characters of the array are guaranteed to be convertible using
Warning. Do not mutate the array.
composite u1 u2 is the primary composite canonically equivalent to the sequence
<u1,u2>, if any.
Uunf normalizer consumes only a small bounded amount of memory on ordinary, meaningful text. However on legal but degenerate text like a starter followed by 10'000 combining non-spacing marks it will have to bufferize all the marks (a workaround is to first convert your input to stream-safe text format).
A normalizer is a stateful filter that inputs a sequence of characters and outputs an equivalent sequence in the requested normal form.
create returns a new normalizer for a given normal form:
let nfd = Uunf.create `NFD
To add characters to the sequence to normalize, call
`Uchar _. To end the sequence, call
`End. The normalized sequence of characters is returned, character by character, by the successive calls to
The client and the normalizer must wait on each other to limit internal buffering: each time the client adds to the sequence by calling
`End it must continue to call
`Await until the normalizer returns
`Await. In practice this leads to the following kind of control flow:
let rec add acc v = match Uunf.add nfd v with | `Uchar u -> add (u :: acc) `Await | `Await | `End -> acc
For example to normalize the character
U+00E9 (é) with
nfd to a list of characters we can write:
let e_acute = Uchar.of_int 0x00E9 let e_acute_nfd = List.rev (add (add  (`Uchar e_acute)) `End)
The next section has more examples.
utf_8_normalize nf s is the UTF-8 encoded normal form
nf of the UTF-8 encoded string
s. This example uses
Uutf to fold over the characters of
s and to encode the normalized sequence in a standard OCaml buffer.
let utf_8_normalize nf s = let b = Buffer.create (String.length s * 3) in let n = Uunf.create nf in let rec add v = match Uunf.add n v with | `Uchar u -> Uutf.Buffer.add_utf_8 b u; add `Await | `Await | `End -> () in let add_uchar _ _ = function | `Malformed _ -> add (`Uchar Uutf.u_rep) | `Uchar _ as u -> add u in Uutf.String.fold_utf_8 add_uchar () s; add `End; Buffer.contents b
Note that this functionality is available directly through