package matrix
Module Matrix_textSource
Grapheme clusters, display widths, and line and word breaks of Unicode text, as a terminal shows it.
A text is a string of UTF-8 bytes, and every offset into it is a byte offset. Each ill-formed subsequence reads as one U+FFFD (Unicode §3.9, maximal subparts), and the bytes stay as they are.
A grapheme cluster is what a reader takes as one character, such as a letter with its accents or an emoji sequence, as the extended grapheme cluster boundaries of UAX #29 delimit it. A terminal places text in cells. A display unit is one or more scalars that it places as one, in a number of cells, the unit's width. A width method (width_method) says which scalars make a unit and how wide it is, as one kind of terminal does. A control character is a unit of its own, of width 0, except TAB, whose width is the tab width.
The module therefore has two kinds of boundaries. iter_graphemes and grapheme_count follow grapheme clusters whatever the width method, for editing text. measure, iter_units, find_pos and the others of the section Measuring follow display units, for placing it. A Segmenter finds display units in a stream that arrives in pieces, and Layout cuts a text into rows of a given width.
Every function that takes a tab_width raises Invalid_argument if it is below 1.
This measures a text with two wide characters and a TAB:
assert (Matrix_text.measure ~width_method:`Unicode ~tab_width:4 "日本\t" = 8)Display units
A terminal receives its output in pieces of any size and must cut it into the same display units as if it had arrived whole. The grapheme cluster boundary rules of UAX #29 look only backwards, so a boundary is known when the scalar after it arrives.
The units are extended grapheme clusters for `Unicode and `Unicode_wide. For `Wcwidth they are what a terminal that measures each character with wcwidth places, a scalar of positive width and the scalars of width 0 that follow it. In every method a control character (General_Category Cc, which is C0, DEL and C1, TAB included) is a unit of its own, since a terminal performs it. CR LF is therefore two units, where UAX #29 makes one cluster of it (GB3).
The width of a unit never decreases as scalars are added to it. A terminal places a unit before it knows that the unit is complete, and widens it in place as it grows. It could not take back a character that already wrapped the line, so a later scalar never narrows a unit. VS15 (U+FE0E), which asks for the text presentation of an emoji (UTS #51 §4), leaves an emoji that is already two cells wide at two cells.
The type for width methods.
`Unicodemakes each extended grapheme cluster one unit, as wide as its first scalar of positive width, as a terminal does in grapheme cluster mode (mode 2027 of "Unicode in Terminals", and kitty's text-sizing protocol). What continues a cluster, whether a mark, a joined emoji or a conjoined consonant, takes no cells of its own, so the conjunct क्ष (U+0915 U+094D U+0937) takes the one cell of क. A spacing mark (General_Category Mc) has width0, as kitty's protocol gives every mark, so a cluster that one starts, as U+093F does alone, takes no cells, unless the mark is wide (U+302E). A cluster is two cells wide when VS16 follows an emoji or an emoji modifier follows a modifier base, as in ☝🏽 (U+261D U+1F3FD), both of which have the emoji presentation (UTS #51 §2.7.1, §4), when it is a pair of regional indicators, and when it holds Thai SARA AM (U+0E33) or Lao AM (U+0EB3), which kitty's protocol widens.`Unicode_widemakes each extended grapheme cluster one unit, as`Unicodedoes, and a cluster that holds more than one scalar of positive width at least two cells wide, as Ghostty does.`Wcwidthgives each scalar itswcwidthwidth, and makes one unit of each scalar of positive width and the scalars of width0after it.
Characters
uchar_width ~tab_width u is the width of u alone under `Wcwidth: tab_width for TAB, 0 for the other controls and for combining characters, and 1 or 2 otherwise. The other methods give a scalar alone the same width, except that `Unicode gives a spacing mark 0 unless it is wide. Measuring a sequence of scalars takes measure.
The type for the values of the Grapheme_Cluster_Break property, which the boundary rules of UAX #29 read (Table 2). The tags are the property's short value aliases, as in uucp's Uucp.Break.grapheme_cluster, into which a value coerces:
`CRand`LF, CARRIAGE RETURN and LINE FEED, which make one cluster of CR LF (GB3) and none with anything else.`CN, Control, the other controls, the line and paragraph separators, and most format characters. A cluster boundary falls on both sides (GB4, GB5).`EX, Extend, the characters that continue the cluster before them (GB9), which are nonspacing and enclosing marks, emoji modifiers, ZWNJ and variation selectors.`ZWJ, ZERO WIDTH JOINER, which also continues the cluster before it and joins an emoji to the pictograph after it (GB11).`SM, SpacingMark, spacing combining marks that continue the cluster before them (GB9a), as U+0903 DEVANAGARI SIGN VISARGA does.`PP, Prepend, characters that the character after them joins (GB9b), as the digits it precedes join U+0600 ARABIC NUMBER SIGN.`RI, Regional_Indicator, the letters U+1F1E6..U+1F1FF, which pair into flags (GB12, GB13).`L,`V,`T,`LVand`LVT, Hangul leading consonants, vowels, trailing consonants and precomposed syllables, which conjoin into one syllable (GB6 to GB8).`XX, Other, everything else, letters and pictographs among them.
UAX #29 keeps four more values, E_Base, E_Modifier, Glue_After_Zwj and E_Base_GAZ, as "obsolete and unused". No character has had them since Unicode 11.0, so the type leaves them out.
Measuring
These functions read a text's display units, a TAB tab_width cells wide.
The type for positions found by width. byte_offset is the position, units the number of units of positive width before it, and columns_used their width.
measure ~width_method ~tab_width s is the sum of the widths of the units of s.
val measure_sub :
width_method:width_method ->
tab_width:int ->
string ->
pos:int ->
len:int ->
intmeasure_sub ~width_method ~tab_width s ~pos ~len is measure of the len bytes of s from pos, without copying them, and 0 if len <= 0. A scalar cut by either end of the slice is ill-formed, as in the substring.
Raises Invalid_argument if len > 0 and pos and len do not designate a substring of s.
width_at ~width_method ~tab_width s ~byte_offset is the width of the first unit of the text of s from byte_offset, and 0 if byte_offset is outside s.
val iter_units :
width_method:width_method ->
tab_width:int ->
(offset:int -> len:int -> width:int -> unit) ->
string ->
unititer_units ~width_method ~tab_width f s calls f ~offset ~len ~width on each unit of s of positive width, in order, where the unit is the len bytes from offset.
val find_pos :
width_method:width_method ->
tab_width:int ->
?include_start_before:bool ->
string ->
columns:int ->
positionfind_pos ~width_method ~tab_width s ~columns is the end of the longest prefix of s whose units of positive width each end at or before column columns. With ~include_start_before:true, a unit is in the prefix if it starts before column columns, so a wide unit that straddles columns is in it. The units of width 0 before the first unit left out are in the prefix. include_start_before defaults to false.
val find_wrap_pos :
width_method:width_method ->
tab_width:int ->
string ->
max_columns:int ->
positionfind_wrap_pos ~width_method ~tab_width s ~max_columns is find_pos ~width_method ~tab_width s ~columns:max_columns, the longest prefix of s at most max_columns wide. It ends at the start of the first unit that does not fit.
Grapheme clusters
grapheme_count s is the number of grapheme clusters in s.
iter_graphemes f s calls f ~offset ~len on each grapheme cluster of s, in order, where the cluster is the len bytes from offset.
Breaks
A break opportunity is a place where a line may wrap. One follows each grapheme cluster that holds a space, HT, -, /, a backslash, ,, ;, :, !, ?, a bracket, a brace or a parenthesis, or one of U+1680, U+2000..U+2006, U+2008..U+200A, U+200B, U+205F, U+3000, U+00AD, U+2010, U+3001, U+3002, U+FF01 and U+FF1F. The no-break spaces U+00A0, U+2007 and U+202F, of class GL (glue) in UAX #14, make none. One also falls between a cluster that starts with an ASCII letter, digit or _ and one that starts with a CJK character (Han, kana or Hangul), in either order.
An opportunity is reported as ~break_byte_offset, the offset of the cluster before it, ~next_byte_offset, the offset of the cluster after it, where scanning resumes, and ~grapheme_offset, the index of the cluster before it.
val iter_wrap_breaks :
(break_byte_offset:int ->
next_byte_offset:int ->
grapheme_offset:int ->
unit) ->
string ->
unititer_wrap_breaks f s calls f on each break opportunity of s, in order, and also between two adjacent CJK clusters. It makes none of the latter before a kana voicing mark (U+3099, U+309A), U+30A0 or U+30FB, or after a cluster of width 0.
val iter_word_breaks :
(break_byte_offset:int ->
next_byte_offset:int ->
grapheme_offset:int ->
unit) ->
string ->
unititer_word_breaks f s calls f on each break opportunity of s, in order. A run of CJK clusters stays one word, which suits moving the cursor by words and deleting them.
The type for line terminators.
iter_line_breaks f s calls f ~pos ~kind on each line terminator of s, in order, where pos is the offset of its last byte, the LF of a CR LF.