package matrix

Build not yet checked
  1. Overview
  2. Docs
Fast, modern terminal toolkit for OCaml

Install

dune-project
 Dependency

Authors

Maintainers

Sources

mosaic-0.3.0.tbz
sha256=a92cbbcbe016fb19fb0a749222da0e459a9567aef3f4c7340d9bd6b71c53ed6f
sha512=6a7ab054605538bb31a2b73a1a6d093981a2704c9f0af5852221c05735290d74aca7c7282298c4d0f91e86a800e2ccd94b4918af883cd963449c758eb6b272b3

doc/matrix.text/Matrix_text/index.html

Module Matrix_textSource

Grapheme clusters, display widths, and line and word breaks of Unicode text, as a terminal shows it.

A text is a string of UTF-8 bytes, and every offset into it is a byte offset. Each ill-formed subsequence reads as one U+FFFD (Unicode §3.9, maximal subparts), and the bytes stay as they are.

A grapheme cluster is what a reader takes as one character, such as a letter with its accents or an emoji sequence, as the extended grapheme cluster boundaries of UAX #29 delimit it. A terminal places text in cells. A display unit is one or more scalars that it places as one, in a number of cells, the unit's width. A width method (width_method) says which scalars make a unit and how wide it is, as one kind of terminal does. A control character is a unit of its own, of width 0, except TAB, whose width is the tab width.

The module therefore has two kinds of boundaries. iter_graphemes and grapheme_count follow grapheme clusters whatever the width method, for editing text. measure, iter_units, find_pos and the others of the section Measuring follow display units, for placing it. A Segmenter finds display units in a stream that arrives in pieces, and Layout cuts a text into rows of a given width.

Every function that takes a tab_width raises Invalid_argument if it is below 1.

This measures a text with two wide characters and a TAB:

assert (Matrix_text.measure ~width_method:`Unicode ~tab_width:4 "日本\t" = 8)

Display units

A terminal receives its output in pieces of any size and must cut it into the same display units as if it had arrived whole. The grapheme cluster boundary rules of UAX #29 look only backwards, so a boundary is known when the scalar after it arrives.

The units are extended grapheme clusters for `Unicode and `Unicode_wide. For `Wcwidth they are what a terminal that measures each character with wcwidth places, a scalar of positive width and the scalars of width 0 that follow it. In every method a control character (General_Category Cc, which is C0, DEL and C1, TAB included) is a unit of its own, since a terminal performs it. CR LF is therefore two units, where UAX #29 makes one cluster of it (GB3).

The width of a unit never decreases as scalars are added to it. A terminal places a unit before it knows that the unit is complete, and widens it in place as it grows. It could not take back a character that already wrapped the line, so a later scalar never narrows a unit. VS15 (U+FE0E), which asks for the text presentation of an emoji (UTS #51 §4), leaves an emoji that is already two cells wide at two cells.

Sourcetype width_method = [
  1. | `Unicode
  2. | `Unicode_wide
  3. | `Wcwidth
]

The type for width methods.

  • `Unicode makes each extended grapheme cluster one unit, as wide as its first scalar of positive width, as a terminal does in grapheme cluster mode (mode 2027 of "Unicode in Terminals", and kitty's text-sizing protocol). What continues a cluster, whether a mark, a joined emoji or a conjoined consonant, takes no cells of its own, so the conjunct क्ष (U+0915 U+094D U+0937) takes the one cell of क. A spacing mark (General_Category Mc) has width 0, as kitty's protocol gives every mark, so a cluster that one starts, as U+093F does alone, takes no cells, unless the mark is wide (U+302E). A cluster is two cells wide when VS16 follows an emoji or an emoji modifier follows a modifier base, as in ☝🏽 (U+261D U+1F3FD), both of which have the emoji presentation (UTS #51 §2.7.1, §4), when it is a pair of regional indicators, and when it holds Thai SARA AM (U+0E33) or Lao AM (U+0EB3), which kitty's protocol widens.
  • `Unicode_wide makes each extended grapheme cluster one unit, as `Unicode does, and a cluster that holds more than one scalar of positive width at least two cells wide, as Ghostty does.
  • `Wcwidth gives each scalar its wcwidth width, and makes one unit of each scalar of positive width and the scalars of width 0 after it.
Sourcemodule Segmenter : sig ... end

The display units of a stream that arrives in pieces.

Characters

Sourceval uchar_width : tab_width:int -> Uchar.t -> int

uchar_width ~tab_width u is the width of u alone under `Wcwidth: tab_width for TAB, 0 for the other controls and for combining characters, and 1 or 2 otherwise. The other methods give a scalar alone the same width, except that `Unicode gives a spacing mark 0 unless it is wide. Measuring a sequence of scalars takes measure.

Sourcetype grapheme_cluster_break = [
  1. | `CN
  2. | `CR
  3. | `EX
  4. | `L
  5. | `LF
  6. | `LV
  7. | `LVT
  8. | `PP
  9. | `RI
  10. | `SM
  11. | `T
  12. | `V
  13. | `XX
  14. | `ZWJ
]

The type for the values of the Grapheme_Cluster_Break property, which the boundary rules of UAX #29 read (Table 2). The tags are the property's short value aliases, as in uucp's Uucp.Break.grapheme_cluster, into which a value coerces:

  • `CR and `LF, CARRIAGE RETURN and LINE FEED, which make one cluster of CR LF (GB3) and none with anything else.
  • `CN, Control, the other controls, the line and paragraph separators, and most format characters. A cluster boundary falls on both sides (GB4, GB5).
  • `EX, Extend, the characters that continue the cluster before them (GB9), which are nonspacing and enclosing marks, emoji modifiers, ZWNJ and variation selectors.
  • `ZWJ, ZERO WIDTH JOINER, which also continues the cluster before it and joins an emoji to the pictograph after it (GB11).
  • `SM, SpacingMark, spacing combining marks that continue the cluster before them (GB9a), as U+0903 DEVANAGARI SIGN VISARGA does.
  • `PP, Prepend, characters that the character after them joins (GB9b), as the digits it precedes join U+0600 ARABIC NUMBER SIGN.
  • `RI, Regional_Indicator, the letters U+1F1E6..U+1F1FF, which pair into flags (GB12, GB13).
  • `L, `V, `T, `LV and `LVT, Hangul leading consonants, vowels, trailing consonants and precomposed syllables, which conjoin into one syllable (GB6 to GB8).
  • `XX, Other, everything else, letters and pictographs among them.

UAX #29 keeps four more values, E_Base, E_Modifier, Glue_After_Zwj and E_Base_GAZ, as "obsolete and unused". No character has had them since Unicode 11.0, so the type leaves them out.

Sourceval grapheme_cluster_break : Uchar.t -> grapheme_cluster_break

grapheme_cluster_break u is the Grapheme_Cluster_Break property of u, by the Unicode version of the tables Segmenter follows. It describes u alone. Whether a boundary falls between two characters also depends on rules that look further back, which only a Segmenter applies. It allocates nothing.

Measuring

These functions read a text's display units, a TAB tab_width cells wide.

Sourcetype position = {
  1. byte_offset : int;
  2. units : int;
  3. columns_used : int;
}

The type for positions found by width. byte_offset is the position, units the number of units of positive width before it, and columns_used their width.

Sourceval measure : width_method:width_method -> tab_width:int -> string -> int

measure ~width_method ~tab_width s is the sum of the widths of the units of s.

Sourceval measure_sub : width_method:width_method -> tab_width:int -> string -> pos:int -> len:int -> int

measure_sub ~width_method ~tab_width s ~pos ~len is measure of the len bytes of s from pos, without copying them, and 0 if len <= 0. A scalar cut by either end of the slice is ill-formed, as in the substring.

Raises Invalid_argument if len > 0 and pos and len do not designate a substring of s.

Sourceval width_at : width_method:width_method -> tab_width:int -> string -> byte_offset:int -> int

width_at ~width_method ~tab_width s ~byte_offset is the width of the first unit of the text of s from byte_offset, and 0 if byte_offset is outside s.

Sourceval iter_units : width_method:width_method -> tab_width:int -> (offset:int -> len:int -> width:int -> unit) -> string -> unit

iter_units ~width_method ~tab_width f s calls f ~offset ~len ~width on each unit of s of positive width, in order, where the unit is the len bytes from offset.

Sourceval find_pos : width_method:width_method -> tab_width:int -> ?include_start_before:bool -> string -> columns:int -> position

find_pos ~width_method ~tab_width s ~columns is the end of the longest prefix of s whose units of positive width each end at or before column columns. With ~include_start_before:true, a unit is in the prefix if it starts before column columns, so a wide unit that straddles columns is in it. The units of width 0 before the first unit left out are in the prefix. include_start_before defaults to false.

Sourceval find_wrap_pos : width_method:width_method -> tab_width:int -> string -> max_columns:int -> position

find_wrap_pos ~width_method ~tab_width s ~max_columns is find_pos ~width_method ~tab_width s ~columns:max_columns, the longest prefix of s at most max_columns wide. It ends at the start of the first unit that does not fit.

Grapheme clusters

Sourceval grapheme_count : string -> int

grapheme_count s is the number of grapheme clusters in s.

Sourceval iter_graphemes : (offset:int -> len:int -> unit) -> string -> unit

iter_graphemes f s calls f ~offset ~len on each grapheme cluster of s, in order, where the cluster is the len bytes from offset.

Breaks

A break opportunity is a place where a line may wrap. One follows each grapheme cluster that holds a space, HT, -, /, a backslash, ,, ;, :, !, ?, a bracket, a brace or a parenthesis, or one of U+1680, U+2000..U+2006, U+2008..U+200A, U+200B, U+205F, U+3000, U+00AD, U+2010, U+3001, U+3002, U+FF01 and U+FF1F. The no-break spaces U+00A0, U+2007 and U+202F, of class GL (glue) in UAX #14, make none. One also falls between a cluster that starts with an ASCII letter, digit or _ and one that starts with a CJK character (Han, kana or Hangul), in either order.

An opportunity is reported as ~break_byte_offset, the offset of the cluster before it, ~next_byte_offset, the offset of the cluster after it, where scanning resumes, and ~grapheme_offset, the index of the cluster before it.

Sourceval iter_wrap_breaks : (break_byte_offset:int -> next_byte_offset:int -> grapheme_offset:int -> unit) -> string -> unit

iter_wrap_breaks f s calls f on each break opportunity of s, in order, and also between two adjacent CJK clusters. It makes none of the latter before a kana voicing mark (U+3099, U+309A), U+30A0 or U+30FB, or after a cluster of width 0.

Sourceval iter_word_breaks : (break_byte_offset:int -> next_byte_offset:int -> grapheme_offset:int -> unit) -> string -> unit

iter_word_breaks f s calls f on each break opportunity of s, in order. A run of CJK clusters stays one word, which suits moving the cursor by words and deleting them.

Sourcetype line_break_kind = [
  1. | `LF
  2. | `CR
  3. | `CRLF
]

The type for line terminators.

Sourceval iter_line_breaks : (pos:int -> kind:line_break_kind -> unit) -> string -> unit

iter_line_breaks f s calls f ~pos ~kind on each line terminator of s, in order, where pos is the offset of its last byte, the LF of a CR LF.

Layout

Sourcemodule Layout : sig ... end

Texts cut into rows of a given width, as a text view or an editor shows them.