12.1 Letters, bytes and currents — not the same

Content and key concepts

The computer stores lines through code units and bytes.
An external sign can be written in several code units, first we divide the way the line is stored and the rules of its breakup by the model.
The breakup is done by a particular dictionary and rules. Byte, symbol, word, and current are not required to match. The number received is the address, not the numerical meaning of the word.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Four ways to count text
The number of visible signs, Unicode code points, UTF-16 code units and UTF-8 bytes may vary.
.length
Considers UTF-16 units.
Intl.Segmenter
It is not yet the language model.
The "Y" line has two code points, three UTF-16 units and six UTF-8 bytes. The letter is divided into two bytes, emoji four. The "é" sign can be written by one U+00E9 or two code points: e and combining acute contact. External lines are similar, bytes differ.
Experiment.
Add a kyrillitz, emoji, a space, a line translation and a composite sign. Compare code point U+..., UTF-8 bytes and grapheme count. You cannot replace absence.
Intl.Segmenter
A approximate number and declare it accurate.
Normalization of NFC may make equivalent records the same, but it's a change in the image. In our reversible beyte-tokenizer, it is
not implemented
UTF-16 surrogates are rejected: standard UTF-8 coding would replace their U+FFD, breaking the original line restoration promise.
Communication with LLM.
The context window is usually measured by currents, not letters, or bytes. For one sentence, different tokenizers can produce different numbers of IDs. Before comparing the size of the context, it is the tokenizer of the model that needs to know.
text='Я🙂'
assert len(text)==2
assert len(text.encode('utf-16-le'))//2==3
assert len(text.encode('utf-8'))==6
assert 'é'.encode('utf-8') != 'e\u0301'.encode('utf-8')
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Emoji may occupy two UTF-16 units and one code point.
The dictionary asks the number. How does this question continue the result?
Python examples of Chapter 12
Academic base