12.4 Break and restore the text

Content and key concepts

On the previous page, frequent adjacent pieces come together.
We're going to apply the dictionary's rules, and we're gonna get all the bytes together first, then decode the string.
The breakup is done by a particular dictionary and rules. Byte, symbol, word, and current are not required to match. The number received is the address, not the numerical meaning of the word.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Training and application of rules are divided
During encode, the new text does not re-train the tokenizer. The string becomes UTF-8 bytes, then the previously learned pairs are applied in order of rank. On decode IDs, all bytes are connected and decoded.
one
UTF-8 through.
Why can't you decode each token separately and glue the strings together? Some tokens contain only part of a multibyte sign. The Unicode application recorder U+FFFD will permanently damage the text. Therefore, the main check:
decode
⁡
(
encode
⁡
(
s
)
)
=
s
.
\operatorname{decode}(\operatorname{encode}(s))=s.
decode
(
encode
(
s
))
=
s
.
This equality applies to the permissible Unicode strings of our tokenizer without hidden normalization. The gaps, liine breaks and initial BOM are retained. The empty line is coded empty list and back; the empty training corpus for merges learning is separately rejected.
Experiment.
Copy the IDs received to manual decode mode. Change one ID to an unknown or form an incorrect UTF-8 sequence: there should be an understandable error, not a false "failing rollback." Byte fallsback allows for the presentation of any validal text, but does not guarantee the validity of an arbitrary random bytes sequence.
Easy to be wrong.
Decode arbitrary IDs is not obliged to return the grammatical text. The Decoder vocabular and tokenizer vocable must coincide with the content and order. If you re-train merges by keeping the old weights embedding, the lines can be broken.
s=' Я🙂\n'
ids=list(s.encode('utf-8'))
assert bytes(ids).decode('utf-8')==s
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
One token may contain an incomplete UTF-8 sequence.
A big dictionary and a short sequence. How does this question go on with the result?
Python examples of Chapter 12
Academic base