12.2 The dictionary gives the number value

Content and key concepts

On the previous page, the letter, bytes, and token are not the same.
The number indicates the entry in a particular dictionary. If the dictionary has changed, the previous representation table is no longer required.
The breakup is done by a particular dictionary and rules. Byte, symbol, word, and current are not required to match. The number received is the address, not the numerical meaning of the word.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Index is an address, not a meaning.
Vocabulary sets the token toID display. For dimension V, the whole IDs of 0 to V-1 are allowed. The number is used to select the line embedding matrix; the arithmetical distance between IDs is not semantic distance.
In our byte-BPE, the reference dictionary contains all 256 bytes. ID97 corresponds to the ASCII byte.
a
, ID98 —
b
, ID32 is a gap. ID0 here means byte0,
not to add and not unknown
Special BOS/EOS currents would require a separate contract; they are not now available.
After merge, the new token keeps a sequence of parents' byte and receives a new ID. The byte sequence of one token does not have to be completed by UTF-8. For example, one byte multibyte letter cannot honestly be called a separate symbol. The Inspector shows hex and the text only where separate decoding is acceptable.
Experiment.
Find the original bytes of the cyrillic letter in the table. Then turn on the merges and find the token containing several bytes. It is important to distinguish between the size of the dictionary, the number of currents of a particular text and the numbers of unique currents encountered in it.
Boundary.
Full coating 256 byte allows new correct Unicode lines to be coded without
<unk>
but it doesn't mean that the model can understand them. Embedding for a rare byte combination can be poorly trained. Tokenizer and neurosmart are connected, but different parts of the system.
assert list('ab '.encode('utf-8'))==[97,98,32]
assert bytes([97,98,32]).decode('utf-8')=='ab '
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
The initial IDs of 0.255 correspond to the bytes themselves; special tokens are not added.
Frequent neighbouring pieces come together. How does this matter continue?
Python examples of Chapter 12
Academic base