12.5 Large dictionary and short sequence

Content and key concepts

On the previous page, break and restore the text.
More pieces in the dictionary may reduce the number of currents, but increase the table of submissions and the numbers of output estimates.
The breakup is done by a particular dictionary and rules. Byte, symbol, word, and current are not required to match. The number received is the address, not the numerical meaning of the word.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Savings in one place cost resources in another
After the successful merges, our vocabulary contains 256+Mtokens. Frequent byte sequences are given a short representation in the IDs. On the new text, the length may be weaker than the turing corpus; this depends on content and language.
In vocabularyV and embedding with the E matrix, the Vdparameter takes over. The independent LM head adds an additional dV, often also Vbias. Therefore, the larger dictionary makes the input/output tables more expensive. However, the shorter sequence reduces the number of attention-pair n2.
If the sequence is reduced by half, the dense attention matrix has four times less cells. But the entire cost of Transformer is not equal to n2: the projection and the FFN require operations, and the LM head depends on V. You cannot choose the tokenizer for one ratio of selection.
Experiment.
Change requested merges0.24 with fixed corpus and query. Compare the actual mergues, vocables, bytes count and token count. The algorithm can stop earlier than the requested budget; this is not an error. If the quaery consists of unreceived combinations, byte tokens will remain a long sequence.
Boundary.
The tokens obtained do not have to be morphisms or words. The frequent gaps with letters can merge, and the rare letters can remain divided into bytes. This implementation does not contain language-special pre-tokenizer or purport to be the optimal tokenizer for Russian.
V,d,n=256,4,12
assert V*d==1024
assert n*n==144
assert (n//2)**2==36
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
More rows/candidates mean more input/output table parameters.
Different dictionaries have different rules. How does this question continue?
Python examples of Chapter 12
Academic base