12.3. Frequent adjacent pieces are assembled

Content and key concepts

On the previous page, the dictionary gives the number.
So we start with bytes, we count the next pairs, and we combine the selected pair. The frequency of the pair and the number of non-interrupting replacements can vary.
The breakup is done by a particular dictionary and rules. Byte, symbol, word, and current are not required to match. The number received is the address, not the numerical meaning of the word.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
We're gonna do couples, then we'll replace them.
BPE iterativeally combines frequent adjacent units. The original NMT work L02 uses a character/subword units inside words.
Byte-level Flow
including gaps: this is a real merge-algorithm, but not an exact copy of production tokenizer.
On each iteration, all adjacent pairs of current IDs are counted. Select the maximum frequency, with equals, the lower numeric pair (leftID, lightID). The new token receives the ID256+merge number. The entrances are replaced from left to right without closing. If the best pair is less than 2, stop.
For corpus
banana banana
The pairs (a,n) and (n,a) meet four times. Tie-break chooses (97.110), the new ID256 means bytes.
an
After the first merge
banana
After four merges, the whole expression becomes [259.32.259], where 259--
banana
.
Thickness overlap.
B
aaa
A pair (a,a) has a frequency, but only one non-overtended replacement can be made over the passage. The candidate's frequency and the number of replacements performed are different fields of the track.
Research.
Repeat the hull and compare the first pair, its count and actual replicas. Tie-break can change the IDs and later segmentation. Therefore, the list of merges and their order are part of the stored tokenizer model, not the random design of the report.
ids=list(b'banana')
result=[]; i=0
while i<len(ids):
if ids[i:i+2]==[97,110]: result.append(256); i+=2
else: result.append(ids[i]); i+=1
assert result==[98,256,256,97]
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
In aaa, aa pair has a count2, but one non-overlapping replacement.
Broke and restore the text. How does this matter continue?
Python examples of Chapter 12
Academic base