15.6 From the presentation in the evaluation of words

Content and key concepts

On the previous page, several blocks in a row.
The last table is evaluated for each dictionary candidate. Probability and choice are determined by the following individual actions.
The vector obtains positional information, exchanges information and updates its status separately. Residual amounts maintain the same path. The last table gives estimates of the dictionary.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
The last line predicts the next ID
After final norm, each line H corresponds to a processed prefix. Lineline projection gives logs on vocabular:
Z
=
H
U
+
c
∈
R
n
×
V
,
p
(
t
i
+
1
=
j
∣
t
≤
i
)
=
softmax
⁡
(
Z
i
)
j
.
Z=HU+c\in\mathbb R^{n\times V},\qquad
p(t_{i+1}=j\mid t_{\le i})=\operatorname{softmax}(Z_i)_j.
Z
=
H
U
+
c
∈
R
n
×
V
,
p
(
t
i
+
1
​
=
j
∣
t
≤
i
​
)
=
softmax
(
Z
i
​
)
j
​
.
For the current extension, the last line is used. During the future training, Targets will be moved into one current; you cannot compare output positions to input positions.
Individual manual head: h=[1,-1],U=[1.0.1],[0.1],c0.Logits[1, - 1.0], the norm divides [e,1/e,1] by their sum. All three scores are received by one general section, not three pre-prepared answers.
Experiment.
All logs and probabilities are available on the inspector, not only top features. The IDs order is aligned with a specific BPE document. A single candidate may contain incomplete UTF-8 sequences: then it appears bytes/hex rather than a fictional word.
Our seeded webs are not yet trained: high probabilities do not confirm grammar or truth. already working tokenizer, E, full stem and head;
The next chapter 16 should add the hull, tags, backward/optimizer, validation and checkpoints
It is the training that will make it possible to talk about a learned small language model, not just the right architecture.
from math import exp
h=[1,-1]; U=[[1,0,1],[0,1,1]]
z=[sum(h[i]*U[i][j] for i in range(2)) for j in range(3)]
p=[exp(v)/sum(exp(a) for a in z) for v in z]
assert z==[1,-1,0] and abs(sum(p)-1)<1e-12
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Tokenizer rales are trained separately; pre-training models will appear in 16.
The arithmetic works, but how do you find the useful values of all its coefficients?
Python examples of Chapter 15
Academic base