How to choose a sequel

Content and key concepts

On the previous page, when a model actually learns.
Let's not ask for the whole answer at once: first, let's decide on the next piece.
The model calculates the answer using numerical settings. The program around it prepares the entrance and can trigger a search or a tool.
"Cat"
→
Evaluations of continuation
→
Share of options
→
One piece selected
This is a preliminary map, not a new algorithm that needs to be remembered now. In the next chapters, we'll find out the origin of vectors, estimates and probabilities; a whole computational model will be collected later.
Authoritative factorization
Let the text be presented by a sequence of token IDs
t₀,…,tₙ₋₁
The authorisation model estimates the distribution of the following ID:
P
θ
(
t
n
=
j
∣
t
0
,
…
,
t
n
−
1
)
P_\theta(t_n=j\mid t_0,\ldots,t_{n-1})
P
θ
​
(
t
n
​
=
j
∣
t
0
​
,
…
,
t
n
−
1
​
)
for each
j
The probability of the whole sequence is spread over the chain rule:
P
θ
(
t
0
,
…
,
t
N
−
1
)
=
∏
i
=
0
N
−
1
P
θ
(
t
i
∣
t
<
i
)
.
P_\theta(t_0,\ldots,t_{N-1})
=\prod_{i=0}^{N-1}P_\theta(t_i\mid t_{<i}).
P
θ
​
(
t
0
​
,
…
,
t
N
−
1
​
)
=
i
=
0
∏
N
−
1
​
P
θ
​
(
t
i
​
∣
t
<
i
​
)
.
In learning, the next token is known from the shell, so the objective often contains negative log-likelihood.
L
(
θ
)
=
−
∑
i
ln
⁡
P
θ
(
t
i
∣
t
<
i
)
.
L(\theta)=-\sum_i\ln P_\theta(t_i\mid t_{<i}).
L
(
θ
)
=
−
i
∑
​
ln
P
θ
​
(
t
i
​
∣
t
<
i
​
)
.
When a Target is generated, it is not available: from the given distribution it is selected token, added to the content, and repeated forward. The learning model only executes the page.
One
It's a choice and it doesn't finish the story.
Toy contract
Model below fixed and
untrained
The Vocabulary contains 12 entries, idden width equals 2, one causal application head and one FFN. Layer normalitation is not available. No pre-traded lights, API and equivalence with GPT. The numbers are selected for manual verifiability rather than for the quality of the Russian text.
Tokenizer performs NFC noormalisation, lower-case and regex-battering into groups of letters/chips. An unknown group is receiving ID 0; a line only from puncture or emoji does not create token.
not BPE
III. Real byte-BPE in detail and without substitution of terms implemented in Chapter 12 [
S12
, abstract-only].
Vocabulary in order of IDs:
Table embeddings
E
has a shape
[12,2]
. Lookup picks the line
E[tᵢ]
then add the training position vector
pᵢ=[i/10,−i/10]
:
X
i
=
E
[
t
i
]
+
p
i
.
X_i=E[t_i]+p_i.
X
i
​
=
E
[
t
i
​
]
+
p
i
​
.
The ID number is not a distance or a meaning.
E
It's hand-made. Position vector changes the activation of the current position, but not the parameter table.
Causal scaled attention
In a toy
W_Q=W_K=W_V=W_o=I₂
so numerically
Q=K=V=X
although the roles of these matrices are different.
i
and key position
j
:
D
i
j
=
Q
i
⋅
K
j
2
,
M
i
j
=
{
0
,
j
≤
i
,
−
∞
,
j
>
i
,
D_{ij}=\frac{Q_i\cdot K_j}{\sqrt2},\qquad
M_{ij}=\begin{cases}0,&j\le i,\\-\infty,&j>i,\end{cases}
D
ij
​
=
2
​
Q
i
​
⋅
K
j
​
​
,
M
ij
​
=
{
0
,
−
∞
,
​
j
≤
i
,
j
>
i
,
​
A
i
j
=
exp
⁡
(
D
i
j
+
M
i
j
)
∑
r
exp
⁡
(
D
i
r
+
M
i
r
)
,
B
=
A
V
.
A_{ij}=\frac{\exp(D_{ij}+M_{ij})}
{\sum_r\exp(D_{ir}+M_{ir})},\qquad B=AV.
A
ij
​
=
∑
r
​
exp
(
D
i
r
​
+
M
i
r
​
)
exp
(
D
ij
​
+
M
ij
​
)
​
,
B
=
A
V
.
Softmax applies separately to each line. Permitted co-officents are non-negative and add up to 1; prohibited future entries receive 0.
A
is calculated from the current input and is action, and
W_Q
,
W_K
,
W_V
- Parameters. The word "attention weights" should not hide this difference.
S04
;
S07
].
After the exit of the projection is performed residual
R=X+BW_o
Then the coordinated FFN and the second residual:
F
=
ReLU
⁡
(
R
W
1
+
b
1
)
W
2
+
b
2
,
H
=
R
+
F
.
F=\operatorname{ReLU}(RW_1+b_1)W_2+b_2,\qquad H=R+F.
F
=
ReLU
(
R
W
1
​
+
b
1
​
)
W
2
​
+
b
2
​
,
H
=
R
+
F
.
Here.
W₁=I₂
,
W₂=.5I₂
, biases zero.
H=R+.5ReLU(R)
III. Residual means adding compatible activities rather than adding learning data.
From hidden state to logits
Take the last line.
h=H[n−1,:]
and independent output matrix
U
shape
[2,12]
and bis
c
shape
[12]
:
z
=
h
U
+
c
.
z=hU+c.
z
=
h
U
+
c
.
Each
zⱼ
- Logit candidate. Logits can be negative and not add up to 1.
T>0
:
π
j
=
exp
⁡
(
z
j
/
T
)
∑
r
exp
⁡
(
z
r
/
T
)
.
\pi_j=\frac{\exp(z_j/T)}{\sum_r\exp(z_r/T)}.
π
j
​
=
∑
r
​
exp
(
z
r
​
/
T
)
exp
(
z
j
​
/
T
)
​
.
For numerical stability first compute
aⱼ=zⱼ/T
and subtract the total maximum
a_max
and then the exhibitors.
exp(aⱼ−a_max)
. The overall shift does not change the exact distribution, but reduces the risk overflow [
S10
===Temperature===The term does not change the tokenizer, embeddings, attention or logic.
T
relationship
πₐ/π_b=exp((zₐ−z_b)/T)
becomes more contrasting if the logits aren't equal.
A specific example does not confuse softmax with magical norms:
Logit
Exponent
Probability
0
1
1/6
ln 2
2
1/3
ln 3
3
1/2
CDF equals
[1/6,1/2,1]
III. Number
u=.4
The probability is not a guaranteed schedule frequency, much less the probability of the statement being true.
Trace for cat
For one position, the mask allows only one connection:
Stage
Meaning
Shape
Token IDs
[1]
[1]
E[1]
,
p₀
,
X
[1,0]
,
[0,0]
,
[1,0]
[1,2]
Q
,
K
,
V
[1,0]
[1,2]
each
Score, mask,
A
[1/√2]
,
[0]
,
[1]
[1,1]
B
,
R
[1,0]
,
[2,0]
[1,2]
F
,
H
[1,0]
,
[3,0]
[1,2]
z
[-3,3,0,3,4,−2.4,−3,0,1.5,−1.5,1.5,−1.3]
[12]
For example, logit "windows" is equal to
3·(−1)+0·1+.6=−2.4
. The most logit gets a "cow," but seeded sampling in an animation model can choose
<unk>
This isn't a bang of teaching explanation: weights are really untrained in language.
Sampling builds the CDF in order of IDs and chooses the first
j
for which
u<CDFⱼ
The PNG Park-Miller training uses one draw:
s
n
e
x
t
=
16807
s
m
o
d
2147483647
,
u
=
s
n
e
x
t
−
1
2147483646
.
s_{next}=16807s\bmod 2147483647,
\qquad u=\frac{s_{next}-1}{2147483646}.
s
n
e
x
t
​
=
16807
s
mod
2147483647
,
u
=
2147483646
s
n
e
x
t
​
−
1
​
.
Seed affects
u
but not at
z
and
π
The opening of the inspector shall not waste additional random draws.
Parameter ledger and computational cost
E contains
12·2=24
Scalar slots.
4·2·2=16
. FFN:
4+2+4+2=12
Independent U: 24, output bis: 12.
24
+
16
+
12
+
24
+
12
=
88.
24+16+12+24+12=88.
24
+
16
+
12
+
24
+
12
=
88.
All 88 values are manually set; the trained parameters here are zero. JavaScript Number uses binary64, so Payload numbers are equal to
88·8=704 B
Float32 equivalent is 352 B, but this is not the actual format of the browser array.
S11
].
Scores matrix has
n²
Cell, though causal mask allows
n(n+1)/2
In the field of communications.
n=12
It's 144 cells and 78 authorized positions.
O(n²d)
You can't take time in milliseconds without measuring and accounting for the rest of the operations.
The lower lab opens all 13 stages, matries, and shapes. First, change only time, then only seed: in the first experience, all states must be retained up to the logits inclusive; in the second, also, probabilities.
[<unk>, кошка, сидит, на, ковре, окне, собака, спит, рядом