14.4 Cannot read an unknown answer

Content and key concepts

On the previous page, let's prepare the scale of the estimates.
When learning to continue, the future words are already in the example, but they cannot be used as an entry point for early prediction.
The request is compared to the keys; the fractions are weighted by the values transferred. The future is not available when predicting continuation. Several heads make independent comparisons.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Prohibition applies before regulation
Causal Mask allows j≤i items, including the current one. Prohibited scores are assigned to , then each query string is separately normalized:
A
i
j
=
exp
⁡
(
S
i
j
+
M
i
j
)
∑
r
exp
⁡
(
S
i
r
+
M
i
r
)
,
M
i
j
=
{
0
j
≤
i
−
∞
j
>
i
.
A_{ij}=\frac{\exp(S_{ij}+M_{ij})}{\sum_r\exp(S_{ir}+M_{ir})},\quad
M_{ij}=\begin{cases}0&j\le i\\-\infty&j>i.\end{cases}
A
ij
​
=
∑
r
​
exp
(
S
i
r
​
+
M
i
r
​
)
exp
(
S
ij
​
+
M
ij
​
)
​
,
M
ij
​
=
{
0
−
∞
​
j
≤
i
j
>
i
.
​
For stability before exp, the maximum allowed scores are deducted. In prohibited cells, the co-officent becomes 0. If you just put score0, its exhibitor will be1, and the ban won't work.
In default, the first query has only one available key, so the line A=[1,0.0]. The second has scores[0.1 G2, - ,]; its co-officents [α,1 - α,0,] where α=1/(1+exp(1/G2).
Experiment.
Change only the last line of the X. The first two output rows at causal mask should remain the same. Then turn off the mask and repeat: the influence of the future may appear. This is a diagnostic abbreviation, the non-acceptable mode of the authored decoder during the nonext-token training phase.
A completely banned line does not have a normal distribution. A standard causal mask with an authorized diagonal does not create it; if a pedding mask is added in the future, this case needs to be dealt with clearly.
from math import exp,sqrt
a=1/(1+exp(1/sqrt(2)))
weights=[a,1-a,0]
assert abs(sum(weights)-1)<1e-12
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
exp(0)=1 does not exclude contribution; exp (-)=0.
How do shares become a mixture? How does this question continue?
Python examples of Chapter 14
Academic base