14.1 Why words exchange data

Content and key concepts

We know how to transform vectors and turn estimates into fractions.
To process one word, the information about the neighbor is useful. First we compare the request and the keys, then we mix the transmitted values.
The request is compared to the keys; the fractions are weighted by the values transferred. The future is not available when predicting continuation. Several heads make independent comparisons.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Mixing depends on the current entrance
The coordinated FFN processs each item separately. To connect the representations of different currents, you need an exchange mechanism. Attension forms output as a weighted amount of valutes, and co-officents depend on query and keys. No literal "word question" has a querey vector.
o
i
=
∑
j
a
i
j
v
j
,
a
i
j
≥
0
,
∑
j
a
i
j
=
1.
o_i=\sum_j a_{ij}v_j,\qquad a_{ij}\ge0,\quad\sum_j a_{ij}=1.
o
i
​
=
j
∑
​
a
ij
​
v
j
​
,
a
ij
​
≥
0
,
j
∑
​
a
ij
​
=
1.
For values[1.0] and [0.1], weights[.25,75] are received.[25,75]. This mixture is not necessarily a copy of one vale. If all scores are equal, the regulation gives equal co-officers in accessible positions.
Experiment.
Change one line X and see which outputs have changed. In causal mode, the late position should not affect the earlier position. In unasked mode, this path is allowed.
Easy to be wrong.
Co-officents output is not a parameters W. It is re-released at the entrance. A bright heatmap does not in itself prove the human interpretation of "important" and is not the complete causal map of the big model's solution.
values=[[1,0],[0,1]]; weights=[.25,.75]
out=[sum(a*v[j] for a,v in zip(weights,values)) for j in range(2)]
assert out==[.25,.75]
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Weights output depends on the entrance and is normalized by keys.
Three roles of one submission. How does this issue continue?
Python examples of Chapter 14
Academic base