14.2 Three roles of one submission

Content and key concepts

On the previous page, why do words exchange data?
The request -- what to compare with -- the key -- what's compared, the value -- what is to convey -- is an analogy of roles; all three objects are numerical vectors.
The request is compared to the keys; the fractions are weighted by the values transferred. The future is not available when predicting continuation. Several heads make independent comparisons.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Three roles of one submission
For X[n,d_model] matrices W_Q,W_K,W _V translate each row into query, key and vale. In a simple case, the widths are the same:
Q
=
X
W
Q
,
K
=
X
W
K
,
V
=
X
W
V
,
W
∗
∈
R
d
×
d
.
Q=XW_Q,\quad K=XW_K,\quad V=XW_V,\qquad W_*\in\mathbb R^{d\times d}.
Q
=
X
W
Q
​
,
K
=
X
W
K
​
,
V
=
X
W
V
​
,
W
∗
​
∈
R
d
×
d
.
If W_Q=I, then Q=X. But identity is a projection matrix; Q is not required to be square or single. In default X =[1.0],[0.1], [1.1] has Shape[3.2] and each projection[2.2].
At W_Q=[2.0],[0.1]] the first quary becomes [2.0] the third [2.1]. W_K and W_V can be left unchanged: then the method of comparing query is changed, but not values themselves. In a real model, these matrices are studied in conjunction with los; the lab slider only sets them up manually.
Experiment.
First, change W_Q, then W_V. In the first case, change scores and co-officents; in the second case, the Q/K co-afficients remain the same, change the content of the mixture. Shapes are checked before calculation: arbitrary reshape does not correct inconsistent matrices.
X=[[1,0],[0,1],[1,1]]; W=[[2,0],[0,1]]
Q=[[sum(row[k]*W[k][j] for k in range(2)) for j in range(2)] for row in X]
assert Q==[[2,0],[0,1],[2,1]]
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Symilarity is defined by Q/K, the contents of the mixture are defined by V.
How does this matter continue?
Python examples of Chapter 14
Academic base