14.6 Several separate ways to collect information

Content and key concepts

On the previous page, how shares become a mixture.
Each head functions with its own portion of the coordinates and its fractions. The results are connected, not averaged automatically.
The request is compared to the keys; the fractions are weighted by the values transferred. The future is not available when predicting continuation. Several heads make independent comparisons.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Several subspaces, then integration
In multi-head, the projection is divided into h-group coordinates. At d_model=d and d_head=d/h each head independently calculates scores, mask, softmax and mixing.
to be completed
rather than average:
MHA
⁡
(
X
)
=
Concat
⁡
(
O
1
,
…
,
O
h
)
W
O
.
\operatorname{MHA}(X)=\operatorname{Concat}(O_1,\ldots,O_h)W_O.
MHA
(
X
)
=
Concat
(
O
1
​
,
…
,
O
h
​
)
W
O
​
.
If d=4,h=2, each hed has a width2. For n=3, each attention matrix [3.3], Head output[3.2], concat[3.4], W_O[4.4]. With fixed d, the dense projection matrix itself preserves the number of parameters; the grouping of dot products and softmax changes.
Experiment.
Select preset3×4 and two heads. Compare their coefficients and outputs; then set one head at the same projection. These are not two copies of the same function: the breakup of the coordinates changes the calculation of similarity.
If d is not divided by h, the configuration of this implementation is rejected. In a full framework, other dimensions of Q/K/V are possible, but they need to be clearly asked. Head with matching projections may duplicate another; the number of heads does not in itself guarantee a richer or better representation.
Under the hood.
The lab keeps all h·n2 co-officents for observation. This is not a promise of optimal memory input from the real inference-engine.
Regulation, Positions and FFN
And we're gonna get a whole decoder lock.
Concat restores total width, W_O mixes coordinates.
How do you connect it to a network you already know?
Python examples of Chapter 14
Academic base