14.3 Prepare the scale of assessments

Content and key concepts

On the previous page: Three roles of the same representation.
In large vectors, the amounts of the works can have greater dispersion. The root distribution of the head size helps to control the scale before the estimates become fractions.
The request is compared to the keys; the fractions are weighted by the values transferred. The future is not available when predicting continuation. Several heads make independent comparisons.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Dot product receives a controlled scale
For query/key dimension d_k the reference score is their scalar work. Scaled dot-product output uses the division on Gd_k:
S
=
Q
K
T
/
d
k
,
S
∈
R
n
×
n
.
S=QK^T/\sqrt{d_k},\qquad S\in\mathbb R^{n\times n}.
S
=
Q
K
T
/
d
k
​
​
,
S
∈
R
n
×
n
.
If the components are independent, centralized, and have a unit dispersion, the variance of the sum d_k production is d_K. The division on Gd_k returns the magnitude of the order1. This is the motivation from L06,
not automatically true for any manual Q/K input
.
For the third quary[1.1] and keys[1.0],[0.1], [1.1] dots=[1.1.2]. After scaling this[1/ G2.1/G2,2], without scalling the maximum and the rest more, softmax becomes more concentrated.
Experiment.
Turn off scaling at the same X/W. Compare co-officents, especially when increasing the element module. The order of the scores is retained, but the distribution can change markedly. With zero Q, all spores are equal to 0, and scalming has no effect.
Under the hood.
For npositions and width d_k, n2dot products of length d_K are calculated. Linear projections themselves have additional cost; the formula attention-matrix does not describe all FLOPs Transformer. FlashAttence and other accurate optimization changes performance/storage, but should not replace mathematical operation.
Independence, zero average and one-size-fits-all is a condition for reasoning, not an automatic fact.
You can't read the answer yet. How does this question go on?
Python examples of Chapter 14
Academic base