15.3. Position through the turn of coordinates

Content and key concepts

On the previous page, why is the order of words relevant?
You can enter order to rotate the request pairs and the key. The length is retained and the comparison depends on the relative position.
The vector obtains positional information, exchanges information and updates its status separately. Residual amounts maintain the same path. The last table gives estimates of the dictionary.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
From absolute angles to relative differences
RoPE uses rotations to Q and K coordinate pairs. For vapour[a,b] in positionp and frequency :
R
(
p
ω
)
(
a
b
)
=
(
a
cos
⁡
(
p
ω
)
−
b
sin
⁡
(
p
ω
)
a
sin
⁡
(
p
ω
)
+
b
cos
⁡
(
p
ω
)
)
.
R(p\omega)\begin{pmatrix}a\\b\end{pmatrix}=
\begin{pmatrix}a\cos(p\omega)-b\sin(p\omega)\\a\sin(p\omega)+b\cos(p\omega)\end{pmatrix}.
R
(
p
ω
)
(
a
b
​
)
=
(
a
cos
(
p
ω
)
−
b
sin
(
p
ω
)
a
sin
(
p
ω
)
+
b
cos
(
p
ω
)
​
)
.
The rotation retains the L2-normal. Because R(m)TR(n)=R(n-m), dot rotated query/key depends on the difference of positions in fixed reference vectors.
For q=k=[1.0], ×=1, entries 0 and pi/2 as mathematical angles give orthogonal directions and dot0. In the real position code p whole and anglep , this is not an indication to use a fractional token index.
For each head width d_h pairs are given frequencies 10,000 (-2j/d_h). In this implementation, d_H should be even. V does not rotate; RoPE does not add a position vector to embedding. The switch between sinusoidal and RoPE changes the way the position is introduced, rather than both.
Experiment.
Move the query/key entries to the same number and compare dot with constant reference vectors. Do not conclude that a monotonic reduction of score with distance is achieved by rotating the coefficient components. Supporting the large position indications does not prove the quality of the long context.
from math import sin,cos,pi
a,b=3,4; angle=pi/2
rot=[a*cos(angle)-b*sin(angle),a*sin(angle)+b*cos(angle)]
assert abs(rot[0]+4)<1e-12 and abs(rot[1]-3)<1e-12
assert abs(sum(v*v for v in rot)-25)<1e-12
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
The relative dot is taken from R_mTR_n=R_(n-m).
How does this question continue?
Python examples of Chapter 15
Academic base