10.5 Why Reuse Data Is Important

Content and key concepts

On the previous page, how one cell of a work is calculated.
The value is not only to be calculated, but also to be read from memory. Reuse of small blocks reduces unnecessary reading.
The numbers have final accuracy; they have to be stored and moved. The matrix blocks produce a lot of independent work, but memory and group organization also affect the results.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Arifmetics needs data.
Weights can be in performance memory, then enter the cache or the welled memory and registers. Specific levels depend on the device. Capability, bathwidth and latancy are different values: large memory is not necessarily faster and high capacity does not eliminate the start delay.
For C=AB, if you read each input once and write out one time, minimum model traffic:
Q
m
i
n
=
(
m
k
+
k
n
+
m
n
)
b
,
Q_{min}=(mk+kn+mn)b,
Q
min
​
=
(
mk
+
kn
+
mn
)
b
,
whereb is a byte on a scalar. The oldCne reads: β=0, not general GEMMsβC. The model gives:
Q
t
i
l
e
=
b
(
m
k
⌈
n
/
t
⌉
+
k
n
⌈
m
/
t
⌉
+
m
n
)
.
Q_{tile}=b\left(mk\lceil n/t\rceil+kn\lceil m/t\rceil+mn\right).
Q
t
i
l
e
​
=
b
(
mk
⌈
n
/
t
⌉
+
kn
⌈
m
/
t
⌉
+
mn
)
.
For 2×2 matrices andb=2 we get a minimum of 24 bytes.
Experiment.
In the evaluation of the big GEMM, you can zoom in the same, k,n. Compare idial and modeled trade, not replace one other. The formula does not account for downloading addresses, cache display, synchronizing and double buffering.
Under the hood.
Copying the websCPUNAGPU is another transfer not included in the development-memory trade. KV-cache, activations and optimizer states should also not be hidden in the balance memory line.
m=k=n=2; b=2
qmin=(m*k+k*n+m*n)*b
qtile1=(m*k*n+k*n*m+m*n)*b
assert qmin==24 and qtile1==40
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Reuse reduces re-reading without changing the set operator.
Calculations and transports limit speed. How does this question continue?
Python examples of Chapter 10
Academic base