10.6 Calculations and transports limit speed

Content and key concepts

On the previous page, why is it important to reuse data?
Sometimes there is a lack of arithmetical capacity, sometimes memory speeds, and the evaluation of these boundaries and capacity estimates answer different questions.
The numbers have final accuracy; they have to be stored and moved. The matrix blocks produce a lot of independent work, but memory and group organization also affect the results.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Upper limit thrughput, lower time estimate
LetF - FLOPs Tasks, Q - Traffic B - Bandwidth Intakes/s, P_peak - FLOP/s. Aristmetic Intensity I=F/Q. Simplified roofline model:
P
≤
min
⁡
(
P
p
e
a
k
,
B
I
)
,
T
≥
max
⁡
(
F
/
P
p
e
a
k
,
Q
/
B
)
.
P\le\min(P_{peak},BI),\qquad T\ge\max(F/P_{peak},Q/B).
P
≤
min
(
P
p
e
ak
​
,
B
I
)
,
T
≥
max
(
F
/
P
p
e
ak
​
,
Q
/
B
)
.
It's an optimistic estimate of the assumptions selected,
not benchmark
For F=2e9,Q=1e9,B=100e9,P_peak=10e12, I=2, memory limit200GFLOP/s and T≥.01c.Compute lower time=0002c, less memorry time.Presets in the lab is conditional, not the results of the equipment found.
Wights Pparameter b_wbyte are taken by Pb_w without metadata. KV-cache for decoder in the selected model:
M
K
V
=
2
L
B
S
H
K
V
d
h
b
K
V
.
M_{KV}=2LBSH_{KV}d_hb_{KV}.
M
K
V
​
=
2
L
BS
H
K
V
​
d
h
​
b
K
V
​
.
Two means KiV, Lslouch, Bbatch, Stocken, H_KV numeral KVheads,d_h sizehead.
In the training evaluation of this laboratory, FP32 gradiants, 2FP32Adam moments and FP 32master webs are added; KV-cache is not used. Activations/workspace are given by a separate budget, rather than accurately calculated by billions of parameters. fractation, rentimontext and quantization metadata are not included. Therefore, "a sum less than VRAM" means only passing this assessment, not a guarantee of launch.
Now it's clear why local re-use operations are important.
by signals and folding
.
kv=2*32*1*2048*8*128*2
assert kv==268435456 and kv/2**20==256
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
The calculator includes only signed components, not the whole ringtime.
How do the same local amounts process the image or signal?
Python examples of Chapter 10
Academic base