17.2. Next start-up uses a saved

Content and key concepts

On the previous page, the first launch processes the input phrase.
Past keys and values have already been calculated. For a new current, add new numbers and compare the result with a full recalculation.
The first start-up process the prefix. Each subsequent one adds one entry; K/V past entries can be saved. Sample and compressed storage solve different tasks.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
We don't count keys and values of the past.
After prefill, each layer retains the key/value testors of the past tokens. For the new token, only new Q/K/V are listed; query compares with all cachaed keys and the new K/V is signed. Without cache decoder repeats the projections of the former prefix at every step. Cache is an activation specific to the test, not a model parameters.
For
L
layers,
H_kv
KV-heads, head width
d_h
,
T
tokens and
b
Bytes on scalar rough Payload:
M
K
V
=
2
L
H
k
v
d
h
T
b
.
M_{KV}=2LH_{kv}d_hTb.
M
K
V
​
=
2
L
H
k
v
​
d
h
​
T
b
.
The multiplier 2 corresponds to K and V. Batch requests, beam copis, allocator blocks, alignment and runtime workspace are added from above. Multi-query/grouped-querie engagement reduces the number of KV-heads, but quaries can still have more heads. F08, F09.
That's what the model shows.
The visual model shows an append-only logical reference and separately a phisical lock. PagedAttention is motivated by a variable of lengths and fragmentation: the logical neighbourhood does not require one large continuous allocation. Sharing prefix is allowed only with identical model inputs and correct lifetime control.
Scope of applicability.
The Payload formula is not a promise of actual VRAM. Quantified cache can reduce bytes, but adds scales/kerrels and error. The Operation or sliding Windows changes the available content. You cannot re-use cache after changes to the webs, adapter, positions or parts of prefix simply because shape coincides.
layers_17b,kv_heads_17b,head_dim_17b,tokens_17b,bytes_17b=2,1,4,3,2
cache_bytes_17b=2*layers_17b*kv_heads_17b*head_dim_17b*tokens_17b*bytes_17b
assert cache_bytes_17b==96
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Cache re-uses causal K/V and grows at every token processed.
How is one option chosen? How does this question continue?
Python examples of chapter 17
Academic base