17.1. The first launch handles the input phrase

Content and key concepts

The weights have already been picked up; now they're fixed.
The first choice is to calculate the performances of all known currents. This launch is called pre-filling: prefill.
The first start-up process the prefix. Each subsequent one adds one entry; K/V past entries can be saved. Sample and compressed storage solve different tasks.
Use fixed weights
Общая модель требуетJavaScript; её вычисления, формулы и код разобраны в открытом тексте далее.
One forward creates the state of all entries
Inference decoder consists of two modes.
i
uses keys
0..i
For length
n
exists
n(n+1)/2
This is not a complete FLOP account: projects, FFN, normalization and memorry trade.
N
s
c
o
r
e
s
p
r
e
f
i
l
l
=
∑
i
=
1
n
i
=
n
(
n
+
1
)
2
,
N
s
c
o
r
e
s
n
e
x
t
=
n
+
1.
N_{scores}^{prefill}=\sum_{i=1}^{n}i=\frac{n(n+1)}2,
\qquad N_{scores}^{next}=n+1.
N
scores
p
re
f
i
ll
​
=
i
=
1
∑
n
​
i
=
2
n
(
n
+
1
)
​
,
N
scores
n
e
x
t
​
=
n
+
1.
Prefill usually fills the accelerator better with the large matriix multiplications, while the decode adds one entry and is more often limited to reading the weights/cache. FlashAttence changes the IO-plan for exact application and is not obliged to materialize the entire score matrix in HBM; the mathematical result remains the same with the same accuracy.
That's what the model shows.
The prefill matrix reveals the lower triangle, then the decode adds one new quary string. Propt tokens are not re-generated: the model calculates their hidden states and cache. The prefill metrics are convenient to express TTFT and input token/s, and the decode is time per output tokins/s.
Scope of applicability.
The pairs formula does not take into account grouped/multi-query engagement, padding, packed responses, kernels and parallelism. A longer context increases not only arithmetic, but also cache/transfer priority. The title "prefill parallel" does not mean zero relationship between layers or infinite acceleration.
prompt_len_17a=4
prefill_pairs_17a=sum(range(1,prompt_len_17a+1))
next_decode_pairs_17a=prompt_len_17a+1
assert prefill_pairs_17a==10 and next_decode_pairs_17a==5
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Prefill creates hidden states and K/V for all prompt positions with fixed webs.
The next launch uses the saved. How does this question continue with the result?
Python examples of chapter 17
Academic base