17.3 How one option is selected

Content and key concepts

On the previous page, the next launch uses the saved.
The ratings have already been received, and the temperature and limitations of the candidates now determine the selection rule; the weight of the candidate is not taught.
The first start-up process the prefix. Each subsequent one adds one entry; K/V past entries can be saved. Sample and compressed storage solve different tasks.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Logits sets model, token selects decoder
Greedy decoding chooses argmax; sampling first converts logits workshop, then can limit candidates top-k or nucleus top-p.
T>0
used
softmax(z/T)
:Little T concentrates distribution, big makes it more flat.
T=0
It's better to ask for a separate greedy mode than to divide it by zero.
Nucleus selects a minimum prefix of candidates that are probably graded with a cumulative mass at least
p
F10.
S
p
=
min
⁡
{
S
:
∑
j
∈
S
p
j
≥
p
}
,
p
~
j
=
p
j
∑
k
∈
S
p
p
k
1
[
j
∈
S
p
]
.
S_p=\min\left\{S:\sum_{j\in S}p_j\ge p\right\},\qquad
\tilde p_j=\frac{p_j}{\sum_{k\in S_p}p_k}\mathbf1[j\in S_p].
S
p
​
=
min
⎩
⎨
⎧
​
S
:
j
∈
S
∑
​
p
j
​
≥
p
⎭
⎬
⎫
​
,
p
~
​
j
​
=
∑
k
∈
S
p
​
​
p
k
​
p
j
​
​
1
[
j
∈
S
p
​
]
.
That's what the model shows.
Inspector is required to show the original logs, test discrubation, remaining candidates, and remanormalized probabilities. Seed refers to peudo-random draw, not model distribution. For the same webs/prefix/policy/seated deterministic ruleime, the IDs must repeat, but this test is reproducibility, not quality.
Scope of applicability.
Top-p does not verify facts and security. The application of re-entry design, top-k, top-p and construants affects the result and is part of the API contract. Floating-point ties require stable tee-break. Beam search optimizes its own sequence score and also does not guarantee a good human response.
from math import exp
logits_17c=[2.0,1.0,0.0]; temperature_17c=1.0; top_p_17c=0.8
raw_17c=[exp(v/temperature_17c-max(logits_17c)) for v in logits_17c]
probs_17c=[v/sum(raw_17c) for v in raw_17c]; order_17c=sorted(range(3),key=lambda i:(-probs_17c[i],i))
kept_17c=[]; mass_17c=0.0
for i_17c in order_17c:
kept_17c.append(i_17c); mass_17c+=probs_17c[i_17c]
if mass_17c>=top_p_17c: break
assert kept_17c==[0,1] and abs(sum(probs_17c[i] for i in kept_17c)-mass_17c)<1e-12
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Seed controls a reproducible sample, not a model forward.
Several queries use common computing. How does this question continue?
Python examples of chapter 17
Academic base