04.6 Error in the number of options

Content and key concepts

On the previous page, how to fine wrong confidence.
The average error gives you an indicator similar to the number of equal options; this is a convenient comparison of models, but not an assessment of the truth of their response.
The model gives each of the four candidates a 1/4 probability.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Efficient scale of average log-loss
On a fixed sequence of lengthN, the model evaluates the probability
the following current observed
The average negative log-probability of the insides is given by the CE_token.
C
E
t
o
k
e
n
=
−
1
N
∑
t
=
1
N
ln
⁡
q
(
t
t
∣
t
<
t
)
,
P
P
=
exp
⁡
(
C
E
t
o
k
e
n
)
.
\mathrm{CE}_{token}=-\frac1N\sum_{t=1}^{N}\ln q(t_t\mid t_{<t}),\qquad
PP=\exp(\mathrm{CE}_{token}).
CE
t
o
k
e
n
​
=
−
N
1
​
t
=
1
∑
N
​
ln
q
(
t
t
​
∣
t
<
t
​
)
,
PP
=
exp
(
CE
t
o
k
e
n
​
)
.
This is the opposite geometric mean probability of the right currents. If for two observations they are equal to 5 and 25, CE=-(ln.5+ln.25)/2=1.039720771 and PP=G8E28288427. If the model is equal to the x results, PP=k. For log2, instead of the logn is right PP=2\CE_bits, not exp(CE_bits).
The laboratory shows the exp(H(p,q)) for the introduced categorical Target p: this expected perplexity relative to this p. It becomes an evaluation of the hull if p corresponds to the empirical frequency of the events to be used with an agreed context evaluation. It is not possible to give this three-catagorial experience as a benchmark of this LLM.
Easy to be wrong.
Comparison of perplexity models with different tokenizers, hulls or averaging rules is not directly correct. A low PP does not mean that the answer, safe behaviour, or ability to use tools is true. On a small learning sample, PP may decrease when transport is deteriorating.
Synthesis.
We've got a target function, disferentiable on logs: softmax plus CE for one-hot Target has a derivative of q-p. The output is released via a derivative logsumexp: xlog\exp(z)/xj=qj, then subtracts the Target contribution. Now both the error and its sensitivity are known. It was left to choose the update procedure:
Optimization
.
from math import log, exp, sqrt
observed = [.5,.25]
ce = -sum(log(p) for p in observed)/len(observed)
assert abs(exp(ce)-sqrt(8))<1e-12
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
First the error of the correct answer is measured. The exhibitor translates the average error to another scale; the quality of the text is not exhausted.
Now the error and its derivatives are known.
Python examples of Chapter 04
Academic base