04.5 How to Fine Wrong Confidence

Content and key concepts

On the previous page, how much uncertainty remains.
If the model gave the right result, a very small percentage, the fine should be bigger, we'll take the average fine and we're different from our own uncertainty.
The model gives each of the four candidates a 1/4 probability.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Surprising under the wrong model
Let the true/target distribution p, and model q projection by
the same
The average fine is ln qi weighted by the frequency p, not q:
H
(
p
,
q
)
=
−
∑
i
p
i
ln
⁡
q
i
,
D
K
L
(
p
∥
q
)
=
∑
i
p
i
ln
⁡
p
i
q
i
=
H
(
p
,
q
)
−
H
(
p
)
.
H(p,q)=-\sum_i p_i\ln q_i,\qquad
D_{KL}(p\Vert q)=\sum_i p_i\ln\frac{p_i}{q_i}=H(p,q)-H(p).
H
(
p
,
q
)
=
−
i
∑
​
p
i
​
ln
q
i
​
,
D
K
L
​
(
p
∥
q
)
=
i
∑
​
p
i
​
ln
q
i
​
p
i
​
​
=
H
(
p
,
q
)
−
H
(
p
)
.
For fixed p, the minimum CE is q equivalent to the minimum KL. For normalized discrete distributions KL≥0, the equation at p=q. The non-negative is derived from ln t≤t-1; it cannot be "maintained" with an erroneous result module.
For p=[5,.25,.25],q=[25,5,.25]:H(p)=1.09720771,EC=1.2307566,KL=173286795=ln2/4. If q=p, CE is not required to be 0: it is equal to entropy Target. For one-hot Target category j entropia0 and CE=-ln qj.
Boundary.
If pi>0,qi=0, CE and KL are endless: the model has banned a possible target result. This is a meaningful mathematical result, not NaN. If Pi=0, input0 regardless of qi.
Research.
At constant p, gradually reduce q of one frequent category to 0. The fine grows unlimited. Then change p and q in places: KL is not at all symmetrical and is not a metric of distance. In particular, for [1.0] and [5.5,5] one direction is ln2, the reverse is infinite.
from math import log
p,q = [.5,.25,.25],[.25,.5,.25]
ce = -sum(a*log(b) for a,b in zip(p,q) if a>0)
kl = sum(a*log(a/b) for a,b in zip(p,q) if a>0)
assert abs(ce-kl-1.5*log(2))<1e-12
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
First the error of the correct answer is measured. The exhibitor translates the average error to another scale; the quality of the text is not exhausted.
Error on the scale of the number of options. How does this question continue the result?
Python examples of Chapter 04
Academic base