04.4 How much uncertainty remains

Content and key concepts

On the previous page, which settings better explain observations.
The predicted outcome is hardly surprising, the rare one is more surprising, and the entropy measures the average surprise of distribution.
Three outfits with fractions [0.5.0.0.25].
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Why the logarithm?
For independent events, the probability is multiplied. If the information is to be compiled, it is a logarithm. The sign minus makes the rare event more unexpected: I(i)=-ln pi. The expectation is Shannon enropy. Sources M04 §3.13 and M07; the original work of Shannon is not stated here.
H
(
p
)
=
−
∑
i
p
i
ln
⁡
p
i
.
H(p)=-\sum_i p_i\ln p_i.
H
(
p
)
=
−
i
∑
​
p
i
​
ln
p
i
​
.
By default natural logarithms, units
Total cost
. Log2 gives bits: H_bits=H_nats/ln2. For p=[5,.25,25] the deposits of .346574, 346574,.346574; the sum of 1.039720771. Battered is 1.5. The zero mass gives a zero contribution by agreement lim(t lnt) = 0 at t ± 0+, not an attempt to compute 0~(-) normal multiplication.
For k the final results 0 ≤H≤ln k. The lower boundary is achieved by concentrating the mass on one result; the upper one with even distribution. For three results, maximum ln3E1.098612. The technically upper boundary is also derived from KL(p
Experiment.
Change p, leaving q untouched. The entropy p should depend only on p. Then change the parts of the snails: the form of distribution does not change, the number is scaled.
Easy to be wrong.
The entropy is not "quantity of meaning", it is not the moral value of the message or the quality of the classification. The error model can have a low entropathy of its q. To evaluate it relative to the correct results, it is necessary to compare q with the Target distribution p.
from math import log
p = [.5,.25,.25]
entropy = -sum(v*log(v) for v in p if v>0)
assert abs(entropy-1.5*log(2))<1e-12
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
The less known about the outcome, the greater the average surprise, it's a pattern of distribution, not an assessment of the truth of the answer.
How do you fine the wrong confidence, how does this question continue?
Python examples of Chapter 04
Academic base