06.4 How to compare errors of different types

Content and key concepts

On the previous page, familiar dots and new data.
The total number of correct answers does not tell us what mistakes have been made.
The selection of straight lines and the division of points makes it possible to see the learning process on simple data. Verification answers do not participate in the corrections.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
What kind of mistake do we care about?
TP is a positive example that has been found to be positive; FP is a negative, positive; the FN is a missed positive; TN is an accurate negative example; first we capture the protitive class, then we count the metrics.
a
c
c
u
r
a
c
y
=
T
P
+
T
N
N
,
p
r
e
c
i
s
i
o
n
=
T
P
T
P
+
F
P
,
r
e
c
a
l
l
=
T
P
T
P
+
F
N
,
F
P
R
=
F
P
F
P
+
T
N
.
\mathrm{accuracy}=\frac{TP+TN}{N},\quad
\mathrm{precision}=\frac{TP}{TP+FP},\quad
\mathrm{recall}=\frac{TP}{TP+FN},\quad
\mathrm{FPR}=\frac{FP}{FP+TN}.
accuracy
=
N
TP
+
TN
​
,
precision
=
TP
+
FP
TP
​
,
recall
=
TP
+
FN
TP
​
,
FPR
=
FP
+
TN
FP
​
.
For TP=2,FP=1,FN=1,TN=2 accuracy=2/3,precision=recall=2/3,FPR=1/3, if the model does not predict a progitative, denominator precion equals 0. In our lab, the result is "undefined," not arbitrary 100%. Recall remains certain if there are real assumptions.
Experiment.
Increase the threshold on one constant set of scores. The number of protitative preferences does not increase; TP and FP may fall by steps. ROC considers pairs (FPR,TPR) at different thresholds, and one conference matrix corresponds to one. The area below the curve does not replace the choice of the working threshold and the price of errors.
Easy to be wrong.
The high accuracy does not solve the problem itself. Precion also depends on prevalence, which is related to Bayes in Chapter 04.
tp,fp,fn,tn=2,1,1,2
assert (tp+tn)/(tp+fp+fn+tn)==2/3
assert tp/(tp+fp)==2/3 and fp/(fp+tn)==1/3
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
The denominator TP+FP is equal to 0. This shows uncertainty.
Groups without ready correct answers: How does this question continue?
Python examples of Chapter 06
Academic base