07.3. Clean function and saturation

Content and key concepts

On the previous page, the boundary between decisions.
The function can compress a wide range of entrances to a limited range of exits. But at the edges, a slight change in the input almost ceases to affect the response.
He collects inputs, adds a shift, and applies a non-linear rule. All parts of this calculation are already available in school arithmetic.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
The smoothness does not preclude saturation
Sigmoid translates z to (0.1), tanh to (-1.1.
σ
(
z
)
=
1
1
+
e
−
z
,
σ
′
(
z
)
=
σ
(
z
)
(
1
−
σ
(
z
)
)
,
tanh
⁡
′
(
z
)
=
1
−
tanh
⁡
2
(
z
)
.
\sigma(z)=\frac1{1+e^{-z}},\quad \sigma'(z)=\sigma(z)(1-\sigma(z)),\quad
\tanh'(z)=1-\tanh^2(z).
σ
(
z
)
=
1
+
e
−
z
1
​
,
σ
′
(
z
)
=
σ
(
z
)
(
1
−
σ
(
z
))
,
tanh
′
(
z
)
=
1
−
tanh
2
(
z
)
.
At z=0 the sigmoid value and its derivative are equal to .5 and .25; for tanh this is 0 and 1. For large
Experiment.
Compare the activations at the same x,w,b. Increase the z module through the weight: Shape does not change, but sensoryity can decrease dramatically. The graph shows separately the function value and its derivative.
For stable sigmoid at z >0 use 1/(1+exp(-z)) and at z<0 is exp(z)/(1 +exp(z). This does not require a high positive number exponent. With high saturation, floating-point can still round p to 0 or 1; BCE is therefore better to compute through logit.
Boundary.
Sigmoid as a hidden activation and a sygmoid+BCE bundle at the exit have different derivatives relative to a common error. Surely the wrong log does not have to get almost zero gradient combined loss. Tanh values are not the probability distribution.
from math import exp,tanh
s=1/(1+exp(0))
assert s==.5 and s*(1-s)==.25
assert tanh(0)==0 and 1-tanh(0)**2==1
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
It's not full derivative any loss.
Non-linear rules of transformation. How does this question continue?
Python examples of Chapter 07
Academic base