07.4. Non-linear conversion rules

Content and key concepts

On the previous page, smooth function and saturation.
ReLU, GELU, and SiLU are different ways to convert the sum of the neuron.
He collects inputs, adds a shift, and applies a non-linear rule. All parts of this calculation are already available in school arithmetic.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Different negative branches
ReLU(z)=max(0,z). The output is 0 at z<0 and 1 at z>0. No classical derivative is available at zero; runtime chooses 0 as a clear agreement. The threshold perceptron has another derivative: 0 outside the threshold and uncertainty on it.
SiLU(z)=z\(z). According to the rule of the work:
SiLU
⁡
′
(
z
)
=
σ
(
z
)
+
z
σ
(
z
)
(
1
−
σ
(
z
)
)
.
\operatorname{SiLU}'(z)=\sigma(z)+z\sigma(z)(1-\sigma(z)).
SiLU
′
(
z
)
=
σ
(
z
)
+
z
σ
(
z
)
(
1
−
σ
(
z
))
.
At z=0 result 0, derived from .5; at z= -1 a value of approximately 268941. Small negative exits are allowed.
The exact GELU is z(z) where the standard Gaussian CDF is used.
tanh-approximation
and not the exact CDF (N05.N08):
u
=
2
/
π
(
z
+
0.044715
z
3
)
,
g
(
z
)
=
1
2
z
(
1
+
tanh
⁡
u
)
.
u=\sqrt{2/\pi}(z+0.044715z^3),\qquad g(z)=\tfrac12z(1+\tanh u).
u
=
2/
π
​
(
z
+
0.044715
z
3
)
,
g
(
z
)
=
2
1
​
z
(
1
+
tanh
u
)
.
Its derivative is: .5(1+tanh u)+.5z(1-tanh2u) G(2/Pi)(1)+3· 044715z2). It corresponds specifically to the selected approximation.
Research.
Compare the functions at -1.0.1 and explain the differences in derivatives. The format of the graph does not give a universal rating: utility depends on architecture, data, and learning. Switching is changing the family of functions, not just the design of the line.
from math import tanh,sqrt,pi
z=0
u=sqrt(2/pi)*(z+.044715*z**3)
g=.5*z*(1+tanh(u))
gp=.5*(1+tanh(u))+.5*z*(1-tanh(u)**2)*sqrt(2/pi)*(1+3*.044715*z*z)
assert g==0 and gp==.5
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
The formula and derivative are one of the selected approximations.
How does this matter continue?
Python examples of Chapter 07
Academic base