09.3 Sensitivity of one action

Content and key concepts

On the previous page: First we get the values.
We'll only take one multiplication or addition, and their local rules will be needed for the whole long chain.
And we're going to save the behavior and we'll go back to the error, and the deposits of the different paths to the total weight add up. It's an automatic differentiation, not a new way to guess the direction.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Small rules instead of one huge formula
For c=a+b, the local derivatives for both operands are equal.1 For c = a, they are equal to b and a. For c is equal to tanh(a) derivative is equal 1 - c2. For ReLU, the aa>0.0 aa<0; t0 is chosen as an agreement, not as a classic derivative.
In L=z*z each of the two entrances mul has a local derivative z=7. These are two ribs to one node; the combined effect.14. If you turn operands into set and transfer the deposit only once, the gradient will be half smaller.
a
ˉ
+
=
c
ˉ
∂
c
∂
a
,
b
ˉ
+
=
c
ˉ
∂
c
∂
b
.
\bar a\mathrel{+}=\bar c\frac{\partial c}{\partial a},\qquad
\bar b\mathrel{+}=\bar c\frac{\partial c}{\partial b}.
a
ˉ
+
=
c
ˉ
∂
a
∂
c
​
,
b
ˉ
+
=
c
ˉ
∂
b
∂
c
​
.
The line represents the derivative selected output L by the node, not its forward value. Incoming gradient scalees the local derivative, then the assembly adds to the already accumulated result.
Experiment.
Select a step mul and compare the operand, local derivative, incoming gradient and contribution. Add local derivative1 does not mean that the gradiant of his parent is mandatory1: the entire chain is taken into account after this node.
Easy to be wrong.
A user-friendly visualiser cannot set up the same derivative1 for all types of operations. Any new primitive requires a local rule and a numerical difference check.
z=7
grad_z=0
grad_z+=1*z
grad_z+=1*z
assert grad_z==14
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
The second multiplier creates local sensitivity.
From one error to all weights, how does this question continue the result?
Python examples of Chapter 09
Academic base