09.4 From one error to all weights

Content and key concepts

On the previous page: Sensitivity of one action.
We start with the effect of the error on ourselves: it's equal to one, then we pass the sensitivity back.
And we're going to save the behavior and we'll go back to the error, and the deposits of the different paths to the total weight add up. It's an automatic differentiation, not a new way to guess the direction.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
From one output to many inputs
We start with dL/dL=1, the rest of the gradients are off. It's going backwards to the topological order. For scalar L, it's how you get the derivatives for many parameters by one inverse passage.
Processed Node
Incoming gradient
Contributions
L=z*z
1
z:7+7=14
z=a+b
14
a:14,b:14
a=x*w
14
x:14×3=42,w:14×2=28
An independent symbolic calculation of L=(xw+b)2 gives \\L/\x=2(xW+b)(w)=42, \L /\\\w=2[xw +b)x=28, £L/ \ \~b=2 (xw plus b)=14.
Experiment.
Stop before processing a: rad(a) is already ready, grad(x) has not yet received the contribution through a. One next step will convey this sensitivity. The "back" button restores snapshot rather than does the reverse optimizer.
Boundary.
Reverse-mode does not change function: it does not restore input on output. The Derivative can also be calculated for irreversible changes. After backward, the reference w=3,b=1 has not yet been updated. Learning requires a separate subdate using the gradiants of the selected parameters simultaneously.
x,w,b=2,3,1
z=x*w+b
assert [2*z*w,2*z*x,2*z]==[42,28,14]
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
The identity function by its own output has a derivative1.
One weight is used several times. How does this question continue?
Python examples of Chapter 09
Academic base