03.6 Change passes through chain

Content and key concepts

On the previous page, how the slope itself changes.
If one action depends on the result of the other, the effect is transmitted along the chain. The response chain is multiplied and the different ways are combined.
First u=2x+1, then L=u2; x=2.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Local derivatives are connected by the structure of the calculation
Let
u
=
g
(
x
)
,
z
=
h
(
u
)
.
u=g(x),\qquad z=h(u).
u
=
g
(
x
)
,
z
=
h
(
u
)
.
Small change goes through first.
g
g
g
and then through
h
h
h
:
d
u
≈
g
′
(
x
)
d
x
,
d
z
≈
h
′
(
u
)
d
u
.
du\approx g'(x)dx,
\qquad dz\approx h'(u)du.
d
u
≈
g
′
(
x
)
d
x
,
d
z
≈
h
′
(
u
)
d
u
.
The setup gives chain rule:
d
z
d
x
=
d
z
d
u
d
u
d
x
.
\frac{dz}{dx}=\frac{dz}{du}\frac{du}{dx}.
d
x
d
z
​
=
d
u
d
z
​
d
x
d
u
​
.
For
u
=
2
x
+
1
u=2x+1
u
=
2
x
+
1
,
z
=
u
2
z=u^2
z
=
u
2
to
x
=
2
x=2
x
=
2
:
u
=
5
,
z
=
25
,
d
u
d
x
=
2
,
d
z
d
u
=
2
u
=
10
,
u=5,\qquad z=25,
\qquad \frac{du}{dx}=2,
\qquad \frac{dz}{du}=2u=10,
u
=
5
,
z
=
25
,
d
x
d
u
​
=
2
,
d
u
d
z
​
=
2
u
=
10
,
d
z
d
x
=
10
⋅
2
=
20.
\frac{dz}{dx}=10\cdot2=20.
d
x
d
z
​
=
10
⋅
2
=
20.
The second local multiplier is
2
u
2u
2
u
, not
2
x
2x
2
x
: Derivative of each node is calculated in its own input state.
Reverse-mode carries one upstream response back.
We start with the scalar exit, so...
z
ˉ
=
∂
z
/
∂
z
=
1
\bar z=\partial z/\partial z=1
z
ˉ
=
∂
z
/
∂
z
=
1
Then
u
ˉ
=
z
ˉ
∂
z
∂
u
=
10
,
x
ˉ
=
u
ˉ
∂
u
∂
x
=
20.
\bar u=\bar z\frac{\partial z}{\partial u}=10,
\qquad
\bar x=\bar u\frac{\partial u}{\partial x}=20.
u
ˉ
=
z
ˉ
∂
u
∂
z
​
=
10
,
x
ˉ
=
u
ˉ
∂
x
∂
u
​
=
20.
Forward pass stores
x
,
u
,
z
x,u,z
x
,
u
,
z
; backward applies local rules in reverse topological order. Backprint calculates derivatives in the current graph. It does not change the parameters: extradate performs a separate optimizer.
There are multipliers along the way, between the tracks the sum
Uh-oh.
z
=
x
2
+
x
z=x^2+x
z
=
x
2
+
x
The input of the square branch is equal to the input of a square branch
2
x
2x
2
x
, straight branch 1:
d
z
d
x
=
2
x
+
1.
\frac{dz}{dx}=2x+1.
d
x
d
z
​
=
2
x
+
1.
These deposits add up because the differential sum is equal to the sum of the differential.
2
x
⋅
1
2x\cdot1
2
x
⋅
1
It would be a mistake to connect parallel paths as sequential.
If
u
=
x
2
u=x^2
u
=
x
2
, a
z
=
u
+
u
z=u+u
z
=
u
+
u
one intermediate node is used twice.
x
=
3
x=3
x
=
3
:
u
ˉ
=
1
+
1
=
2
,
x
ˉ
=
u
ˉ
⋅
2
x
=
12.
\bar u=1+1=2,
\qquad \bar x=\bar u\cdot2x=12.
u
ˉ
=
1
+
1
=
2
,
x
ˉ
=
u
ˉ
⋅
2
x
=
12.
Autograd must accumulate a gradient common node. Rewrite the second contribution first or vice versa loses part of the graph.
A numerical check checks the point, not the output.
For
z
(
x
)
=
(
2
x
+
1
)
2
z(x)=(2x+1)^2
z
(
x
)
=
(
2
x
+
1
)
2
Central difference at
x
=
2
x=2
x
=
2
,
h
=
10
−
5
h=10^{-5}
h
=
1
0
−
5
It's about 20.
z
(
x
+
h
)
−
z
(
x
−
h
)
2
h
≈
20.
\frac{z(x+h)-z(x-h)}{2h}\approx20.
2
h
z
(
x
+
h
)
−
z
(
x
−
h
)
​
≈
20.
The coincidence is useful as regression check local backward. It doesn't prove a symbolic formula for everyone.
x
x
x
may miss compensatory errors and require reasonable floating-point tolerance.
For vector nodes, the same rules are recorded via Jacobian products. Reverse-mode does not necessarily build a full Jacobian layer: it transmits upstream vector via local vector-Jacobian product. Therefore, the cost depends on the graph and operations, not on the physical storage of all pairs of derivatives.
x = 2
u = 2*x + 1
z = u*u
dz_du = 2*u
du_dx = 2
dz_dx = dz_du*du_dx
h = 1e-5
numeric = (((2*(x+h)+1)**2) - ((2*(x-h)+1)**2)) / (2*h)
assert (u, z, dz_dx) == (5, 25, 20)
assert abs(numeric - dz_dx) < 1e-8
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
We didn't have to guess the impact of the whole formula: it's made up of the effects of individual actions.
How do you turn the certainty of the forecast into a numerical estimate of error?
Python examples of Chapter 03
Academic base