08.6 Direct route through the block

Content and key concepts

On the previous page, the average and scale of one vector.
The input comes with the result of the transformation. This preserves a separate path for both values and derivatives.
One layer creates intermediate signs for the next layer. Non-linearity does not allow the entire sequence to be folded into one table. Direct branches add corrections to the previous state.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Learn the Correction and Maintain the Direct Path
The residual block calculates y=x+F(x). Shapes must coincide; otherwise, a clear projection is required, not a random Broadcast. F=0 results in identity. This is a property of design, not an assertion that the initial F is necessarily small.
J
y
=
I
+
J
F
,
∇
x
L
=
(
I
+
J
F
)
T
∇
y
L
.
J_y=I+J_F,\qquad \nabla_xL=(I+J_F)^T\nabla_yL.
J
y
​
=
I
+
J
F
​
,
∇
x
​
L
=
(
I
+
J
F
​
)
T
∇
y
​
L
.
Let F(x)=g·ReLU(x), x=[-1.2], g=.5. Then F=[0.1], y=[ -1.3], Jacobian diag(1.1.5). Without skip output[.1.1], Jacobian diago(0.5): the negative coordinate has lost direct dependence.
Experiment.
Turn off the skip at the previous F. Then put g= -1: on the positive 1+F\=0. So residual does not guarantee the non-zero gradient at any weights - deposits can be compensated. "One path" does not mean that the total derivative is always >1.
Identity scip does not add weights. Addition costs one operation on a component, and preserving activity is important for turing memory. The next chapter opens an algorithm that summarizes the deposits of both branches:
Backup and Mini-autograde
.
x,g=[-1,2],.5
F=[g*max(0,v) for v in x]
y=[a+b for a,b in zip(x,F)]
assert F==[0,1] and y==[-1,3]
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Both branches participate in the derivative and can be compensated.
How do we find the impact of all its parameters on one common mistake?
Python examples of Chapter 08
Academic base