08.4 Where the education begins

Content and key concepts

On the previous page, what is stored and what is calculated.
Before the first correction, weight values are also needed. Same neurons can get the same corrections, so initial differences are important.
One layer creates intermediate signs for the next layer. Non-linearity does not allow the entire sequence to be folded into one table. Direct branches add corrections to the previous state.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Why should hidden neurons differ?
In the same scales, hidden units can receive the same entrances and gradients, then be updated equally. The network is not required to use width as independent features. Zero initialization illustrates this symmetry.
In close independence and centralization:
Var
⁡
(
∑
i
w
i
x
i
)
≈
f
a
n
_
i
n
Var
⁡
(
w
)
Var
⁡
(
x
)
.
\operatorname{Var}\left(\sum_iw_ix_i\right)\approx
\mathrm{fan\_in}\,\operatorname{Var}(w)\operatorname{Var}(x).
Var
(
i
∑
​
w
i
​
x
i
​
)
≈
fan_in
Var
(
w
)
Var
(
x
)
.
This motivates the magnitude of the order1/Gfan_in. Non-linearity and dependence complicates the picture; the formula is not the exact law for any network.
In the Webs lab, it's even on [-scale/Gfan_in,scale/Gfan_in], biases0.
not fully implemented by Xavier or He
. Scale0.5 affects the actual effects; mean and RMS shoulden activities are measured after the forward.
Experiment.
Compare scale0,1.1.5. When scalet0 hasn't received a useful signal, although output bias can learn medium target. With a large scalé tanh is satisfied, and ReLU can give high values or inactive branches. Falling loss does not prove that all hidden units have become independent.
Reset restores the parameters of experience, initialization, and history. This is a testable reproducibility, not a promise of the same trajectory of any library and GPU.
Accidental initialization helps to disrupt symmetry.
The average and scale of one vector. How does this question continue?
Python examples of Chapter 08
Academic base