08.3 What is stored and what is calculated

Content and key concepts

On the previous page, the width and depth of the network.
Weights are settings; intermediate numbers depend on entry, gradients depend on error; let's not mix these arrays.
One layer creates intermediate signs for the next layer. Non-linearity does not allow the entire sequence to be folded into one table. Direct branches add corrections to the previous state.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Weights are not multiplied by the size of the batch
The layer contains n_in·n_out balance and n_out bises. The number of activations after it is B·n _out, but this is another object. The Gradient in the balance matrix has the same shape as the matrix itself.
P
=
∑
ℓ
=
1
L
(
d
ℓ
−
1
d
ℓ
+
d
ℓ
)
,
shape
⁡
(
∇
W
ℓ
)
=
shape
⁡
(
W
ℓ
)
.
P=\sum_{\ell=1}^{L}(d_{\ell-1}d_\ell+d_\ell),\qquad
\operatorname{shape}(\nabla W_\ell)=\operatorname{shape}(W_\ell).
P
=
ℓ
=
1
∑
L
​
(
d
ℓ
−
1
​
d
ℓ
​
+
d
ℓ
​
)
,
shape
(
∇
W
ℓ
​
)
=
shape
(
W
ℓ
​
)
.
For a network of 2x3x2x1 you get 9.8.3 parameters by layer, so 20. For batch4, layer exits contain 12.8.4 numbers. Fixed element activation does not add learning parameters itself.
Under the hood.
Twenty float32 scalars are 80 bytes without overhead. Gradient adds another 20 numbers; Adam usually stores two moment states for each parameter. In addition, actions and buffers are needed. In JavaScript Noumber Binary64, 20×8=160 bytes are just idealized numeric payloads, not sheap measurements.
Debug.
For X[B,d] and W[h,d][direct XW is not consistent. Transponder or other declared agreement is needed. Accidental reshup does not correct semantics and can only hide an error. Inspector contains all matrices, Shapes and active accounts calculated by arrays.
dims=[2,3,2,1]
counts=[a*b+b for a,b in zip(dims,dims[1:])]
assert counts==[9,8,3] and sum(counts)==20
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Each scalar parameter has its own sensitivity.
Where does the training start? How does this question continue?
Python examples of Chapter 08
Academic base