08.5 Average and scale of one vector

Content and key concepts

On the previous page, where the training begins.
We subtract the average and divide the coordinates into the total scale, then compare the rule that takes into account the value without subtracting the average.
One layer creates intermediate signs for the next layer. Non-linearity does not allow the entire sequence to be folded into one table. Direct branches add corrections to the previous state.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Statistics of one example
For a vector x of d, the LayerNorm component uses its mean and population representation.
μ
=
1
d
∑
i
x
i
,
v
=
1
d
∑
i
(
x
i
−
μ
)
2
,
y
i
=
x
i
−
μ
v
+
ϵ
.
\mu=\frac1d\sum_i x_i,\quad v=\frac1d\sum_i(x_i-\mu)^2,\quad
y_i=\frac{x_i-\mu}{\sqrt{v+\epsilon}}.
μ
=
d
1
​
i
∑
​
x
i
​
,
v
=
d
1
​
i
∑
​
(
x
i
​
−
μ
)
2
,
y
i
​
=
v
+
ϵ
​
x
i
​
−
μ
​
.
For [1,2.3] the average, the result is not required to be exactly 1. N06 confirms the selection of the statistical axes; the numbers are independently selected.
RMS-normalization does not subtract mean: y=x/G(mean(x2)+\). For the same input of the second moment14/3 constant [2.2.2] after LN zero, after RMS almost [1.1.1].
Experiment.
Add the same offset to all components: LN will continue, RMS will change in general. Cancel the entrance: epsilon prevents division by 0. This separate model does not add hidden normalitation to XOR trainer.
BatchNorm uses a different axis of statistics and can have running estimats and differences train/eval. It is not possible to transfer these properties to LN. In the present affine LayerNorme, the educated β add parameters; here, their fixedness is signed.
from math import sqrt
x=[1,2,3]
mu=sum(x)/3
variance=sum((v-mu)**2 for v in x)/3
y=[(v-mu)/sqrt(variance+1e-5) for v in x]
assert mu==2 and variance==2/3 and abs(sum(y))<1e-12
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Subtracting mean snatches the numerator, epsilon keeps the denominator positive.
How does this question go about the result?
Python examples of Chapter 08
Academic base