05.3 What examples were involved in the correction

Content and key concepts

On the previous page: A small step against the increase in error.
The error and gradient depend on the data selected. Some examples can give a different direction than the whole set.
Individual derivatives: [1 - 1 - 3 - 5].
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Not every gradient requires all data.
If L(-)=N-1-Li(-), the complete g=N-1-xLi. The Mini-batch selects the subset of B and uses its average gradient. If an even random sample is the appropriate way, this is a non-replaceable estimate of the full gradient. The noise depends on the sample and the structure of the data, is not added by arbitrary decorative shaking of the line.
g
B
=
1
∣
B
∣
∑
i
∈
B
∇
L
i
(
θ
)
.
g_B=\frac1{|B|}\sum_{i\in B}\nabla L_i(\theta).
g
B
​
=
∣
B
∣
1
​
i
∈
B
∑
​
∇
L
i
​
(
θ
)
.
For Li(-)=5(-ti)2 and Targets[-1.1.3.5], at *=0 individual gradiants[1-1-1-3-3-5], the full average is -2. Batch from the first and second examples gives 0, although the total g-0.0.Batch of all four returns -2.
Experiment.
Change the size of batch1/2/4 with fixed seed. The panel shows the selected indices and their gradients. Seed changes choice, not data. The complete barch should not be dependent on reset. Fixed seed provides reproducible experience, but does not in itself prove the statistical properties of incoherence.
Race laboratory used
Exact
The gradient of the analytical surface, so the first method is called GD, not SGD. This separate batch lab really chooses examples. You can't give the first demonstration for stochastic learning just because the algorithms apply to SGDs.
Under the hood.
The big batch often uses matrix computations better, but requires more memory of activations. The noise reduction, transport quality and speed per second depend on the task; the equality "batch is always better" does not follow the middle formula.
targets = [-1,1,3,5]
theta = 0
gradients = [theta-t for t in targets]
assert sum(gradients)/4 == -2
assert sum(gradients[:2])/2 == 0
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Gradient is calculated from the data, not by random noise, so different parts of the data can give different steps.
How does this matter continue?
Python examples of Chapter 05
Academic base