05.4 Take into account past corrections

Content and key concepts

On the previous page, what examples were involved in the correction.
The history of change helps not to respond only to the latter example, but to separate the accumulation of direction and scale.
New gradient4; accumulated history so far0; share of history0.9.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Coding smoothing and scaling
Momentum stores the history of gradients. Here we use the EMA-option mk=βmk-1+(1-β)gk, extradate
At g1=[4.6] the first EMA is equal[4,6] and the step with the steps is equal to [2.1] in [1.96,.94]. History accumulates between steps but reset deflates it. Momentum can smooth the variable elements and reinforce the steady direction; it is also capable of flying the minimum.
RMSProp stores the second moment vk=βvk-1+(1-β)gk2, then scales each coordinate:
θ
k
+
1
,
j
=
θ
k
,
j
−
η
g
k
,
j
v
k
,
j
+
ϵ
.
\theta_{k+1,j}=\theta_{k,j}-\eta\frac{g_{k,j}}{\sqrt{v_{k,j}}+\epsilon}.
θ
k
+
1
,
j
​
=
θ
k
,
j
​
−
η
v
k
,
j
​
​
+
ϵ
g
k
,
j
​
​
.
In our version of β=9,\=1e-8, without bias control. For g1=[4.6] receive v1= [1.6.3.6], the two steps are almost the same:\ / G.1316228. New \[1.68772,683772] This is an adaptation of scale rather than the calculation of the total reverse Hessian. M06 explains the connection to Adam; the specific agreements are clearly announced here.
Experiment.
On the extended square, compare the GD,Momentum,RMSProp at one . Move the italics and look not only at the path but also at the state m/v. If the state has not fallen after reset, the comparison is incorrect.
Boundary.
v is the average
squares
Dispersion would require that the square of average be subtracted. Adding epsilon to the denominator prevents division by 0, but does not solve all problems of bad step or scale.
It's the memory of the direction, not the new weight itself, and it's involved in the next rule of correction.
How does this question continue?
Python examples of Chapter 05
Academic base