05.5. Two gradient stories

Content and key concepts

On the previous page, consider past corrections.
Adam's optimizer stores the smooth gradient and its square. It affects the length of the next correction, and the initial displacement is corrected.
First gradient four, both stories start with zero.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Two moments and two denominators
Adam connects the EMA direction and second point. When initialization is zero, the first estimates are offset to zero: the amount of EMA-weights used is less than 1. For constant g, the weight sum is equal to 1-βk, which explains the bias correction. M06; the origin of the M09 abstract algorithm.
m
k
=
β
1
m
k
−
1
+
(
1
−
β
1
)
g
k
,
v
k
=
β
2
v
k
−
1
+
(
1
−
β
2
)
g
k
2
,
m_k=\beta_1m_{k-1}+(1-\beta_1)g_k,\quad
v_k=\beta_2v_{k-1}+(1-\beta_2)g_k^2,
m
k
​
=
β
1
​
m
k
−
1
​
+
(
1
−
β
1
​
)
g
k
​
,
v
k
​
=
β
2
​
v
k
−
1
​
+
(
1
−
β
2
​
)
g
k
2
​
,
m
^
k
=
m
k
1
−
β
1
k
,
v
^
k
=
v
k
1
−
β
2
k
,
θ
k
+
1
=
θ
k
−
η
m
^
k
v
^
k
+
ϵ
.
\hat m_k=\frac{m_k}{1-\beta_1^k},\quad
\hat v_k=\frac{v_k}{1-\beta_2^k},\quad
\theta_{k+1}=\theta_k-\eta\frac{\hat m_k}{\sqrt{\hat v_k}+\epsilon}.
m
^
k
​
=
1
−
β
1
k
​
m
k
​
​
,
v
^
k
​
=
1
−
β
2
k
​
v
k
​
​
,
θ
k
+
1
​
=
θ
k
​
−
η
v
^
k
​
​
+
ϵ
m
^
k
​
​
.
All vector operations in the last fraction are coordinated. in laboratory β1=9,β2=999,\=1e-8; k begins with 1 for the first subdate. g=[4.6]:m1=[4,.6],v1= [.016,036]; after correction receives[4.6][16.36]. C=1 new almost [1.9,.9], with microscopic correction epsilon.
Debug.
Without control, the first step is scaled to about 1/ G.001E.3.1623 relative to the normalized g.
eta*mhat/sqrt(vhat) + eps
Doesn't equal
eta*mhat/(sqrt(vhat)+eps)
. The brackets, the position of the epsilon and the step number are important.
Research.
Set the start [0.0]. For zero moments and gradient, all algorithms must be left without NaN first. Then return the normal start and compare the 1.10.50 stags. Adam does not guarantee a better result for each k and is not universally consistent for any gradiants sequences.
g, beta1, beta2 = 4,.9,.999
m, v = (1-beta1)*g,(1-beta2)*g*g
mhat, vhat = m/(1-beta1),v/(1-beta2)
assert abs(mhat-4)<1e-12 and abs(vhat-16)<1e-12
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Adam keeps two different stories, not just yesterday's gradient. The exact correction depends on both.
We limit the step and change the schedule.
Python examples of Chapter 05
Academic base