16.3. Correct and independently verify

Content and key concepts

On the previous page, error and weight influence.
After a step, we measure a learning and verification error. The weight itself does not change.
Each known electron produces a projection of the following. The error goes backwards on the program, the optimizer changes the weight, and the individual data checks the result. The dialog and preferences mark changes the objective.
One model correction
Общая модель требуетJavaScript; её вычисления, формулы и код разобраны в открытом тексте далее.
Gradient and Rule of Renewal — Different Objects
Backward calculates the gradient at old parms; optimizer stores his own state and then changes the parameters. Adam uses exponential moments and bias control. AdamW adds decoupled weightdecay directly to the subdate, rather than mixing it with gradiant before addictive normalisation.
m
t
=
β
1
m
t
−
1
+
(
1
−
β
1
)
g
t
,
v
t
=
β
2
v
t
−
1
+
(
1
−
β
2
)
g
t
2
,
m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\quad
v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2,
m
t
​
=
β
1
​
m
t
−
1
​
+
(
1
−
β
1
​
)
g
t
​
,
v
t
​
=
β
2
​
v
t
−
1
​
+
(
1
−
β
2
​
)
g
t
2
​
,
θ
t
=
θ
t
−
1
−
η
t
(
m
t
/
(
1
−
β
1
t
)
v
t
/
(
1
−
β
2
t
)
+
ϵ
+
λ
θ
t
−
1
)
.
\theta_t=\theta_{t-1}-\eta_t\left(
\frac{m_t/(1-\beta_1^t)}{\sqrt{v_t/(1-\beta_2^t)}+\epsilon}
+\lambda\theta_{t-1}\right).
θ
t
​
=
θ
t
−
1
​
−
η
t
​
(
v
t
​
/
(
1
−
β
2
t
​
)
​
+
ϵ
m
t
​
/
(
1
−
β
1
t
​
)
​
+
λ
θ
t
−
1
​
)
.
Learning-rate class is defined by
optimizer steps
and, in the case of gradiant accumulation, one step may include several micro-batches. Checkpoint for continuation should contain not only webs but also moments, sceduler position, scaler mixed definition, step, RNG states, and data identity/config. The preservation of one webs is suitable for inference, but does not reproduce the continuation of training.
That's what the model shows.
The step model divides
zero_grad → forward → masked loss → backward → clip → optimizer → scheduler
In accumulation, the parameters should not be updated after each micro-batch. Weight decay does not usually automatically apply to all densors: bis and non-mormalization gains often receive a separate parameter group, which must be recorded in config.
Scope of applicability.
One scalar fixture does not reflect the distribution of densor gradients, mixed precinct and distributed release. Same seed does not guarantee bitwise reproducibility between hardware/kernels. Checkpoint from an untrusted source may be executed by a container in some formats; safe loading and checking hash is a separate responsibility.
from math import sqrt
w_16c,g_16c,lr_16c,b1_16c,b2_16c,eps_16c,wd_16c=1.0,0.2,0.1,0.9,0.99,1e-8,0.01
m_16c=(1-b1_16c)*g_16c; v_16c=(1-b2_16c)*g_16c*g_16c
w_next_16c=w_16c-lr_16c*((m_16c/(1-b1_16c))/(sqrt(v_16c/(1-b2_16c))+eps_16c)+wd_16c*w_16c)
assert abs(w_next_16c-0.899000005)<1e-9
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Validation measures a fixed state and does not perform an extradate.
Training examples of question and answer: How does this question continue?
Python examples of Chapter 16
Academic base