05.6. Limit step and change schedule

Content and key concepts

On the previous page, two gradient stories.
Too big a gradient, changing the weights themselves, and reducing the step are different reasons, so let's check them separately.
Gradient [3.4] is length5; we want to limit the length to 2.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Different means solve different problems
If norm(g) is a bike,
norm clipping
Limiting length by keeping the direction of non-zero gradient:
g
c
l
i
p
=
g
min
⁡
(
1
,
c
∥
g
∥
2
)
,
c
>
0.
g_{clip}=g\min\left(1,\frac{c}{\|g\|_2}\right),\quad c>0.
g
c
l
i
p
​
=
g
min
(
1
,
∥
g
∥
2
​
c
​
)
,
c
>
0.
Prig=[3.4],norm5,c2 receive[1.2.1.6],norma2. Prig = 0 multipliers are accepted without division. [c,c] is another operation and can change direction. Clipping does not show a reduction in loss: even a limited step may not be appropriate.
Weight decay in a simple separate subductive , the gradient becomes g+. With the usual GD, both concepts are consistent; with an adaptive optimizer, the integration , , into a standardised gradient and a separate decay is usually not equivalent. This is the motivation decoupled decay, not the promise of a single setup for all libraries.
Schedule
This is a procedure setting, not a new parameter of the model itself. Too fast a reduction can stop useful progress; too slow to leave fluctuations. The hypothesis needs to be checked by correct validation-task, not just by a beautiful training platform.
Final experiment.
On the square, select a large, watch the growth and stop at the safe limit, then reduce it. In a separate tool, compare nonm clipping and decay: Name which object is changing, gradient or parameters. Do not attribute a change in the schedule to a flag that does not appear in its model.
We've linked the representations, the transformations, the sensitivity, the probability criterion and the update.
Classical ML
This chapter is still in the programme; the mathematical framework and its self-checks are already available.
from math import hypot
g,c = [3,4],2
factor = min(1,c/hypot(*g))
clipped = [factor*v for v in g]
assert abs(hypot(*clipped)-2)<1e-12
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Restricting the gradient does not guarantee a smaller error at any step; it solves a narrower problem of sensitivity size.
How will these actions become a learning model?
Python examples of Chapter 05
Academic base