05.1 Error as a surface

Content and key concepts

The Gradient describes the local increase in error.
Record the data and change the model settings. The height of the surface is the error of these settings.
Function x2+3y2 at point [2.1].
Координаты и расчёт геометрического примера приведены в тексте. Для изменения пространственного рисунка включитеJavaScript.
Points now mean parameters
In previous graphs x could be the input. In optimization, the variableta means the model parameters, and the data are recorded. Each point of theta sets a different model, the value of L(XXVI) its error on the selected data. A small train loss does not prove quality in new examples. M05.
The learning surface of L(x,y)=ax2+by2 at a,b>0 has a minimum of 0 at first. Here are x,y two co-ordinates of theta rather than two learning examples. The level line L=c is an ellipse; the ratio of coefficients determines the length. For a=1,b=3, theta=[2.1] the value.
∇
L
=
(
2
a
x
,
2
b
y
)
T
,
H
=
diag
⁡
(
2
a
,
2
b
)
.
\nabla L=(2ax,2by)^T,\qquad H=\operatorname{diag}(2a,2b).
∇
L
=
(
2
a
x
,
2
b
y
)
T
,
H
=
diag
(
2
a
,
2
b
)
.
If b/a is great, the error changes faster along y. One common lightning rate should take into account the coolest direction. In large networks, a two-dimensional picture is only an illustration: the real surface has millions of coordinates, saddles and symmetry.
Experiment.
Before starting, select b=1,3.10 and predict the form of the levels. Then start the algorithms from the same start. Compare
The same budget for steps
But remember, one step of different methods has different costs and status.
Easy to be wrong.
The loss graph of the step number and the coordinate map are not the same. The long trajectory does not automatically mean a slow program, and the quick descent on the toy square does not prove the superiority in the language. There are no measured GPU througput and universal optimization ratings in the lab.
The settings have changed; the data and the rule of error evaluation remain the same. The success of the step needs to be rechecked.
How does this question go about the result?
Python examples of Chapter 05
Academic base