03.5 How the inclination changes itself

Content and key concepts

On the previous page, there are many entrances and many exits.
To understand the curve of the error surface, the derivatives are examined. The table of the second derivatives is called the Gessa matrix.
Surface f=x2+3y2.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
The production gradient is a matrix
For the scalar twice differentiated function, Hessian contains the second private derivatives:
H
f
(
x
)
=
[
∂
2
f
/
∂
x
1
2
⋯
∂
2
f
/
∂
x
1
∂
x
n
⋮
⋱
⋮
∂
2
f
/
∂
x
n
∂
x
1
⋯
∂
2
f
/
∂
x
n
2
]
.
H_f(x)=
\begin{bmatrix}
\partial^2f/\partial x_1^2&\cdots&\partial^2f/\partial x_1\partial x_n\\
\vdots&\ddots&\vdots\\
\partial^2f/\partial x_n\partial x_1&\cdots&\partial^2f/\partial x_n^2
\end{bmatrix}.
H
f
​
(
x
)
=
​
∂
2
f
/
∂
x
1
2
​
⋮
∂
2
f
/
∂
x
n
​
∂
x
1
​
​
⋯
⋱
⋯
​
∂
2
f
/
∂
x
1
​
∂
x
n
​
⋮
∂
2
f
/
∂
x
n
2
​
​
​
.
If mixed second derivatives are continuous, Hessian symmetry.
f
(
z
+
δ
)
≈
f
(
z
)
+
∇
f
(
z
)
T
δ
+
1
2
δ
T
H
f
(
z
)
δ
.
f(z+\delta)\approx
f(z)+\nabla f(z)^T\delta+\frac{1}{2}\delta^TH_f(z)\delta.
f
(
z
+
δ
)
≈
f
(
z
)
+
∇
f
(
z
)
T
δ
+
2
1
​
δ
T
H
f
​
(
z
)
δ
.
For
f
=
x
2
+
3
y
2
f=x^2+3y^2
f
=
x
2
+
3
y
2
:
H
=
[
2
0
0
6
]
.
H=\begin{bmatrix}2&0\\0&6\end{bmatrix}.
H
=
[
2
0
​
0
6
​
]
.
Prix
z
=
[
2
,
1
]
T
z=[2,1]^T
z
=
[
2
,
1
]
T
,
δ
=
[
0.01
,
0.02
]
T
\delta=[0.01,0.02]^T
δ
=
[
0.01
,
0.02
]
T
2nd order equals
1
2
δ
T
H
δ
=
1
2
(
2
⋅
0.01
2
+
6
⋅
0.02
2
)
=
0.0013.
\frac12\delta^TH\delta
=\frac12\left(2\cdot0.01^2+6\cdot0.02^2\right)
=0.0013.
2
1
​
δ
T
Hδ
=
2
1
​
(
2
⋅
0.0
1
2
+
6
⋅
0.0
2
2
)
=
0.0013.
That's how much linear prediction was missing. Because the function is quadratic, there are no third and higher-order members, so the second-order model is accurate.
Eigenvalues Hessian classify unborn stationary point
Fixed point
∇
f
=
0
\nabla f=0
∇
f
=
0
square-shaped sign
δ
T
H
δ
\delta^TH\delta
δ
T
Hδ
Determines the local picture:
positive specific
H
H
H
Provides a strict local minimum;
negative
H
H
H
gives a strict local maximum;
eigenvalues give a saddle.
Uh-oh.
x
2
+
3
y
2
x^2+3y^2
x
2
+
3
y
2
eigenvalues 2 and 6 are positive.
x
2
−
y
2
x^2-y^2
x
2
−
y
2
H
=
diag
⁡
(
2
,
−
2
)
,
H=\operatorname{diag}(2,-2),
H
=
diag
(
2
,
−
2
)
,
So in one direction the function grows, and in the other the function loses: the beginning is the saddle, although the gradient is zero.
Semi-determination not complete analysis
Functions
x
4
x^4
x
4
and
−
x
4
-x^4
−
x
4
in zero have the same
f
′
(
0
)
=
0
,
f
′
′
(
0
)
=
0
,
f'(0)=0,\qquad f''(0)=0,
f
′
(
0
)
=
0
,
f
′′
(
0
)
=
0
,
But the first has a minimum, the second has a maximum, the zero Hessian doesn't see the first non-zero cock of the fourth order, so the second order test at zero eigenvalues may be unauthorised.
The ratio of the largest and least positive eigenvalues squares describes the conditioning.
H
=
diag
⁡
(
2
,
6
)
H=\operatorname{diag}(2,6)
H
=
diag
(
2
,
6
)
The cooler direction limits the stable overall step of the gradient descent, and the more flat may come down slower.
Newton step uses a local square model
Minimizing the second-order model by
δ
\delta
δ
, received condition
∇
f
+
H
δ
=
0
,
H
δ
=
−
∇
f
.
\nabla f+H\delta=0,
\qquad H\delta=-\nabla f.
∇
f
+
Hδ
=
0
,
Hδ
=
−
∇
f
.
Point
(
2
,
1
)
(2,1)
(
2
,
1
)
:
[
2
0
0
6
]
δ
=
−
[
4
6
]
,
δ
=
[
−
2
,
−
1
]
T
.
\begin{bmatrix}2&0\\0&6\end{bmatrix}
\delta=-\begin{bmatrix}4\\6\end{bmatrix},
\qquad \delta=[-2,-1]^T.
[
2
0
​
0
6
​
]
δ
=
−
[
4
6
​
]
,
δ
=
[
−
2
,
−
1
]
T
.
One step goes exactly to the beginning, because the function being examined is itself a positive quadratic. For a total nonconvex loss Hessian, it can be indefinite, poorly conditioned, or change along the step; no one guarantees one exact transition.
Heavy Hessian for
P
P
P
parameters contain:
P
2
P^2
P
2
In the case of a vehicle which is not a part of the vehicle, the vehicle shall be capable of being subjected to the following conditions:
P
=
10
6
P=10^6
P
=
1
0
6
It's
10
12
10^{12}
1
0
12
So big models use Hessian-vector products, diagonal or structural approaches, rather than storage the full array.
In the lab, zoom in.
b
/
a
b/a
b
/
a
The Ellipse will become, and one common gradient step will be more limited to the cool axis. The negative coefficients are deliberately not accepted by the lab: the saddle is shown separately, without a false demonstration of "confluence".
g = [4, 6]
H = [[2, 0], [0, 6]]
newton_delta = [-g[0]/H[0][0], -g[1]/H[1][1]]
assert newton_delta == [-2, -1]
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
The second derivatives describe the change in inclinations. They are not the most wrong and do not prove the quality of the neurosnet.
The change is going through the chain. How does this question continue with the result?
Python examples of Chapter 03
Academic base