03.3. Collect sensitivity into vector

Content and key concepts

On the previous page, change one reason at a time.
The output from the individual coordinates can be collected together. The gradient obtained shows the direction of local growth of the function.
f=x2+3y2; point [2.1].
Координаты и расчёт геометрического примера приведены в тексте. Для изменения пространственного рисунка включитеJavaScript.
Private derivatives form a linear function
For differentialized scalar function
f
:
R
n
→
R
f:\mathbb R^n\to\mathbb R
f
:
R
n
→
R
gradient in column agreement equals
∇
f
(
x
)
=
[
∂
f
/
∂
x
1
⋮
∂
f
/
∂
x
n
]
∈
R
n
×
1
.
\nabla f(x)=
\begin{bmatrix}
\partial f/\partial x_1\\
\vdots\\
\partial f/\partial x_n
\end{bmatrix}
\in\mathbb R^{n\times1}.
∇
f
(
x
)
=
​
∂
f
/
∂
x
1
​
⋮
∂
f
/
∂
x
n
​
​
​
∈
R
n
×
1
.
It encodes the first order for any small one.
δ
\delta
δ
:
f
(
x
+
δ
)
≈
f
(
x
)
+
∇
f
(
x
)
T
δ
.
f(x+\delta)\approx f(x)+\nabla f(x)^T\delta.
f
(
x
+
δ
)
≈
f
(
x
)
+
∇
f
(
x
)
T
δ
.
Uh-oh.
f
(
x
,
y
)
=
x
2
+
3
y
2
f(x,y)=x^2+3y^2
f
(
x
,
y
)
=
x
2
+
3
y
2
at point
(
2
,
1
)
(2,1)
(
2
,
1
)
:
g
=
∇
f
(
2
,
1
)
=
[
4
,
6
]
T
,
∥
g
∥
2
=
52
.
g=\nabla f(2,1)=[4,6]^T,
\qquad \|g\|_2=\sqrt{52}.
g
=
∇
f
(
2
,
1
)
=
[
4
,
6
]
T
,
∥
g
∥
2
​
=
52
​
.
Gradient has a shape entrance, not a shupe.
[
n
,
n
]
[n,n]
[
n
,
n
]
For loss with
P
P
P
parameters this
P
P
P
One at the coordinates of the current parameter state.
Direct derivative compares single movements
For direction
u
u
u
single euclidic length
D
u
f
(
x
)
=
∇
f
(
x
)
T
u
,
∥
u
∥
2
=
1.
D_uf(x)=\nabla f(x)^Tu,
\qquad \|u\|_2=1.
D
u
​
f
(
x
)
=
∇
f
(
x
)
T
u
,
∥
u
∥
2
​
=
1.
You get four and six along the axles.
u
=
g
/
∥
g
∥
2
u=g/\|g\|_2
u
=
g
/∥
g
∥
2
​
:
D
u
f
=
g
T
g
∥
g
∥
2
=
∥
g
∥
2
=
52
≈
7.2111.
D_uf=g^T\frac g{\|g\|_2}=\|g\|_2=\sqrt{52}\approx7.2111.
D
u
​
f
=
g
T
∥
g
∥
2
​
g
​
=
∥
g
∥
2
​
=
52
​
≈
7.2111.
Inequality of Koshi-Bunyakovsky
g
T
u
≤
∥
g
∥
2
∥
u
∥
2
=
∥
g
∥
2
g^Tu\le\|g\|_2\|u\|_2=\|g\|_2
g
T
u
≤
∥
g
∥
2
​
∥
u
∥
2
​
=
∥
g
∥
2
​
Proves the maximum local growth if limited
∥
u
∥
2
=
1
\|u\|_2=1
∥
u
∥
2
​
=
1
The word "as soon as possible" depends on the standard chosen: if there is another limitation, many acceptable directions will be different.
The anti-gradient sets a direction but no safe length
The step gradient descent is kind of
x
n
e
w
=
x
−
η
∇
f
(
x
)
.
x_{\mathrm{new}}=x-\eta\nabla f(x).
x
new
​
=
x
−
η
∇
f
(
x
)
.
Prix
η
=
0.1
\eta=0.1
η
=
0.1
:
[
2
,
1
]
T
−
0.1
[
4
,
6
]
T
=
[
1.6
,
0.4
]
T
,
[2,1]^T-0.1[4,6]^T=[1.6,0.4]^T,
[
2
,
1
]
T
−
0.1
[
4
,
6
]
T
=
[
1.6
,
0.4
]
T
,
f
(
1.6
,
0.4
)
=
2.56
+
0.48
=
3.04
<
7.
f(1.6,0.4)=2.56+0.48=3.04<7.
f
(
1.6
,
0.4
)
=
2.56
+
0.48
=
3.04
<
7.
But under
η
=
0.5
\eta=0.5
η
=
0.5
The same gradiant gives
[
2
,
1
]
T
−
0.5
[
4
,
6
]
T
=
[
0
,
−
2
]
T
,
f
(
0
,
−
2
)
=
12
>
7.
[2,1]^T-0.5[4,6]^T=[0,-2]^T,
\qquad f(0,-2)=12>7.
[
2
,
1
]
T
−
0.5
[
4
,
6
]
T
=
[
0
,
−
2
]
T
,
f
(
0
,
−
2
)
=
12
>
7.
Direction
−
g
-g
−
g
it doesn't give the right lightning line automatically.
Point
(
0
,
0
)
(0,0)
(
0
,
0
)
The gradient of this positive quadratic function is zero, and the point is the global minimum.
∇
f
=
0
\nabla f=0
∇
f
=
0
means only instability: further information is needed on the curvature or behaviour of the function.
Overtaking the final differences for all
P
P
P
parameters require order
P
P
P
This does not make the calculation free: you have to store or restore the intermediate values.
In the lab, compare
η
=
0.1
\eta=0.1
η
=
0.1
and
0.5
0.5
0.5
The new experiment shall restore the reference point: you cannot produce a sequence of different settings per test site.
x, y, eta = 2, 1, .1
g = [2*x, 6*y]
next_point = [x - eta*g[0], y - eta*g[1]]
next_value = next_point[0]**2 + 3*next_point[1]**2
assert all(abs(a - b) < 1e-12 for a, b in zip(next_point, [1.6, .4]))
assert abs(next_value - 3.04) < 1e-12
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Each reason is first changed separately; the Gradient combines sensitivity; a small step backwards has reduced the value in this example.
Many entrances and many exits. How does this question continue?
Python examples of Chapter 03
Academic base