03.4 Many entrances and many exits

Content and key concepts

On the previous page, we collect the sensitivity in the vector.
Let's put it in a table, how each exit depends on each entry. This local sensitivity table is called the Jakobi matrix.
Two exits: [x2, xy]; x=2, y=1.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
One line of sensitivity for each exit
If
F
:
R
n
→
R
m
F:\mathbb R^n\to\mathbb R^m
F
:
R
n
→
R
m
the result has several components, so one gradient vector is not enough.
J
F
(
x
)
∈
R
m
×
n
,
(
J
F
)
i
j
=
∂
F
i
∂
x
j
.
J_F(x)\in\mathbb R^{m\times n},
\qquad (J_F)_{ij}=\frac{\partial F_i}{\partial x_j}.
J
F
​
(
x
)
∈
R
m
×
n
,
(
J
F
​
)
ij
​
=
∂
x
j
​
∂
F
i
​
​
.
The line refers to one exit, the column refers to the same entrance. Some sources use a transferable layout; the formulas are correct only after the agreement has been clearly announced.
For
F
(
x
,
y
)
=
[
x
2
x
y
]
F(x,y)=
\begin{bmatrix}
x^2\\xy
\end{bmatrix}
F
(
x
,
y
)
=
[
x
2
x
y
​
]
received
J
F
(
x
,
y
)
=
[
2
x
0
y
x
]
.
J_F(x,y)=
\begin{bmatrix}
2x&0\\y&x
\end{bmatrix}.
J
F
​
(
x
,
y
)
=
[
2
x
y
​
0
x
​
]
.
Point
z
=
[
2
,
1
]
T
z=[2,1]^T
z
=
[
2
,
1
]
T
:
F
(
z
)
=
[
4
,
2
]
T
,
J
F
(
z
)
=
[
4
0
1
2
]
.
F(z)=[4,2]^T,
\qquad J_F(z)=
\begin{bmatrix}4&0\\1&2\end{bmatrix}.
F
(
z
)
=
[
4
,
2
]
T
,
J
F
​
(
z
)
=
[
4
1
​
0
2
​
]
.
First column
[
4
,
1
]
T
[4,1]^T
[
4
,
1
]
T
describes the change in both exits when the entrance is driven by
x
x
x
; second
[
0
,
2
]
T
[0,2]^T
[
0
,
2
]
T
- when moving on
y
y
y
.
Jacobian is a local transformation matrix
For small
δ
\delta
δ
F
(
z
+
δ
)
≈
F
(
z
)
+
J
F
(
z
)
δ
.
F(z+\delta)\approx F(z)+J_F(z)\delta.
F
(
z
+
δ
)
≈
F
(
z
)
+
J
F
​
(
z
)
δ
.
Prix
δ
=
[
0.1
,
−
0.2
]
T
\delta=[0.1,-0.2]^T
δ
=
[
0.1
,
−
0.2
]
T
:
J
δ
=
[
4
0
1
2
]
[
0.1
−
0.2
]
=
[
0.4
−
0.3
]
.
J\delta=
\begin{bmatrix}4&0\\1&2\end{bmatrix}
\begin{bmatrix}0.1\\-0.2\end{bmatrix}
=\begin{bmatrix}0.4\\-0.3\end{bmatrix}.
J
δ
=
[
4
1
​
0
2
​
]
[
0.1
−
0.2
​
]
=
[
0.4
−
0.3
​
]
.
Linear prediction equals
[
4.4
,
1.7
]
T
[4.4,1.7]^T
[
4.4
,
1.7
]
T
and the exact exit.
F
(
2.1
,
0.8
)
=
[
4.41
,
1.68
]
T
.
F(2.1,0.8)=[4.41,1.68]^T.
F
(
2.1
,
0.8
)
=
[
4.41
,
1.68
]
T
.
Error
[
0.01
,
−
0.02
]
T
[0.01,-0.02]^T
[
0.01
,
−
0.02
]
T
is composed of members of the second order:
δ
x
2
\delta_x^2
δ
x
2
​
and
δ
x
δ
y
\delta_x\delta_y
δ
x
​
δ
y
​
.
Forward and Backward use different works
Production
J
δ
J\delta
J
δ
Transports a slight change in the front entrance and has a shape
[
m
,
1
]
[m,1]
[
m
,
1
]
Now let the scalar loss depend on
F
F
F
and upstream gradiant equals
g
F
=
∇
F
L
∈
R
m
×
1
.
g_F=\nabla_F L\in\mathbb R^{m\times1}.
g
F
​
=
∇
F
​
L
∈
R
m
×
1
.
Chain rule gives a gradiant at the entrance:
g
z
=
J
T
g
F
,
[
n
,
m
]
[
m
,
1
]
=
[
n
,
1
]
.
g_z=J^Tg_F,
\qquad [n,m][m,1]=[n,1].
g
z
​
=
J
T
g
F
​
,
[
n
,
m
]
[
m
,
1
]
=
[
n
,
1
]
.
For example, when
g
F
=
[
1
,
−
2
]
T
g_F=[1,-2]^T
g
F
​
=
[
1
,
−
2
]
T
:
J
T
g
F
=
[
4
1
0
2
]
[
1
−
2
]
=
[
2
−
4
]
.
J^Tg_F=
\begin{bmatrix}4&1\\0&2\end{bmatrix}
\begin{bmatrix}1\\-2\end{bmatrix}
=\begin{bmatrix}2\\-4\end{bmatrix}.
J
T
g
F
​
=
[
4
0
​
1
2
​
]
[
1
−
2
​
]
=
[
2
−
4
​
]
.
This is the sum of the deposits of all the exits to each entrance. Transpondering comes from the alignment of shapes and scalar differential:
d
L
=
g
F
T
d
F
=
g
F
T
J
d
z
=
(
J
T
g
F
)
T
d
z
.
dL=g_F^T\,dF=g_F^TJ\,dz=(J^Tg_F)^Tdz.
d
L
=
g
F
T
​
d
F
=
g
F
T
​
J
d
z
=
(
J
T
g
F
​
)
T
d
z
.
In large systems, the full Jacobian often doesn't materialize.
J
v
Jv
J
v
, reverse-mode — vector-Jacobian product
u
T
J
u^TJ
u
T
J
equivalent
J
T
u
J^Tu
J
T
u
These are different ways to apply the same derivative, not different definitions.
In the lab, change.
x
x
x
and
y
y
y
The zero element
J
01
=
0
J_{01}=0
J
01
​
=
0
means that the first exit is not locally dependent on
y
y
y
but does not cancel the existence of a second exit.
J = [[4, 0], [1, 2]]
delta = [.1, -.2]
forward = [sum(a*b for a, b in zip(row, delta)) for row in J]
upstream = [1, -2]
backward = [sum(J[i][j]*upstream[i] for i in range(2)) for j in range(2)]
assert all(abs(a - b) < 1e-12 for a, b in zip(forward, [.4, -.3]))
assert backward == [2, -4]
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
One modified entry can affect several exits. The Jakobi table collects all such local influences.
How does the inclination change itself? How does this question continue?
Python examples of Chapter 03
Academic base