02.1 The matrix changes space

Content and key concepts

The vector describes the object with several coordinates.
One table of coefficients converts each point according to the same rule. The grid will help to see the tensile and the twist.
Take [1.2] and table[2.1], [0.1]: The first co-ordinate of the result: 2×1+1×2=4; the second: 0×1 + 1×2 =2; this is the example in the figure before the parameter changes.
Координаты и расчёт геометрического примера приведены в тексте. Для изменения пространственного рисунка включитеJavaScript.
The matrix sets the action immediately to the entire space.
The vector describes one object of space. The matrix sets a rule that works in harmony for each acceptable vector.
A
∈
R
m
×
n
,
x
∈
R
n
,
y
=
A
x
∈
R
m
.
A\in\mathbb R^{m\times n},
\qquad x\in\mathbb R^n,
\qquad y=Ax\in\mathbb R^m.
A
∈
R
m
×
n
,
x
∈
R
n
,
y
=
A
x
∈
R
m
.
Number of columns
A
A
A
has to match the size of the entrance, and the number of lines determines the dimension of the exit. This is not only the syntax rule: each string contains the coefficients of one output point,
y
i
=
∑
j
=
1
n
A
i
j
x
j
.
y_i=\sum_{j=1}^{n}A_{ij}x_j.
y
i
​
=
j
=
1
∑
n
​
A
ij
​
x
j
​
.
For
A
=
[
2
1
0
1
]
,
x
=
[
1
2
]
A=\begin{bmatrix}2&1\\0&1\end{bmatrix},
\qquad x=\begin{bmatrix}1\\2\end{bmatrix}
A
=
[
2
0
​
1
1
​
]
,
x
=
[
1
2
​
]
received
A
x
=
[
2
⋅
1
+
1
⋅
2
0
⋅
1
+
1
⋅
2
]
=
[
4
2
]
.
Ax=\begin{bmatrix}2\cdot1+1\cdot2\\0\cdot1+1\cdot2\end{bmatrix}
=\begin{bmatrix}4\\2\end{bmatrix}.
A
x
=
[
2
⋅
1
+
1
⋅
2
0
⋅
1
+
1
⋅
2
​
]
=
[
4
2
​
]
.
Tables are images of base vectors
Any entrance shall be arranged according to the standard base:
x
=
x
1
e
1
+
⋯
+
x
n
e
n
x=x_1e_1+\cdots+x_ne_n
x
=
x
1
​
e
1
​
+
⋯
+
x
n
​
e
n
​
Linearity gives
A
x
=
x
1
A
e
1
+
⋯
+
x
n
A
e
n
.
Ax=x_1Ae_1+\cdots+x_nAe_n.
A
x
=
x
1
​
A
e
1
​
+
⋯
+
x
n
​
A
e
n
​
.
Multiplier
e
j
e_j
e
j
​
Chooses
j
j
j
- the column, so
A
x
=
x
1
A
:
,
1
+
⋯
+
x
n
A
:
,
n
.
Ax=x_1A_{:,1}+\cdots+x_nA_{:,n}.
A
x
=
x
1
​
A
:
,
1
​
+
⋯
+
x
n
​
A
:
,
n
​
.
In our example,
A
e
1
=
[
2
,
0
]
T
Ae_1=[2,0]^T
A
e
1
​
=
[
2
,
0
]
T
,
A
e
2
=
[
1
,
1
]
T
Ae_2=[1,1]^T
A
e
2
​
=
[
1
,
1
]
T
, a
A
x
=
1
[
2
,
0
]
T
+
2
[
1
,
1
]
T
=
[
4
,
2
]
T
.
Ax=1[2,0]^T+2[1,1]^T=[4,2]^T.
A
x
=
1
[
2
,
0
]
T
+
2
[
1
,
1
]
T
=
[
4
,
2
]
T
.
It's easy to see how the whole grid is deformed: each straight grid is built from the same converted base directions.
Which saves a linear display
For any compatible
x
,
z
x,z
x
,
z
and scalars
α
,
β
\alpha,\beta
α
,
β
:
A
(
α
x
+
β
z
)
=
α
A
x
+
β
A
z
.
A(\alpha x+\beta z)=\alpha Ax+\beta Az.
A
(
αx
+
β
z
)
=
α
A
x
+
β
A
z
.
There are three geometrical properties that follow from here.
A
0
=
0
A0=0
A
0
=
0
. Directs move to straight lines or collide; parallels are retained if the directions have not been destroyed; angles and lengths are not kept by each matrix.
Single square with vertices
0
,
e
1
,
e
2
,
e
1
+
e
2
0,e_1,e_2,e_1+e_2
0
,
e
1
​
,
e
2
​
,
e
1
​
+
e
2
​
goes into parallelograms with the side of the table.
[
2
,
0
]
T
[2,0]^T
[
2
,
0
]
T
and
[
1
,
1
]
T
[1,1]^T
[
1
,
1
]
T
Its target area is multiplied by determinant, here by 2.
Linear and Affinity Change
Rule
x
↦
A
x
+
b
x\mapsto Ax+b
x
↦
A
x
+
b
to
b
≠
0
b\ne0
b

=
0
transports the beginning to
b
b
b
The difference between the two exits is still linear:
(
A
x
+
b
)
−
(
A
z
+
b
)
=
A
(
x
−
z
)
,
(Ax+b)-(Az+b)=A(x-z),
(
A
x
+
b
)
−
(
A
z
+
b
)
=
A
(
x
−
z
)
,
but strict equality
f
(
0
)
=
0
f(0)=0
f
(
0
)
=
0
Therefore, the bias of the neural layer cannot be undetected in the geometrical story of the linear operator without extending the coordinates or calling the display affinity.
90-degree turn
R
=
[
0
−
1
1
0
]
R=\begin{bmatrix}0&-1\\1&0\end{bmatrix}
R
=
[
0
1
​
−
1
0
​
]
translates
[
1
,
2
]
T
[1,2]^T
[
1
,
2
]
T
in
[
−
2
,
1
]
T
[-2,1]^T
[
−
2
,
1
]
T
The Diagonal matrix independently scalees the coordinates. The same number table can be the operator, dataset or covariance matrix; the role is defined by the task contract.
Temporal matvec size
[
m
,
n
]
[m,n]
[
m
,
n
]
in compliance
m
n
mn
mn
multipliers and
m
(
n
−
1
)
m(n-1)
m
(
n
−
1
)
It's in the direct algorithm.
[
4096
,
4096
]
[4096,4096]
[
4096
,
4096
]
It contains 16,777,216 numbers, i.e. 64 MiB float32 payload. This is only one numerical array: activations, gradiants, optimizer state and overhead are considered separately.
A, x = [[2, 1], [0, 1]], [1, 2]
y = [sum(row[j] * x[j] for j in range(2)) for row in A]
assert y == [4, 2]
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Each entry made its contribution. The big table repeats this rule for the rest of the exits. Here the vector is recorded with the column Ax; in Mini-GPT the lines use the entry xW.
How does this matter continue?
Python examples of Chapter 02
Academic base