02.6 Correction through narrow intermediate space

Content and key concepts

On the previous page, turn, stretch, turn.
You don't always have to change the whole big table. A little update can be given by two smaller tables; this limits possible changes.
Update of table 4×4.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Bottlenecks limit the scope of updates
Let's have a basic matrix.
W
∈
R
m
×
n
W\in\mathbb R^{m\times n}
W
∈
R
m
×
n
It's still fixed. Let's do two updates.
C
∈
R
m
×
r
,
D
∈
R
r
×
n
,
Δ
W
=
C
D
.
C\in\mathbb R^{m\times r},
\qquad D\in\mathbb R^{r\times n},
\qquad \Delta W=CD.
C
∈
R
m
×
r
,
D
∈
R
r
×
n
,
Δ
W
=
C
D
.
Each column
C
D
CD
C
D
is a linear combination
r
r
r
Columns
C
C
C
, so
rank
⁡
(
Δ
W
)
≤
r
.
\operatorname{rank}(\Delta W)\le r.
rank
(
Δ
W
)
≤
r
.
This restriction applies to the update, not necessarily to the Frank reference
W
W
W
.
Forward can be executed without materializing the complete matrix
For the entrance
x
x
x
y
=
(
W
+
C
D
)
x
=
W
x
+
C
(
D
x
)
.
y=(W+CD)x=Wx+C(Dx).
y
=
(
W
+
C
D
)
x
=
W
x
+
C
(
D
x
)
.
The brackets on the right show the computational route: first
D
D
D
it's tight.
n
n
n
- the measuring entrance
r
r
r
coordinates, then
C
C
C
Returns the amendment to
m
m
m
- a dimensional space.
Here we go.
C
=
[
1
2
]
,
D
=
[
3
4
]
.
C=\begin{bmatrix}1\\2\end{bmatrix},
\qquad D=\begin{bmatrix}3&4\end{bmatrix}.
C
=
[
1
2
​
]
,
D
=
[
3
​
4
​
]
.
Then.
C
D
=
[
3
4
6
8
]
,
CD=\begin{bmatrix}3&4\\6&8\end{bmatrix},
C
D
=
[
3
6
​
4
8
​
]
,
And the second line is twice the first line, so Frank is equal to 1.
x
=
[
1
,
−
1
]
T
x=[1,-1]^T
x
=
[
1
,
−
1
]
T
:
D
x
=
[
−
1
]
,
C
(
D
x
)
=
[
−
1
,
−
2
]
T
.
Dx=[-1],
\qquad C(Dx)=[-1,-2]^T.
D
x
=
[
−
1
]
,
C
(
D
x
)
=
[
−
1
,
−
2
]
T
.
Explicit multiplication
(
C
D
)
x
(CD)x
(
C
D
)
x
It gives the same result.
When factorization actually saves the parameters
Full update
m
n
mn
mn
The factors contain
m
r
+
r
n
=
r
(
m
+
n
)
.
mr+rn=r(m+n).
m
r
+
r
n
=
r
(
m
+
n
)
.
There's only savings at the time.
r
(
m
+
n
)
<
m
n
.
r(m+n)<mn.
r
(
m
+
n
)
<
mn
.
For
m
=
n
=
4
m=n=4
m
=
n
=
4
:: at
r
=
1
r=1
r
=
1
8 instead of 16 scalars; at
r
=
2
r=2
r
=
2
I need 16, and the savings are gone.
m
=
n
=
4096
,
r
=
8
m=n=4096,r=8
m
=
n
=
4096
,
r
=
8
the factors are 65,536 versus 16,777,216 in total
Δ
W
\Delta W
Δ
W
, i.e. 256 times less than for the selected update.
Basic
W
W
W
And so "256 times less than the Trainable values of this matrix" cannot be reduced to "the whole model takes 256 times less memory".
The SVD compress of the finished matrix selects the factors from the already existing singular instructions. LoRA teaches low-range updating near frozen pre-traded matrix; this is a different procedure.
r
r
r
It limits the changes available and does not guarantee the quality of the task.
In the lower calculator change
m
,
n
,
r
m,n,r
m
,
n
,
r
, separately calling the payload
W
W
W
The next question is how to calculate the direction of change.
C
,
D
C,D
C
,
D
By mistake; for this purpose
Derivatives and gradients
.
x = [1, -1]
hidden = 3*x[0] + 4*x[1]
update = [hidden, 2*hidden]
assert hidden == -1 and update == [-1, -2]
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Savings limit the type of possible addition. The old large matrix does not disappear, and the usefulness of updating is not guaranteed by the number of parameters.
How do you know what happens with a small change in the coefficient?
Python examples of Chapter 02
Academic base