16.6 Small addition to the large table

Content and key concepts

On the previous page, we learn from two answers.
Freeze the old weights and train the smaller tables that make an addition. This brings back the idea of a limited update from Chapter 02.
Each known electron produces a projection of the following. The error goes backwards on the program, the optimizer changes the weight, and the individual data checks the result. The dialog and preferences mark changes the objective.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
We're training a low-range supplement, not a copy of the whole matrix.
For frozen base matrix
W∈R^(d_out×d_in)
LoRA asks update as a product of two smaller matrices.
r
substantially less than both dimensions, the trainable scalar count is
r(d_in+d_out)
and the dense update would have
d_in*d_out
. F05.
h
=
W
x
+
α
r
B
A
x
,
A
∈
R
r
×
d
i
n
,
B
∈
R
d
o
u
t
×
r
.
h=Wx+\frac{\alpha}{r}BAx,
\qquad A\in\mathbb R^{r\times d_{in}},\quad
B\in\mathbb R^{d_{out}\times r}.
h
=
W
x
+
r
α
​
B
A
x
,
A
∈
R
r
×
d
in
​
,
B
∈
R
d
o
u
t
​
×
r
.
The initialization of one factor zero is equal to zero, and base model behavior is retained. After learning adapter, you can store separately or merge in
W
;merge changes deployment artefact and must be recorded in manifest. The choice of Target modules, Frank, Alpha, dropout and dtype is in config.
That's what the model shows.
Inspector is separately visible frozen path
Wx
, adapter path
BAx
, scaling and sum. The change in Frank changes the maximum ring update and the number of parameters being taught, but does not guarantee the right quality. Several adapters can use one base model if the text is consistent with architecture, tensor names and tokenizer contract.
Scope of applicability.
PEFT reduces the Trainable variables and optimizer state, but forward still uses base lines. Adapter is able to inherit and reinforce the limitations of base model. You cannot mix incompatible heckpoints with matching Shapes without promotion; Shape compativity does not prove semantic compatibility.
W_16f=[[1,0,0],[0,1,0]]; A_16f=[[1,2,0]]; B_16f=[[0.5],[-0.5]]; x_16f=[1,1,1]
rank_16f,alpha_16f=1,2
ax_16f=[sum(a*x for a,x in zip(A_16f[0],x_16f))]
delta_16f=[alpha_16f/rank_16f*row[0]*ax_16f[0] for row in B_16f]
base_16f=[sum(a*x for a,x in zip(row,x_16f)) for row in W_16f]
assert base_16f==[1,1] and delta_16f==[3.0,-3.0] and rank_16f*(3+2)==5
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Trainable facts are asked by low-rank ΔW without rewriting the base W.
How can you use it without repeating your training on every request?
Python examples of Chapter 16
Academic base