Alterna · learning material
17.5. Close balance storage
Replace a lot of exact values with a smaller set of codes and sizes. It saves memory, but it can cause a recovery error.
What this material covers
- Keep the approach on a discrete grid.
Content and key concepts
On the previous page, several queries use common computing.
Replace a lot of exact values with a smaller set of codes and sizes. It saves memory, but it can cause a recovery error.
The first start-up process the prefix. Each subsequent one adds one entry; K/V past entries can be saved. Sample and compressed storage solve different tasks.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Keep the approach on a discrete grid.
Symetric per-tensor in8 quantization selects scal using maximum module and rounds
x/s
in range
[-127,127]
. Dequantization returns
sq
not usually equal to the reference
x
Zero seems to be correct; one outlier extends the step for the whole tensor.
s
=
max
i
∣
x
i
∣
127
,
q
i
=
clip
(
round
(
x
i
/
s
)
,
−
127
,
127
)
,
x
^
i
=
s
q
i
.
s=\frac{\max_i|x_i|}{127},\quad
q_i=\operatorname{clip}(\operatorname{round}(x_i/s),-127,127),\quad
\hat x_i=sq_i.
s
=
127
max
i
∣
x
i
∣
,
q
i
=
clip
(
round
(
x
i
/
s
)
,
−
127
,
127
)
,
x
^
i
=
s
q
i
.
Per-channel scales reduces part of the error, but adds metadata and kernel requirements.
is not SmoothQuant
. F11.
That's what the model shows.
The graph shows the reference values, grid, codes and reconstructing error.Memory ledger is required to account for scales/zero-points/group metadata, and benchmark is actually used kernel. A smaller file without an accelerated kernel may not give lateny gain.
Scope of applicability.
A small max error of one testor does not prove the preservation of the end-task quality. Errors go through many layers; calibration data may not cover deployment distribution. "INT8" does not define the whole class, and "4-bit model" doesn't report group size, compute dype and quantized components.
values_17e=[-1.0,0.2,0.9]; scale_17e=max(map(abs,values_17e))/127
codes_17e=[max(-127,min(127,round(value/scale_17e))) for value in values_17e]
restored_17e=[code*scale_17e for code in codes_17e]
max_error_17e=max(abs(a-b) for a,b in zip(values_17e,restored_17e))
assert codes_17e==[-127,25,114] and max_error_17e<scale_17e/2
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Quantification shows code, scale and reconstruction error.
What is stored during the answer. How does this question continue the result?
Python examples of chapter 17
Academic base