17 Checkpoint: Inference and quantization

Content and key concepts

Progress is available with JavaScript. View is not equal to development.
Eight reviews of the mechanism
Threshold: 7 out of 8. without a timer. Balls are given for answers, not for visits. This is a local self-check; the decisions will be shown after dispatch.
1. Prefill processes...
Total prompt causal forward
Only the last token
Optimizer state
2. KV-cache stores...
Past K/V acts
New Webs
Trancing text
3. Change of seed at the same logs and policy...
Changes logs
Changes draw, but not reference distribution
Trains head
4. Batching first changes...
Requests service plan
Vocabulary order
Trained values
5. KV bytes: L2,B1,S4,H2,d4,FP16?
Your numerical response
6. INT4 symmetric qmax?
Your numerical response
7. Top-k at k=3 leaves the maximum candidates?
Your numerical response
8. Two requests for 3 decode tokens — use token steps?
Your numerical response
The result will appear after all the answers have been sent.
Research and explain
First, prove the coincidence of the cached decode and the full causal recompute on one sheckpoint. Then turn on the sampling filters and quantization separately, holding seed and prompt fixed. Do not release the browser aritmetic for hardware benchmark.
First, shape the hypothesis, then change one parameter. Compare the numbers to the formula; state the extreme case and the limit of applicability. The free conclusion is not evaluated by a dummy automatic score.