Academic base of the full life cycle

Content and key concepts

Sources received prior to the writing of the lectures. Articles use available XML/abstract pages arXiv, for API and formats - official documentation. No published metrics are reprinted: all numbers in lectures and
verification.py
are small local fixtures.
Issues and map of evidence
Chapter
Research issues
Main support
16
How do you build next-token tags and masks? What is the difference between cross-entropy? How does AdamW differ from L2 inside the gradient? How is SFT separating prompt from the answer? What optimizes preference observations? How do LoRA change the number of learning parameters?
F01–F06, F10–F11
17
What does KV-cache have? How do we make a change in distribution? Why continuous batching? How does Quantification invent an error? What metrics describe service?
F07–F09, F12–F14
18
How do documents turn into targeted chunks? How does retrieval rank candidates? Where does RAG? How do you validate tools? How is the memory of an application different from the webs?
F15-F17; local engineering contracts
19
What really follows from scaling laws? How do data/tensor/pipeline parallelism share work? How does MoE route tokens? How are the modalities compared?
F18–F23
20
How do you formulate a system contract? How do we connect tran and genetics without leaking data? How can we check grounding and several quality measurements? What is included in the transfer package?
F24-F25; local through contract
Sources of training and recognition
F01. Language Models are Few-Shot Learners
Tom B. Brown et al., 2020, arXiv:2005.14165.
https://arxiv.org/html/2005.14165
The results of GPT-3 are not translated into a learning model or a promise by an emergent capacity.
F02. Decoupled Weight Decay Regularization
Ilya Loshchilov, Frank Hutter, 2017, arXiv:1711.05101.
https://arxiv.org/html/1711.05101
A support to separate the web decay from the addict gradiant update in AdamW. The training scalar step shows the order of operations, but does not reproduce benchmark articles.
F03. Training language models to follow instructions with human feedback
Long Ouyang et al., 2022, arXiv:2203.02155.
https://arxiv.org/html/2203.02155
The multi-stage scheme used is demonstrations \preference data/reward model \ RLHF. The lecture does not claim that SFT and RLHP are identical or that human assessment automatically makes the answer true and safe.
F04. Direct Preference Optimization
Rafael Rafailov et al., 2023, arXiv:2305.18290.
https://arxiv.org/html/2305.18290
Using pairwiss logical DOO on-projective using the different log-ratios policy/reference. Tiny fixture calculates only the loss of one pair; this is not a complete preference training or proof of DPO's superiority over other methods.
F05. LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu et al., 2021, arXiv:2106.09685.
https://arxiv.org/html/2106.09685
Support for frozen base matrix and low-range update BA.
alpha/r
It's clear that the savings are not automatically less lateny after the adapter is connected.
F06. PyTorch CrossEntropyLoss and AdamW
PyTorch subsidies, official reference documentation.
https://docs.pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html
https://docs.pytorch.org/docs/stable/generated/torch.optim.AdamW.html
The course code is not dependent on PyTorch, but only on small stdlib calculations.
Sources of information
F07. FlashAttention
Tri Dao et al., 2022, arXiv:2205.14135.
https://arxiv.org/html/2205.14135
The idea of IO-aware tiling accurate attentation and the difference in the mathematical operation and its mode of execution has been used. The published accelerations are hardwarely and configurationally dependent; they are not issued for a universal coefficient.
F08. Efficiently Scaling Transformer Inference
Reiner Pope et al., 2022, arXiv:2211.05102.
https://arxiv.org/html/2211.05102
The output of the article is not transferred to a browser or arbitrary GPU.
F09. Efficient Memory Management for LLM Serving with PagedAttention
Woosuk Kwon et al., 2023, arXiv:2309.06180.
https://arxiv.org/html/2309.06180
The dynamic growth of KV-cache, fragmentation and continuous batching/page management has been used. The training line does not implement vLM and does not repeat the declared thrugput applications.
F10. The Curious Case of Neural Text Degeneration
Ari Holtzman et al., 2019/2020, arXiv:1904.09751.
https://arxiv.org/html/1904.09751
And the support for nucleus sumpling and the saying that decoding policy changes output with a fixed model is empirical work, not assurance that one value top-p is optimal for all tasks.
F11. SmoothQuant
Guangxuan Xiao et al., 2022/2024, arXiv:2211.10438.
https://arxiv.org/html/2211.10438
The lecture uses the difference between weight/activation quantization and the role of outsiders. The lecture is a regular symmetric scalar quantisation, not SmoothQuant and not W8A8 kernel.
Sources of external systems
F12. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Patrick Lewis et al., 2020, arXiv:2005.11401.
https://arxiv.org/html/2005.11401
The support for the difference between parametric and non-parametic memory and co-retriever/generator pipeline. Local cosine retriever does not reproduce DPR, Wikipedia or RAG training.
F13. ReAct
Shunyu Yao et al., 2022/2023, arXiv:2210.03629.
https://arxiv.org/html/2210.03629
The overall resoning/action/observation cycle has been used. Our final machine does not generate hidden chain-of-thought or be declared an autonomous intellectual agent.
F14. Toolformer
Timo Schick et al., 2023, arXiv:2302.04761.
https://arxiv.org/html/2302.04761
The idea of external tools calls from the language model: Allowlist, JSON validation, timeout, idempency and audit are local production contracts of the course, not the results attributed to Toolformer.
Sources of scaling and multidimensionality
F15. Training Compute-Optimal Large Language Models
Jordan Hoffmann et al., 2022, arXiv:2203.15556.
https://arxiv.org/html/2203.15556
Joint planning has been used for parameter account and train tokens with limited compute.
6ND
In the lecture is a common rough estimate of the dense Transformer trainer, obviously not the exact profiler or the law of quality.
F16. Megatron-LM
Mohammad Shoeybi et al., 2019, arXiv:1909.08053.
https://arxiv.org/html/1909.08053
A small breakdown of the matrix demonstrates algebraic equivalence, but not communication kernels or the effectiveness of a particular cluster.
F17. GPipe
Yanping Huang et al., 2018/2019, arXiv:1811.06965.
https://arxiv.org/html/1811.06965
The support for pipeline steps, micro-batches and bubble. The course performance formula refers to a simplified synchronous schedule with equal stages.
F18. Switch Transformers
William Fedus, Barret Zoph, Noam Shazeer, 2021, arXiv:2101.03961.
https://arxiv.org/html/2101.03961
The support for sparse expert routing and capacity. Training top-1 router does not include auxiliary balancing loss, all-to-all communication or a trained gate.
F19. Learning Transferabl