Academic base of the language unit

Content and key concepts

Date: 2026-09-14. Sources are available via webfetch prior to implementation. The following are the specific parts read and the limits of the output; the research results have not been reproduced.
Questions and experience chap
Chapter
Research issues
Support
12
What are the differences between UTF-16 units, code points, grapemes and bytes? How does the ID relate to the dictionary? How do BPE merges learn? How to allow ties and overlap? Why should the decode restore the line? What gives byte fallsback? How is SenencePiece different from the segmentation algorithm?
L01,L02,L03; own byte-BPE contract
13
Why is one-hot multiplication equal to lookup? What parameters actually teach skip-gram? Why is full softmax different from negative sumpling? Where do cosine neighbours come from? What is lost in PCA? Why are static embedding and coherent state different?
L04,L05,L06; Linear algebra of previous chapters
14
Why mix positions? How to get Q/K/V? Why divide on Gd_head? What axis is softmax? Where is causal mask? What happens to V? How is concat different from averaging heads?
L06 §3.2
15
How does the pre-norm decoder? How does it differ from the original encoder-decoder ? How do positions change symmetry? How to remove RoPE relative dot? What matrices do SwiGLU need? Why does stack not increase vocabular? What are the links between hidden states, LMhead and next-token tags?
L06,L07,L08; own distinct decoder contract
Sources
L01. String.length — MDN
MDN contributors, reference page; revised with date of change 2026-09-02.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/length
The definitions of UTF-16 code units, eterator/code points and Intl.Segmenter/graphes are read. This is a technical description, not a separate reading of the full Unicode Standard.TextEncoder/TextDecoder works with UTF-8; lone surrogate is excluded from the reversible contract, BOM is obviously retained.
L02. Neural Machine Translation of Rare Words with Subword Units
Rico Sennrich, Barry Haddow, Alexandra Birch, ACL2016, DOI10.18653/v1/P16-1162.
https://aclanthology.org/P16-1162/
https://arxiv.org/html/1508.07909v5
ACL landing gave a bibliography; it was then received by HTML paper, reading §3.2 from algorichm1, tranying/apply merge rules and vocablary trade-off. Late part of the answer is cut in terms. In article character/subword BPE inside words, with end-of-word marker.
Our version is different:
Starts with 256 UTF-8 bytes, saves spaces, allows merges through them, without linguistic pre-tokenizer. It does not appear as tiktoken, SenencePiece or accurate tokenizer GPT.
L03. SentencePiece
Taku Kudo, John Richardson, EMNLP2018, arXiv:1808.06226.
https://arxiv.org/abs/1808.06226
Abstact/metadata has been received: subword models are taught from raw sentences without an external word pre-tokenisation. Full paper and finished models were not read/not downloaded. SensencePiece is described as a toolkit, not synonymous with BPE. Side-by-side experience compares our byte-BPE with clearly named byte/codepoint/word-lake basic breakups, does not simulate real WordPiec models.
L04. Distributed Representations of Words and Phrases and their Compositionality
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean,2013, arXiv:1310.4546.
https://arxiv.org/abs/1310.4546
Abstruct-only: Skip-gram, negative sumpling, co-text presentations and statements of the order of words.
complete softmax
the formulas and gradients are removed and tested on their own.
L05. D2L word2vec section
URL
https://d2l.ai/_sources/chapter_natural-language-processing-pretraining/word2vec.rst.txt
was requested, but transport error. It is not considered to have been read by the source. L04 does not include details of this inaccessible text.
L06. Attention Is All You Need
Ashish Vaswani et al.,2017, arXiv:1706.03762v7(2023).
https://arxiv.org/html/1706.03762v7
The contents of TML have been obtained. Use has been made of §3.1 - 3.5: Scaled application, multi-head steps, decoder mask, residual, FFN, positions, embedding/head sting.
post-norm encoder-decoder
in our training decoder, pre-norm, without encoder/cross-attention/dropout, the modern combination of components is not called the exact replica of GPT-2 or original Transformer.
L07. RoFormer
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu,2021; arXiv:2104.09864v5(2023).
https://arxiv.org/html/2104.09864v5
§§3.1 to 3.4:2D roting, pairing, geometric frequancy, relative-position dot. The end of the answer is cut later in experiences/references. We cannot bear the inaccuracy of transferring the HTML form: use an independently verified (R_mq)T(R_nk)=qTR_(n-m)k. We do not promise a monotonic reduction of each score with distance or unlimited extrapolation of context.
L08. GLU Variants Improve Transformer
Noam Shazeer,2020, arXiv:2002.05202v1.
https://arxiv.org/html/2002.05202v1
It's got a full visible XML, used formulas2–6: ReLU-FFN, SiLU/Swiss gate, three SwiGLU matrices, an adjustment of hidden width when comparing dudgets. Our comparison mode keeps the same d_ff and therefore clearly shows a different number of parameters. It is not claimed that SwigLU has the same value or always better.
Substantive corrections and limits
Specification 15.2 calls the activity-invariant. For a non-accelerated Self-attentation without position signals correct
permutation-equivariant
The fixed causal mask itself sets the order limit, so the statement is not literally transferred to it.
Byte-BPE does teach merge rules, Skip-gram does update E/U. Decoder at this stage fulfils the current parameterized forward: byte-BPE
not trained in language
; training the hulls, checkpoints and quality generation are 16–17. Even an entire architecture does not become a trained LLM from the right formulas.
The kit is a targeted review of: primary full texts for BPE/attention/RoPE/GLU, two abstracts and technical Unicode profile. There are eight independent works for each chapter. The course's source and numeric facts are written independently, without copying large parts of paper.