Transformers & Attention

Read these first

1. 01Encoder-Only, Decoder-Only and Encoder-Decoder Designs

   Every transformer-based model is built from the same attention block, but the choice of which tokens are allowed to see which other tokens splits models into three families with very different jobs: understanding, generating, and translating.

2. 02Multi-Head Attention

   A single attention lookup can only capture one notion of 'relevant' at a time. Multi-head attention runs several independent attention lookups in parallel, each free to learn a different kind of relationship, then combines what they each found.

3. 03Pretraining Objectives: MLM, CLM and Span Corruption

   Before a language model can do anything useful, it has to be given a self-supervised game to practice on billions of times, the choice of game (guess a hidden word, guess the next word, or guess a missing chunk) shapes what kind of model comes out the other end.

4. 04Queries, Keys and Values

   Attention lets a model look up relevant information from anywhere in a sequence, and it does that lookup with three learned projections of the same input: a query describing what you're looking for, a key describing what each item offers, and a value carrying what gets returned when there's a match.

5. 05The KV Cache and Incremental Decoding

   Generating text one token at a time would mean re-processing the entire sentence so far at every single step, unless the model remembers what it already computed, the KV cache is that memory, and it's the difference between a chatbot that's usable and one that isn't.

6. 06Rotary Position Embeddings (RoPE)

   RoPE tells a transformer where a word sits in a sentence by rotating its vector, so that the angle between two words' vectors after rotation encodes exactly how far apart they are, a trick that lets attention scores depend only on relative position.

Then the rest

1. 07ALiBi and Relative Position Bias
2. 08Causal and Padding Attention Masks
3. 09Cross-Attention Between Encoder and Decoder
4. 10Feedforward Blocks and Gated Linear Units
5. 11Instruction Tuning and Supervised Fine-Tuning
6. 12Learned Positional Embeddings
7. 13Pre-Norm vs Post-Norm Transformer Blocks
8. 14Scaled Dot-Product Attention and the √d Factor
9. 15Sinusoidal Positional Encoding
10. 16The Quadratic Cost of Attention
11. 17The Residual Stream View of Transformers
12. 18Vision Transformers and Patch Embeddings
13. 19Attention Head Specialisation and Probing
14. 20Chinchilla Compute-Optimal Scaling
15. 21Do Transformers Beat Linear Baselines for Forecasting?
16. 22FlashAttention and IO-Aware Attention
17. 23Induction Heads and In-Context Learning
18. 24Linear Attention and Kernel Feature Maps
19. 25Multi-Query and Grouped-Query Attention
20. 26PatchTST and Patched Time-Series Transformers
21. 27QLoRA and Quantized Fine-Tuning
22. 28Sparse Attention: Longformer and BigBird
23. 29Speculative Decoding
24. 30Transformer Training Instabilities and Loss Spikes