Optimization & Training

Read these first

1. 01Data-Parallel Training

   Split a batch of training data across many machines, each holding a full copy of the model, and average their gradients together after every step, the standard way to make training faster without changing the maths at all.

2. 02Loss Landscape Geometry and Sharpness

   Two training runs can reach the exact same training loss and still behave completely differently on new data, because one settled into a wide, forgiving valley and the other into a narrow, easily-toppled spike.

3. 03Mixed-Precision Training

   Doing most of a neural network's arithmetic in a cheaper, less precise number format roughly halves training time and memory, provided you protect the handful of operations that would otherwise silently break.

4. 04SGD Noise and Implicit Regularization

   Stochastic gradient descent's mini-batch randomness isn't just a computational shortcut, it actively steers training away from the kind of overfit solutions full-batch gradient descent would happily settle into.

5. 05Tensor and Pipeline Parallelism

   When a model is too large to fit on one device, you have to split the model itself rather than just the data, either cutting each layer's math across devices, or handing whole layers to different devices in sequence.

Then the rest

1. 06Random Search vs Grid Search
2. 07AdamW and Decoupled Weight Decay
3. 08Cosine Annealing and Warm Restarts
4. 09Curriculum Learning
5. 10Designing a Hyperparameter Search Space
6. 11Early Stopping Criteria and Patience
7. 12Gradient Checkpointing for Memory
8. 13RMSProp and AdaGrad
9. 14Stochastic Weight Averaging
10. 15Successive Halving and Hyperband
11. 16The Linear Scaling Rule and Learning-Rate Warmup
12. 17The One-Cycle Learning-Rate Policy
13. 18All-Reduce and Distributed Training Collectives
14. 19Catastrophic Forgetting and Continual Learning
15. 20Gradient Noise Scale and Critical Batch Size
16. 21Ill-Conditioning and the Hessian Spectrum
17. 22Saddle Points and Plateaus
18. 23Sharpness-Aware Minimization
19. 24Training Instability and Loss Spikes