ML Foundations

Read these first

1. 01Hypothesis Space and Inductive Bias

   Data alone can never pick a model, infinitely many rules fit any finite dataset perfectly and then disagree completely about the next observation. The hypothesis space is the shortlist of rules you allow, and the inductive bias is the tiebreaker you use inside it. Both are assumptions you make, not facts the data gives you.

2. 02The IID Assumption and When It Breaks

   Almost every error bar, p-value and cross-validation score assumes your rows are independent draws from one unchanging distribution. Financial data violates both halves of that, which is why backtests look significant and then fail. This page shows exactly how much of your sample is real and what happens when the distribution moves.

3. 03Training Error vs Generalization Error

   The error a model reports on the data it was trained on and the error it will make on new data are two different numbers, and confusing them is the single most common way quant models fail in production.

4. 04Cross-Entropy and Log Loss

   Cross-entropy scores a probability forecast by how surprised it would have been by what actually happened, and the penalty for a confident wrong call grows without bound, which is exactly the behavior you want from a loss function.

5. 05Decision Thresholds and Cost-Sensitive Learning

   A model gives you a probability; a threshold turns that probability into an action. Where you put the cutoff should be decided by what mistakes cost you, not by the 0.5 that every library hands you as a default.

6. 06Objective Functions vs Evaluation Metrics

   What a model is trained to minimize and what a human uses to judge whether it's any good are usually two different numbers, and a model can improve on one while barely moving, or even worsening, the other.

7. 07Regularization as a Prior

   Adding a penalty to a loss function is not a hack. It is exactly the same arithmetic as writing down what you believed about the coefficients before you saw any data, and letting the evidence argue you out of it.

8. 08The Bias-Variance Decomposition

   A model's average squared error splits cleanly into three separate pieces, being systematically wrong, being erratically wrong, and noise nothing can fix, and the decomposition explains why a worse-looking model can beat a better-looking one out of sample.

Then the rest

1. 09Hyperparameters vs Parameters
2. 10Interpolation vs Extrapolation
3. 11Learning Paradigms Compared
4. 12Machine Learning vs Classical Statistics
5. 13Parametric vs Nonparametric Models
6. 14Batch, Mini-Batch and Online Training
7. 15Convex vs Non-Convex Loss Surfaces
8. 16Discriminative vs Generative Models
9. 17Early Stopping as Regularization
10. 18Error Analysis and Residual Inspection
11. 19Hinge Loss and Margin Maximization
12. 20Irreducible Error and the Bayes Rate
13. 21Model Capacity and Complexity Control
14. 22Negative Log-Likelihood as a Loss Function
15. 23Point Predictions vs Probabilistic Predictions
16. 24Signal-to-Noise Ratio and Learnability
17. 25Surrogate Losses and Classification Consistency
18. 26The Bayes Optimal Classifier
19. 27The Geometry of L1 vs L2 Penalties