Model Validation & MLOps

Read these first

1. 01Train, Validation and Test Splits

   You have to hold data back twice, for two different reasons, once to choose between models, once to grade the winner. Collapsing those two jobs into one is why so many models look good on paper and disappoint in production.

2. 02Benchmark Design and Leaderboard Overfitting

   A fixed test set stops being a fair test the moment it's used to choose between many models, every comparison leaks a little information back, and enough comparisons can make a leaderboard measure luck as much as skill.

3. 03Calibration Curves and Reliability Diagrams

   A model that says '70% chance' should be right about 70% of the time it says that, calibration checks whether a model's predicted probabilities can actually be trusted as probabilities, separately from whether it ranks outcomes well.

4. 04Concept Drift Detection

   A model's accuracy on live data can quietly decay because the relationship it learned between inputs and outputs has changed, detecting that shift early, before it shows up in P&L, is a separate discipline from validating the model in the first place.

5. 05Conformal Prediction

   A way of turning any trained model's point predictions into intervals that are guaranteed to cover the true value a chosen percentage of the time, without assuming anything about the model or the data's distribution.

6. 06Designing an Ablation Study

   An ablation study removes one piece of a model at a time to find out which parts are actually doing work, the only reliable way to tell a genuinely useful feature or component from one that's just along for the ride.

7. 07Model Monitoring in Production

   A model that scored well in backtest can still fail quietly once it is live, because the world it sees starts to differ from the world it was trained on. Monitoring is the discipline of catching that drift before the P&L does.

8. 08The Confusion Matrix and Classification Metrics

   A classifier can only ever produce four kinds of outcome, and almost every metric you have heard of is one ratio built from those four counts. Learn the little two-by-two table and precision, recall, F1 and specificity stop being jargon.

9. 09Utility-Based Model Evaluation

   Accuracy and AUC treat every mistake as equally bad, but in trading and risk the cost of a false positive and a false negative are almost never the same, evaluating a model on its actual economic payoff often changes which model wins.

Then the rest

1. 10Baseline Models and Naive Benchmarks
2. 11Regression Error Metrics
3. 12AUC Pitfalls Under Severe Class Imbalance
4. 13Bayesian Optimization for Model Search
5. 14Brier Score and Proper Scoring Rules
6. 15Champion-Challenger and Shadow Deployment
7. 16Choosing the Operating Point
8. 17Cost-Sensitive Evaluation With Asymmetric Errors
9. 18Covariate Shift and Domain Adaptation
10. 19Cross-Validation When Samples Are Dependent
11. 20Error Analysis and Slice-Based Evaluation
12. 21Expected Calibration Error
13. 22Experiment Tracking and Reproducibility
14. 23How Much Data You Need to Detect an Improvement
15. 24K-Fold Cross-Validation
16. 25Learning Curves and Sample Complexity
17. 26LIME and Local Surrogate Models
18. 27McNemar's Test for Comparing Classifiers
19. 28Model Governance and SR 11-7
20. 29Model Selection with AIC and BIC
21. 30Partial Dependence Plots
22. 31Permutation Feature Importance
23. 32Precision-at-K and Top-Decile Evaluation
24. 33Precision-Recall Curves
25. 34Repeated K-Fold and the Variance of CV Estimates
26. 35Retraining Cadence and Model Staleness
27. 36Robustness and Adversarial Evaluation
28. 37ROC Curves and AUC
29. 38Subgroup Performance and Fairness Slices
30. 39White's Reality Check and the SPA Test