Overfitting & Validation

Read these first

1. 01In-Sample and Out-of-Sample Protocol

   Splitting history into a part you fit on and a part you test on only works if the test part is genuinely untouched. The protocol is about counting how many times you looked, because a hold-out you have peeked at twenty times is no longer a hold-out.

2. 02Combinatorially Symmetric Cross-Validation and PBO

   A procedure that splits history into blocks, tries every balanced train/test combination, and asks how often the variant that won in-sample lands below the median out-of-sample. The answer is the probability of backtest overfitting.

3. 03The Replication Crisis in Factor Research

   Hundreds of published stock-return predictors have been re-tested by independent teams. Most of them shrink badly and many vanish entirely. This page explains why, and how to read a factor paper without inheriting its optimism.

Then the rest

1. 04Block Bootstrap of Return Series
2. 05Cross-Market Replication Tests
3. 06Economic Versus Statistical Significance
4. 07Effective Number of Independent Trials
5. 08Expanding Versus Rolling Training Windows
6. 09Feature Selection Bias
7. 10Haircut Sharpe Ratio
8. 11Holdout Set Contamination
9. 12Monte Carlo Permutation Tests for Strategies
10. 13Parameter Sensitivity Analysis
11. 14Regime-Stratified Validation
12. 15Researcher Degrees of Freedom
13. 16Statistical Power Analysis for Backtests
14. 17White's Reality Check
15. 18The False Strategy Theorem