Strategy Validation

Read these first

1. 01Adversarial Backtest Design

   Instead of asking whether a strategy passes its own backtest, adversarial design asks a second researcher to actively try to break it, and rewards them for succeeding.

2. 02Designing a Validation Protocol Up Front

   Deciding how a strategy will be judged after you've already seen how it performs is not validation, it's rationalisation. The protocol has to be written down before the first backtest runs.

3. 03Independent Replication by a Second Researcher

   Rebuilding a strategy from the written spec alone, without looking at the original code, catches the bugs and hidden assumptions that reviewing the original code never will.

4. 04Parameter Plateaus Versus Spikes

   A backtest's best parameter setting can sit on a broad plateau of nearby settings that all work, or on a narrow spike surrounded by settings that fail. Only the plateau is a real signal.

5. 05Requiring an Economic Rationale

   A backtest that passes every statistical test but can't explain why the market would pay for the signal is still probably noise. Requiring a rationale before the data mining starts filters most of it out for free.

Then the rest

1. 06Blind Reimplementation from Spec
2. 07Crisis-Period Carve-Out Testing
3. 08Historical Scenario Replay
4. 09Holdout Budget and Peek Accounting
5. 10Paper, Pilot and Full-Size Validation Stages
6. 11Parameter Stability Over Time
7. 12Perturbing Entry and Exit Rules
8. 13Red-Teaming a Strategy
9. 14Reverse Stress Testing a Strategy
10. 15Sensitivity to Cost Assumptions
11. 16Sensitivity to Rebalance Frequency
12. 17Sensitivity to Universe Definition
13. 18Specifying the Null Hypothesis for a Strategy
14. 19Structural Break Checks in Strategy Returns
15. 20Subperiod Decomposition of Returns
16. 21Surrogate Data Testing
17. 22The Lockbox Dataset
18. 23Validating a Strategy on Synthetic Price Paths