Designing a Validation Protocol Up Front
Deciding how a strategy will be judged after you've already seen how it performs is not validation, it's rationalisation. The protocol has to be written down before the first backtest runs.
Prerequisites: Backtest Overfitting
A researcher builds a strategy, tests it, likes the Sharpe ratio, and then decides the out-of-sample window should be "the last 18 months" — a window chosen after glancing at how the strategy did in the last 18 months. That isn't a validation step. It's picking the answer and then drawing the target around it. The protocol was written after the result, which means it wasn't a protocol at all.
What "up front" actually buys you
A validation protocol is the full set of decisions about how a strategy will be judged, fixed before any test result is known: which data window is in-sample, which is held out and untouched, what performance bar counts as a pass, how many variants are allowed before the whole line of research is abandoned, and what statistical correction applies given how many things were tried. Writing this down before running the strategy removes exactly one degree of freedom: the researcher's ability to move the goalposts once they've seen where the ball landed. That degree of freedom is small on any single decision and enormous in aggregate — it is the mechanism behind almost every backtest that looks great in the report and loses money live.
Worked example
Two teams test the same reversal signal on the same ten years of data.
Team A writes a one-page protocol before touching the data: in-sample is years 1–7, out-of-sample is years 8–10 and will be run exactly once, pass bar is Sharpe above 0.8 on the held-out window with a t-statistic above 2, and if the first version fails, the whole idea is shelved rather than re-tuned on the same held-out data. In-sample Sharpe comes out at 1.6. Out-of-sample, run once, comes out at 0.9 — a pass, if a modest one.
Team B has no written protocol. They tune on the full ten years, get an in-sample Sharpe of 2.1, informally treat the last two years as "out-of-sample" after the fact, don't like the 0.4 Sharpe they get, adjust the lookback parameter, check again, get 0.7, and ship it. Nothing here was individually dishonest. But the "out-of-sample" test was run four times against the same data with the freedom to keep adjusting until it passed, which means it wasn't really out-of-sample at all — it was in-sample data wearing an out-of-sample label.
| In-sample Sharpe | Held-out Sharpe reported | Times held-out data was touched | |
|---|---|---|---|
| Team A | 1.6 | 0.9 | 1 |
| Team B | 2.1 | 0.7 (after adjustment) | 4 |
Team B's final number is higher than Team A's honest one, and it is the less trustworthy of the two, because repeated peeking at held-out data converts it into another round of in-sample fitting.
A validation protocol is only doing its job if it is written and locked before the first result is seen. Any decision made after seeing performance — which window counts, what bar to clear, whether to retest — is no longer validation, it's curve-fitting with extra steps.
The common failure isn't skipping validation, it's writing the protocol loosely enough that it can flex around whatever result shows up — "roughly the last couple of years," "a reasonable Sharpe," "we'll know it when we see it." Vague protocols fail exactly as often as no protocol at all, because every ambiguous term is a place where the researcher's hindsight quietly leaks in.
Before running a single backtest, write the pass/fail bar, the held-out window's exact dates, and the maximum number of variants you'll test — in a document time-stamped or committed to version control before the first result exists. If you can't point to that artifact later, the validation didn't happen, no matter what the report says.
A protocol needs teeth: a stated consequence for failing it (shelve the idea, don't quietly retest) and a cap on how many variants get tried before the whole research line counts as one large multiple-testing exercise, not many independent ones. See Researcher Degrees of Freedom for how those unwritten choices accumulate, and Holdout Set Contamination for what happens when the held-out window gets touched anyway.
Related concepts
Practice in interviews
Further reading
- López de Prado, Advances in Financial Machine Learning (ch. 11-12)
- Harvey, Liu & Zhu, ...and the Cross-Section of Expected Returns