In-Sample and Out-of-Sample Protocol

Everyone knows the rule: fit on one slice of history, test on another. Almost nobody follows it, because the rule is not about the split. It is about the count — how many separate times a number from the test slice influenced a decision. That count is almost never one, and once it is not one, the second slice is quietly in-sample too.

Three sets, not two

A working protocol has three disjoint blocks of time, in chronological order.

• Training set. Where parameters are estimated: regression coefficients, lookbacks, thresholds, model weights.
• Validation set. Where variants compete — 20-day versus 60-day momentum, five features or fifty, a tree or a linear model. Look at this as often as you like; that is what it is for.
• Test set. Touched once, at the end, to produce the number you report. Every additional look spends it.

Most researchers merge the second and third. They tune against the "out-of-sample" period, watch the Sharpe climb, and then report that Sharpe as evidence of generalisation. It is not evidence of anything except that the search worked.

Drag complexity — think of it as the number of rules, features or fitted parameters — and watch the two curves diverge. Training error (amber) falls forever; test error (green) turns back up. Now raise noise in data: the sweet spot slides left and the gap widens. Financial returns are almost all noise, which is why the usable complexity in a trading model is far lower than newcomers expect.

Worked example: the hold-out that proved nothing

You have 2010–2019 daily returns for training, and you keep 2022 as a hold-out. You search 1,000 variants of a momentum rule on the training window, take the best (in-sample Sharpe 2.8), and test it on 2022. Sharpe 1.9. Confirmed, surely?

It would be, if you had looked once. In practice the first variant scored 0.3, so you adjusted the volatility filter. Then 0.9. Then you changed the rebalance day, the universe, the stop rule. By the time you saw 1.9 you had queried the hold-out about 20 times.

Here is what 20 queries buys for free. The standard error of an annualised Sharpe estimated over $T$ years is roughly

$$\operatorname{SE}(\widehat{SR}) \approx \sqrt{\frac{1 + \tfrac{1}{2}SR^2}{T}},$$

which for a strategy with no edge at all ($SR = 0$) over one year is $\sqrt{1/1} = 1.0$. In plain terms: measure a worthless strategy over a single year and its Sharpe will typically land somewhere in the range −1 to +1 by luck alone.

Now take the largest of 20 independent draws from that distribution. The expected maximum of 20 standard normals is about 1.87. So the best of 20 peeks at a one-year hold-out, on strategies with zero true edge, is expected to score around 1.9.

Your 1.9 is exactly the number pure noise hands you. The hold-out did not validate the strategy; the hold-out became the training set the moment you used its feedback, and you then reported the in-sample maximum of a 20-way search as if it were out-of-sample.

Worked example: how much hold-out do you actually need?

Suppose the true edge is a respectable Sharpe of 0.5. How long must the test window be to distinguish it from zero at $t = 2$?

Rearranging the standard error above, $T \approx (t / SR)^2 \times (1 + \tfrac{1}{2}SR^2)$. With $t = 2$ and $SR = 0.5$:

$$T \approx \left(\frac{2}{0.5}\right)^2 \times \left(1 + 0.125\right) = 16 \times 1.125 = 18 \text{ years}.$$

Eighteen years of clean, untouched data to confirm a good strategy at a modest confidence level. Almost nobody has that. This is why serious validation stops leaning on one hold-out and moves to methods that reuse data honestly: Walk-Forward Analysis analysis, Purged & Embargoed Cross-Validation, and combinatorial approaches that estimate the whole distribution of out-of-sample outcomes rather than a single draw. See Minimum Backtest Length.

What good practice looks like

Split chronologically, never randomly — random splits scatter the future through the training set. Purge a gap between blocks at least as long as your label horizon, and embargo a few days after each test block so serial correlation cannot carry information backwards. Log every hold-out query with a date and a reason; the length of that log is a direct input to how much you discount the final number. Then report the count. A Sharpe of 1.9 from the first look and a Sharpe of 1.9 from the twentieth are different claims, and only one is a result.