White's Reality Check

Say a researcher tests 500 trading rules against historical data and finds one with an impressively high Sharpe ratio. Is that one rule genuinely good, or is it simply the best performer you'd expect to find among 500 rules purely by chance, even if every single one of them was actually worthless? A standard statistical significance test on that one rule in isolation can't answer this, because it doesn't know 499 other rules were tried and discarded first. White's Reality Check is a test built specifically to answer this question: it accounts for the entire search, not just the winner, when deciding whether the best result is genuinely significant.

How the test works

The core trick is to simulate what "the best of many trials" would look like if none of the trials had any real skill — a proper null hypothesis for the whole search, not just one rule. This is done using a bootstrap: resample the historical return data many times (using a method like the block bootstrap to preserve time-series structure), and for each resampled dataset, compute the performance of all the strategies in the original search, then record the best one. Doing this thousands of times builds up a distribution of "best-of-the-search performance under the null of no real skill." If the real, observed best strategy's performance falls comfortably within that simulated distribution, it's exactly the kind of result you'd expect from a large search of worthless rules — not evidence of genuine skill. If it falls well outside that distribution, into the extreme tail, that's real evidence the best strategy is doing something more than the others.

Worked example: is the winning rule special, or just lucky?

A researcher tests 200 technical trading rules against ten years of daily equity data and finds the best one produces an annualized excess return of 4%. Running White's Reality Check: the block bootstrap resamples the historical return series 1,000 times, and for each resample, all 200 rules are re-evaluated and the best one's excess return is recorded. Suppose this produces a distribution where the 95th percentile of "best-of-200-under-the-null" excess returns is 3.8%. The observed 4% sits just above that threshold — evidence, though not overwhelming, that the winning rule outperforms what pure search luck alone would produce across 200 worthless rules. If instead the 95th percentile of the simulated distribution had been 5.5%, the observed 4% would sit comfortably inside the range expected from chance alone, and the "winning" rule would carry no real statistical backing despite its impressive-looking backtest.

What this means in practice

White's Reality Check requires the researcher to keep and re-test every rule tried, not just the winner, which is exactly why it's a strong defense against data snooping — it's much harder to quietly understate the search size when the test formally requires simulating the full search. It's most useful at the end of a large systematic search over many trading rules or factor definitions, as the final statistical gate before a strategy is taken seriously. Related refinements like Hansen's SPA test and Romano-Wolf stepwise testing address some technical weaknesses in the original method, but the core logic — simulate the whole search under the null, not just the winner — is shared across all of them.