White's Reality Check and the SPA Test

A vendor shows you a backtest of "the" trading strategy with a Sharpe ratio of 1.8 over five years. What they don't mention is that it was the single best performer out of 500 rule variations they tested and quietly discarded. A standard t-test on that one strategy's returns would happily report significance, because it has no way of knowing 499 other candidates were tried and thrown away first. White's Reality Check and Hanson's SPA (Superior Predictive Ability) test exist specifically to answer the question a plain significance test can't: given that you searched over many candidate strategies, is the best one you found still better than a benchmark by more than chance alone would produce?

An analogy: the best of a thousand coin-flippers

Imagine a thousand people each flip a fair coin ten times, and you report only the best performer — someone who happened to get 9 heads out of 10. Judged in isolation, "9 out of 10 heads" looks like powerful evidence the coin is biased. But you already know it's fair; you just selected the luckiest performer out of a thousand tries, and with that many attempts, someone getting 9 heads by pure chance is nearly guaranteed. The Reality Check and SPA test do the equivalent for trading strategies: instead of asking "does this one strategy beat the benchmark," they ask "given that we tried this many strategies, how extreme would the best one need to be before we'd stop suspecting it's just this search's version of the lucky coin-flipper?"

The mechanics, in words

Both tests build a bootstrap distribution of what the best-of-many result would look like under the null hypothesis that none of the strategies actually beat the benchmark. Concretely: resample the historical return data (with replacement, respecting time dependence) many times, recompute every candidate strategy's performance against the benchmark on each resampled dataset, and record the best performance across all candidates on that resample. Doing this thousands of times builds a distribution of "what the best-of-$N$ result looks like purely from luck, when none of the strategies actually work." The observed best strategy's real performance is then compared against that distribution, not against a single-strategy null. In plain English: instead of asking whether your winning number is impressive on its own, the test asks whether it's impressive given how many numbers you rolled to get it.

The SPA test refines White's original Reality Check by giving poorly-performing candidate strategies less weight in constructing the null distribution, which makes the test more powerful — it doesn't get needlessly conservative just because your search also included some obviously bad candidates you never seriously considered.

Worked example: 200 strategies, one survivor

A researcher backtests 200 momentum-rule variants (different lookback windows and thresholds) against a benchmark, and the best one shows an annualized excess return of 4.2%. A plain t-test on that one strategy's returns gives a p-value of 0.02 — seemingly significant. Running White's Reality Check instead: bootstrap resample the return history 2,000 times, and on each resample recompute all 200 strategies' excess returns, keeping only the best one each time. Suppose that bootstrap distribution of "best-of-200 under the null" has a mean around 3.8% with substantial spread, and the observed 4.2% only exceeds about 40% of those bootstrap draws. The reality-check p-value is roughly $1 - 0.40 = 0.60$ — nowhere near significant. The strategy's apparent edge is well within what 200 random tries would be expected to produce, even if none of them had any genuine skill.

What this means in practice

Any time a strategy, factor, or model is the survivor of a search over many candidates — different parameters, different lookbacks, different feature sets — a plain significance test on that survivor is close to meaningless, because it ignores the search itself. These tests give a principled way to ask "would this result have been unremarkable if we'd searched this hard over strategies with zero real skill?" They require knowing (or reasonably bounding) how many candidates were actually tried, including the ones quietly discarded — which is exactly the number a vendor's marketing backtest is least likely to disclose.