White's Reality Check: How to Tell If Your Best Strategy Is Just the Luckiest One

You have built a hundred trading rules. You have backtested all of them. The best one beat buy-and-hold by a comfortable margin.

Is it any good?

The naive way to answer this is to take the winner and run a significance test on it, as though it were the only rule you ever considered. This is wrong, and it is wrong in a way that will cost you real money, because the winner of a hundred-rule contest was selected for being lucky. Testing it as if it arrived unbidden from heaven is like testing whether the tallest person in a crowd of a thousand is unusually tall by comparing them to the average person. Of course they are. That is why you noticed them.

In 2000, Halbert White published a procedure that solves this properly. He called it the Reality Check, and the name is doing some work.

The problem: the winner's curse, formalised

The technical name for this is data snooping, and White's framing of it is precise.

The mistake is not that you searched. Searching is fine. Searching is what research is. The mistake is testing the winner of a search using a statistical test that assumes no search happened. The distribution of "the best result out of a hundred" is a very different animal from the distribution of "the result of one attempt," and if you use the wrong distribution you will call noise a discovery, reliably, forever.

What you actually need to know is this: if every single one of my hundred rules were worthless, how good would the best of them look anyway? That is your true benchmark. Only if your observed champion beats that do you have evidence of anything.

The obstacle is that this is a hard distribution to get at analytically, because your hundred rules are not independent. They are all trading the same underlying market, most of them are variations on each other, and their returns are tangled together in ways no formula is going to capture.

The key idea via analogy: run the tournament again, in a world with no talent

White's solution is to stop trying to derive the distribution and instead simulate it. And the way he simulates it is elegant.

Here is the intuition. You want to know what your best-of-a-hundred result would look like in a world where none of the rules had any skill. So build that world, out of your own data.

The tool is the bootstrap: you resample your historical return series, in blocks (to preserve the fact that financial returns are not independent from one day to the next), to create an artificial history that has the same statistical texture as the real one. Crucially, you do this in a way that strips out any genuine outperformance, so that in this artificial world, by construction, no rule has a real edge.

Then you run all one hundred rules on the artificial history, and you record the best score.

Then you do it again. And again. Thousands of times.

What you end up with is a distribution: "here is what the best-of-a-hundred score looks like when nobody has any skill." Now you take your real champion's score and see where it falls in that distribution. If your real winner scored better than, say, 99% of the fake winners, you have evidence. If it lands in the middle of the pack, you have discovered nothing, and you have been very close to funding a coin flip.

The analogy: you are running your tournament again and again in a world where every competitor is a fraud, and asking how impressive the winner of that tournament tends to look. Your real winner has to beat those fakes, not the average competitor.

The critical design choice, and the reason the procedure works, is that all one hundred rules are resampled together, on the same artificial histories. This automatically preserves all the messy correlations between them. If ninety of your rules are basically the same rule with different parameters, the bootstrap knows, because they will move together in the fake worlds just as they do in the real one. You never have to model that correlation. It comes along for free.

Why it mattered

• It gave data snooping a cure, not just a name. People had complained about data mining in finance for years. White handed them a procedure that produces a p-value you can defend.
• It was immediately turned on the industry's sacred cows. Within a year, Sullivan, Timmermann and White applied the Reality Check to the entire universe of technical trading rules on a century of Dow data, and the results were sobering.
• It taught the profession to think in terms of the whole search space. The deepest contribution is conceptual: your unit of analysis is not the strategy you liked, it is the universe of strategies you could have liked. That reframing underlies everything that came after, including the Deflated Sharpe Ratio and the multiple-testing corrections in the factor literature.
• It made the bootstrap a standard quant tool. After White, resampling stopped being an exotic statistician's toy and became a normal part of a quant's kit.

The honest limitations

• The universe of rules must be specified in advance, and honestly. The procedure corrects for the rules you tell it about. If you spent six months developing intuition and then present the Reality Check with the fifty rules that survived that intuition, you have hidden the real search from the test, and your p-value is a fiction.
• It has a known power problem. This is the most substantive criticism, and Peter Hansen made it forcefully five years later. If you pad your universe with a lot of terrible rules, the Reality Check gets less likely to find a genuinely good one. The bad rules drag the benchmark around and swamp the signal. That is perverse: adding junk to your search should not make it harder to detect the gold. Hansen's Superior Predictive Ability test was built specifically to fix this.
• The bootstrap has to be chosen with care. Financial returns are serially dependent, volatility clusters, and regimes shift. If your resampling scheme does not respect that structure, your artificial worlds do not resemble the real one and the whole exercise is worthless. The choice of block length is not innocent.
• Passing does not mean profitable. The Reality Check answers a narrow statistical question: did the best rule beat the benchmark by more than luck could explain? It says nothing about transaction costs, capacity, or whether the rule will keep working now that you have published it.
• It is a test of one search, not of a career. Run the Reality Check on ten different rule universes and pick the one that passes, and you have just data-snooped the Reality Check.

The one-line takeaway

White's Reality Check answers the only question that matters after a strategy search: if none of my rules had any real skill, how good would the best of them have looked anyway? He answers it by bootstrapping the entire universe of rules together, thousands of times, and checking whether the real champion can beat the fake ones.