p-Hacking and the Garden of Forking Paths

Somebody shows you a strategy with a stellar backtest and a t-statistic of 3.2. It looks unarguable. Six months of live trading later it has made nothing. This story repeats so reliably across finance, medicine and psychology that it needs its own name, and the cause is almost never fraud. It is search. Somewhere between the raw data and the final chart, a great many choices were made, and the result you are being shown is the one that survived them.

The tipster scam

A con artist emails 1,024 strangers. Half are told a stock will rise this week; half are told it will fall. Next week he drops everyone who got the wrong call and splits the remaining 512 the same way. After ten rounds, one person has received ten correct predictions in a row and is ready to wire their savings to a man with no skill whatsoever.

Nobody lied at any point. Every individual email was an honest guess. The magic came entirely from showing you only the survivors. In research you play both roles at once: you generate the forecasts and you select the winner, and because you never see the discarded branches, the survivor looks miraculous.

What noise actually looks like

Before any maths, look at what pure randomness delivers when you test it twenty times. Below are twenty strategies with genuinely zero edge — every bar is noise. The dashed lines mark the conventional significance cutoff.

Putting a number on it

Write $\alpha$ for the false-alarm rate of a single test, conventionally 0.05, and $m$ for the number of tests you ran. If nothing is real, each test independently fails to raise a false alarm with probability $1 - \alpha$, so all $m$ stay quiet with probability $(1-\alpha)^m$. Therefore

$$P(\text{at least one false positive}) = 1 - (1-\alpha)^m .$$

In words: the chance of being fooled at least once climbs quickly with the number of things you tried, and it climbs whether or not you meant to try them.

Worked example one: twenty strategies, no edge

Twenty candidate signals, all worthless, each tested at the 5% level.

Step 1 — chance a single test stays quiet: $1 - 0.05 = 0.95$.

Step 2 — chance all twenty stay quiet: $0.95^{20}$. Taking logs, $20 \times \ln(0.95) = 20 \times (-0.0513) = -1.026$, and $e^{-1.026} = 0.359$.

Step 3 — chance at least one fires:

$$1 - 0.359 = 0.641 .$$

About a 64% chance of finding a "significant" strategy among twenty pieces of pure noise. The expected count of false positives is simply $20 \times 0.05 = 1$ — you should expect one winner every time you run a sweep of twenty. Push to 100 candidates and the chance of at least one false positive is $1 - 0.95^{100} = 99.4\%$. At that point a significant result is not evidence; it is arithmetic.

Worked example two: pricing a t-statistic of 3.2

Now the harder case. Your best strategy has $t = 3.2$, whose one-off p-value is about 0.0014 — apparently overwhelming. But you built it by sweeping 200 variations of a signal.

Step 1 — the crude correction. Bonferroni multiplies the p-value by the number of tests:

$$p_{\text{adjusted}} = 0.0014 \times 200 = 0.28 .$$

In words: to survive 200 attempts at the 5% level, a result must be strong enough that it would have looked significant at the 0.025% level on its own. Yours is nowhere close.

Step 2 — the sharper check. Ask what the best of 200 worthless strategies typically scores. For $m$ independent standard-normal draws the largest is close to

$$\mathbb{E}[\max] \approx \sqrt{2\ln m} .$$

In words: the champion of a large search grows, but only with the square root of the log of how many things you tried — slowly, yet relentlessly. With $m = 200$:

$$\sqrt{2 \times \ln 200} = \sqrt{2 \times 5.298} = \sqrt{10.60} = 3.26 .$$

Step 3 — compare. A search over 200 pure-noise strategies produces a champion with $t \approx 3.26$ on average. Yours scored 3.2. Your best strategy is, if anything, slightly worse than what you would expect from searching a shelf of random numbers.

Redraw the paths above a few times. Every one is a pure random walk with no drift at all, yet in most batches at least one climbs steadily enough to pass as a live track record. Pick that path afterwards and you have "found" a strategy.

The forking paths: it happens even with one test

Here is the part that catches careful people. Suppose you ran exactly one regression and reported it faithfully. Are you safe? Only if you would have run that same regression no matter what the data looked like.

You almost certainly would not. If the effect had been stronger among small caps you would have split by size. If the first year looked odd you would have trimmed it. If monthly rebalancing had disappointed you would have tried weekly. Each of those is a fork in a garden of paths, and you walked the one the data suggested. The number of tests you effectively ran is the number of paths you would have been willing to take — not the number you actually typed.

Count a routine equity study: 3 choices of universe, 4 holding periods, 3 ways of defining the signal, 2 weighting schemes, 2 outlier rules. That is $3\times4\times3\times2\times2 = 144$ paths. Even a scrupulously honest analyst who ran one of them has, in effect, searched a garden of 144, because the data guided every turn.

What this means in practice

Decide the analysis before you see the outcome. Write down the universe, the horizon, the signal definition and the test — then run it once. This is what pre-registration does in clinical trials and what a locked-down research protocol does on a good quant desk. It costs nothing except the freedom to change your mind after peeking, which is precisely the freedom that causes the damage.

Keep an honest count of the search. Every variation you fit, including the ones you abandoned in an afternoon, belongs in $m$. Then raise your bar accordingly: False Discovery Rate (FDR) control if you want a portfolio of discoveries with a known contamination rate, Bonferroni-style corrections if you need every single claim to hold, or a The Deflated Sharpe Ratio if the object is a trading strategy. Harvey, Liu and Zhu argued that after accounting for decades of published factor searching, the credible cutoff in asset pricing is roughly $t > 3$, not $t > 2$.

Finally, hold back data you never look at. An out-of-sample period, untouched until the very end, is the only cheap defence that survives contact with a determined researcher — yourself included. See Backtest Overfitting for how quickly that defence erodes if you peek more than once.

The formal machinery for all of this lives in p-values and Multiple Testing, and its trading-specific form in Data-Snooping Bias.