Data-Snooping Bias

Data-snooping bias (also called data dredging or p-hacking) is the distortion that arises when the same dataset is used to both generate and test many hypotheses. Each test carries a chance of a false positive; run enough tests and a false positive is not a risk but a certainty. In quant research the "tests" are strategy variants, lookbacks, thresholds, universes, factor definitions, and the dataset is a fixed, finite history that gets reused thousands of times. The best-looking backtest is then simply the maximum of many noisy numbers, and its performance is a statement about luck, not edge.

The multiple-testing arithmetic

Test a single true-null hypothesis at significance $\alpha$ and you accept a false-positive probability of $\alpha$ (say $0.05$). Test $N$ independent true nulls and the probability of at least one false positive, the family-wise error rate (FWER), is

$$\text{FWER} = 1 - (1-\alpha)^N \;\approx\; N\alpha \quad (\text{small } \alpha).$$

At $\alpha = 0.05$, just $N=14$ independent tests push the chance of a spurious "significant" result above $50\%$; at $N=100$ it is $99.4\%$. The expected number of false discoveries is simply $N\alpha$, test 1,000 worthless signals and expect ~50 to clear a 5% bar by chance alone. The one you present is, almost by definition, the luckiest of the batch.

Corrections that raise the bar

Bonferroni (controls FWER). To hold the family-wise error at $\alpha$, test each hypothesis at $\alpha/N$:

$$\alpha_{\text{Bonf}} = \frac{\alpha}{N}.$$

Simple and conservative; it assumes the worst-case dependence and can be overly strict when tests are correlated.

Benjamini–Hochberg (controls FDR). The false discovery rate is the expected fraction of discoveries that are false, $\text{FDR} = \mathbb{E}[V/R]$ for $V$ false and $R$ total rejections. Order the $p$-values $p_{(1)} \le \dots \le p_{(N)}$ and reject the largest $k$ with $p_{(k)} \le \frac{k}{N}\alpha$. FDR control is less conservative than FWER, appropriate when you are willing to tolerate a known false-discovery fraction among many signals.

The Sharpe / t-stat implication. Harvey, Liu & Zhu catalogued hundreds of published "factors" and argued that, given how many have been tried across the literature, the conventional $t > 2.0$ bar is far too low. Accounting for multiple testing, they propose a threshold closer to $t > 3.0$ (a haircut that scales with the number of trials). Equivalently, an observed Sharpe must be deflated for the number of configurations tested before it can be believed, the logic formalized in the The Deflated Sharpe Ratio.

Worked example

You test 200 variations of a momentum signal on 20 years of data. Under the null that none has real edge, the in-sample $t$-statistics are approximately standard normal. The expected maximum of $N$ independent standard normals grows like

$$\mathbb{E}[\max_N Z] \approx \sqrt{2\ln N}.$$

For $N = 200$, $\sqrt{2\ln 200} \approx \sqrt{10.6} \approx 3.26$. So even with zero true edge anywhere, you expect your best variant to post a $t$-stat above $3.2$, comfortably past the naive $t>2$ bar and even past a casual $t>3$ bar. Reporting that best variant as a discovery, without disclosing the 199 you discarded, is data snooping. The correct question is not "is $t=3.2$ significant?" but "is $t=3.2$ surprising given that I took the max of 200 tries?", and the answer is no.

Why finance is especially vulnerable

• The dataset is fixed and small. There is only one realized history of markets; you cannot collect an independent sample, so the same data is reused endlessly.
• Trials are hidden. The count $N$ includes every idea you, your team, and the entire published literature ever tried on this data, most are never recorded, so the effective $N$ is far larger than the variants in your final notebook.
• Correlated tests. Strategy variants overlap, so tests are dependent; Bonferroni over-corrects while the effective number of independent trials is hard to pin down.
• Incentives. Publication and allocation reward positive results, selecting for the luckiest backtests, this is the engine behind Backtest Overfitting.

Failure modes

• Underreporting trials. The single most damaging omission: presenting the best variant as if it were the only one tested.
• Garden of forking paths. Even without an explicit grid search, flexible choices made after seeing the data (which universe, which period, which cost model) constitute implicit multiple testing.
• Reusing the "holdout." Peeking at the test set, adjusting, and re-testing turns the holdout into training data.
• Ignoring the literature's trials. Your "new" factor was likely tried by others on the same history; the community's cumulative $N$ is enormous.

In interviews

The canonical question: "If you test 100 strategies at the 5% level, how many do you expect to look significant by chance?", five, and the probability that at least one does is $1-0.95^{100} \approx 99\%$. Be ready to state Bonferroni ($\alpha/N$) versus FDR (Benjamini–Hochberg), to explain the difference between controlling FWER and FDR, and to produce the $\mathbb{E}[\max_N Z]\approx\sqrt{2\ln N}$ intuition for why the best of many backtests is inflated. The mature point to land: the right defense is to count your trials honestly and deflate accordingly, see The Deflated Sharpe Ratio and Backtest Overfitting.