p-values and Multiple Testing

The p-value is the most used and most abused number in quantitative research. Used correctly it calibrates the surprise in a single test; used as most people use it, as "the probability the strategy is real", it is nonsense. And in a world where you can backtest ten thousand signals before lunch, the deeper problem is not any single p-value but the fact that testing many of them guarantees false discoveries. This concept is where careful statistics most directly protects (or destroys) trading capital.

What a p-value is

Fix a test statistic $T$ with a known distribution under the null $H_0$. The p-value is the probability, computed under $H_0$, of observing a statistic at least as extreme as the one you got:

$p = \mathbb{P}\big(\,|T| \ge |t_{\text{obs}}| \ \big|\ H_0\,\big).$

A useful fact: if $H_0$ is true and the test is continuous, the p-value is uniformly distributed on $[0,1]$. That is the entire basis of the framework, under the null, a p-value below 0.05 happens exactly 5% of the time, which is why rejecting when $p < \alpha$ controls the type I error at $\alpha$.

The five things a p-value is not

Small p-values feel like probabilities about hypotheses; they are not.

1. It is not $\mathbb{P}(H_0 \text{ true} \mid \text{data})$, that is a posterior probability and needs a prior (see Bayesian Inference).
2. It is not the probability the result was due to chance.
3. $1 - p$ is not the probability the alternative is true, nor the probability the finding will replicate.
4. A small $p$ does not measure effect size, with enough data a trivially small edge gets a tiny p-value. Report the confidence interval to see magnitude.
5. $p > 0.05$ does not prove the null; absence of evidence is not evidence of absence, especially given how underpowered financial tests are.

Why multiple testing breaks everything

Here is the mechanism that ruins naive backtesting. Suppose you test $m$ strategies that are all worthless (every null true), each at level $\alpha = 0.05$, with independent tests. The probability that at least one clears the bar by luck is the family-wise error rate:

$\text{FWER} = 1 - (1-\alpha)^m.$

For $m = 20$ this is $1 - 0.95^{20} \approx 0.64$; for $m = 100$ it is $0.994$. Test a hundred junk signals and you are all but guaranteed several "significant" ones. Expected false discoveries $= m\alpha$ grows linearly: $m = 10{,}000$ signals yields $\approx 500$ spurious "edges" at $\alpha = 0.05$. This is not a subtle bias; it is the dominant reason published anomalies and dazzling backtests fail out of sample.

Bonferroni: control the family-wise error rate

The simplest fix: to hold the probability of any false discovery at $\alpha$, test each hypothesis at $\alpha/m$. By the union bound (Boole's inequality),

$\mathbb{P}\Big(\textstyle\bigcup_j \{\text{reject } H_{0j}\}\Big) \le \sum_j \mathbb{P}(\text{reject }H_{0j}) = m\cdot \frac{\alpha}{m} = \alpha,$

so Bonferroni controls FWER with no independence assumption. Its cost is brutal conservatism: with $m = 1000$ you need $p < 0.00005$, decimating power and rejecting almost every real-but-modest edge. It answers "am I sure none of my discoveries is false," which is often stricter than a researcher actually needs.

Benjamini–Hochberg: control the false discovery rate

The more useful target in research is the false discovery rate, the expected fraction of your rejections that are false, $\text{FDR} = \mathbb{E}[V/R]$ where $R$ is the number rejected and $V$ the number falsely rejected. Controlling FDR at 10% says "I accept that about 10% of the signals I greenlight are junk," a rational research budget. The Benjamini–Hochberg procedure: sort the p-values ascending $p_{(1)} \le \dots \le p_{(m)}$, find the largest $k$ with

$p_{(k)} \le \frac{k}{m}\,\alpha,$

and reject $H_{(1)}, \dots, H_{(k)}$. This controls FDR at $\alpha$ under independence (and positive dependence). It is far more powerful than Bonferroni because the threshold adapts, when many effects are real, the bar relaxes; when nothing is real, it collapses to roughly Bonferroni. For a research pipeline scanning thousands of features, BH is the standard tool.

Worked example: 200 candidate factors

You test 200 candidate factors for return predictability at $\alpha = 0.05$. Naively you expect $200 \times 0.05 = 10$ false positives among truly useless factors. Suppose 30 factors cross $p < 0.05$. Bonferroni demands $p < 0.05/200 = 0.00025$; maybe 3 survive, you are confident in those but have thrown away every genuine weak factor. BH at FDR 10%: sort the 30 significant p-values, find the largest $k$ with $p_{(k)} \le (k/200)\times 0.10$; perhaps 12 survive, of which about 1 is expected to be false. The BH set is the sane middle ground, this is precisely the discipline Harvey, Liu & Zhu argue the asset-pricing literature failed to apply, leaving a "factor zoo" of anomalies that do not replicate.

Failure modes in financial data

• Hidden multiplicity. The $m$ that matters is the number of things you tried, including informal ones, every parameter you tuned, every universe you sliced, every lookback you swept. Reported $m$ is almost always an undercount, so even BH under-corrects.
• Dependence. Strategy tests are highly correlated (overlapping signals, common market beta), so the independence assumption behind FWER math is violated; Bonferroni stays valid (union bound needs no independence) but is even more conservative, while plain BH may need the Benjamini–Yekutieli variant.
• The garden of forking paths. Sequential, data-dependent choices inflate error rates in ways no post-hoc correction fully captures, the cure is out-of-sample and true hold-out testing, not a bigger denominator. See Overfitting and Walk-Forward Analysis.
• Publication/selection bias. You only see the backtests that worked; the denominator of attempts is invisible, so reported Sharpes are order-statistics maxima, not typical draws.

In interviews

Nail the definition, a p-value is the probability of data this extreme assuming the null is true, and be ready to reject the common misreadings (it is not the probability the null is true, not one minus the replication probability, not an effect size). The multiple-testing question is a favorite for quant research roles: derive $\text{FWER} = 1-(1-\alpha)^m$, explain that testing 20 signals gives you a ~64% chance of a false positive, and contrast Bonferroni (controls FWER, conservative, threshold $\alpha/m$) with Benjamini–Hochberg (controls FDR, adaptive, more powerful). The strongest close is connecting it to backtesting: the reason so many strategies die out of sample is uncorrected multiplicity, and the defense is FDR control plus genuine hold-out data.