Bonferroni and Holm Corrections

Test one strategy at the standard 5% significance threshold and you accept a 5% chance of calling noise a discovery. Test a hundred strategies at that same threshold and, even if every single one is worthless, you expect about five of them to look "significant" purely by luck. Bonferroni and Holm corrections are two different ways to fix the threshold itself so that the overall chance of any false alarm across all your tests stays at 5%, not the chance for each test individually.

An analogy: splitting a fire alarm budget across many rooms

A building has a fixed tolerance for false fire alarms — say, no more than a 5% chance of any alarm going off with no actual fire, across the whole building, all rooms combined. Install one super-sensitive detector per room, each individually tuned to a 5% false-alarm rate, and with a hundred rooms you will get several false alarms most nights — the individual budgets add up. Bonferroni's fix is blunt but safe: give each of the hundred detectors only $5\%/100 = 0.05\%$ of the false-alarm budget, so the sum across all rooms stays capped at 5%. Holm's fix is smarter: don't split the budget evenly upfront — check the least suspicious rooms with a generous budget first, and only tighten the screws for the rooms that already look most alarming.

Bonferroni, one symbol at a time

Testing $m$ hypotheses, each normally judged against a per-test significance level $\alpha$ (commonly $0.05$), Bonferroni replaces $\alpha$ with a stricter threshold for every single test:

$$\alpha_{\text{Bonferroni}} = \frac{\alpha}{m}$$

In plain English: divide your total tolerance for a false alarm evenly across all $m$ tests, and only call a result significant if its $p$-value beats this much smaller bar. Testing $m=100$ strategies at overall $\alpha=0.05$ means each individual strategy needs $p < 0.0005$ to count, not $p<0.05$. This guarantees the family-wise error rate — the chance of at least one false positive anywhere in the batch — stays at or below $\alpha$, no matter how the tests are correlated.

Holm's step-down improvement, one symbol at a time

Bonferroni treats every test identically and is unnecessarily strict on the tests with the smallest $p$-values. Holm's procedure sorts the $p$-values from smallest to largest, $p_{(1)} \le p_{(2)} \le \cdots \le p_{(m)}$, and compares each one to a different, gradually loosening threshold:

$$p_{(i)} < \frac{\alpha}{m - i + 1}$$

In plain English: the most significant result faces the strictest bar (same as Bonferroni's), but each subsequent result, in order, gets an easier bar, because fewer tests remain "in play" once earlier ones have already been confirmed. You walk down the sorted list and stop rejecting the instant one test fails its threshold — every test after that point is declared not significant automatically, regardless of its own $p$-value.

Picture the curve above as the null distribution every one of your $m$ test statistics would follow if nothing real were going on. Bonferroni pushes the rejection cutoff far into the tail for every test simultaneously; Holm pushes it just as far for the single most extreme test, then relaxes it step by step as it works inward.

Worked example 1: Bonferroni on five backtested strategies

Five strategies are backtested, giving $p$-values $0.001, 0.012, 0.018, 0.04, 0.09$. At the raw $\alpha = 0.05$, four of five look "significant." Apply Bonferroni with $m=5$: the adjusted threshold is $0.05/5 = 0.01$. Comparing: $0.001 < 0.01$ (significant), $0.012 > 0.01$ (not), $0.018 > 0.01$ (not), $0.04 > 0.01$ (not), $0.09 > 0.01$ (not). Only one of the five survives — the other three "wins" evaporate once you account for having tried five things.

Worked example 2: Holm on the same five strategies

Sort the $p$-values: $p_{(1)}=0.001, p_{(2)}=0.012, p_{(3)}=0.018, p_{(4)}=0.04, p_{(5)}=0.09$, with $m=5$. Compare each in turn:

• $i=1$: threshold $0.05/(5-1+1) = 0.05/5 = 0.01$. Is $0.001 < 0.01$? Yes — reject, continue.
• $i=2$: threshold $0.05/(5-2+1) = 0.05/4 = 0.0125$. Is $0.012 < 0.0125$? Yes — reject, continue.
• $i=3$: threshold $0.05/(5-3+1) = 0.05/3 = 0.0167$. Is $0.018 < 0.0167$? No — stop here.

Tests 3, 4, and 5 are automatically declared not significant, regardless of their own $p$-values. Net result: Holm finds two significant strategies where Bonferroni found only one, because it did not force the second-best result to clear the full $0.01$ bar — it only had to clear $0.0125$.

What this means in practice

Anywhere a researcher tests many signals, parameter combinations, or strategy variants against history, some correction is mandatory before trusting a "significant" backtest — this is the statistical core of why multiple-testing hurdles exist in quant research. A firm that backtests a thousand variants of a strategy and reports the single best Sharpe ratio without correction is, in effect, running the fire-alarm-budget problem with a thousand rooms and no correction at all — false "discoveries" are not just likely, they are close to guaranteed.