Family-Wise Error Rate

A single hypothesis test run at the standard 5% significance level has a 5% chance of crying wolf when nothing is really there. That sounds tolerably safe — until you run twenty of them. Run twenty independent tests, each individually safe at 5%, and the chance that at least one cries wolf is no longer 5%; it is close to two in three. The family-wise error rate is the name for that combined, batch-level chance of a false alarm, and it is the number that actually matters whenever you test more than one thing.

An analogy: buying many lottery tickets, each with long odds

A single lottery ticket has a tiny chance of "winning" (matching numbers by pure chance, meaning nothing). Buy one ticket, and the chance of an accidental match is genuinely small. Buy a thousand tickets, though, and even though each individual ticket's odds are unchanged, the chance that at least one of the thousand matches by pure luck climbs dramatically. Nobody would call that thousand-ticket match "a real signal" — it is exactly what you'd expect from buying that many tickets. Testing a thousand trading signals against history and reporting the one that happened to look significant is the same lottery, dressed up in $p$-values.

The formula, one symbol at a time

Let $\alpha$ be the significance level for one test — the chance of a false positive on that test alone, commonly $0.05$. Suppose you run $m$ independent tests, each with no real effect behind it. The chance that a specific test is a false positive is $\alpha$, so the chance it is not a false positive is $1-\alpha$. The chance that none of the $m$ independent tests is a false positive is $(1-\alpha)^m$, and so the family-wise error rate — the chance of at least one false positive somewhere in the batch — is:

$$\text{FWER} = 1 - (1-\alpha)^m$$

In plain English: subtract from 1 the probability that every single test correctly stays quiet, and what's left is the probability that at least one of them falsely fires. As $m$ grows, $(1-\alpha)^m$ shrinks toward zero, so the FWER climbs toward 1 (certainty) — not because your tests got worse, but because you gave chance more opportunities to produce a fluke.

Each of the twenty independent yes/no trials above corresponds to one hypothesis test at $\alpha=0.05$, each with a 5% individual chance of a "success" (false alarm) under the null. The distribution's mass sitting on 1-or-more successes, not just its mean, is exactly what the FWER formula is quantifying.

Worked example 1: FWER for twenty backtested signals

A researcher tests twenty candidate trading signals against ten years of history, each at $\alpha = 0.05$, and finds that none of the underlying signals has any real edge (the honest, if depressing, truth). What is the chance at least one still looks "significant" by luck?

$$\text{FWER} = 1 - (1-0.05)^{20} = 1 - (0.95)^{20}$$

Compute $(0.95)^{20}$: $0.95^2 = 0.9025$, $0.95^4 = 0.9025^2 = 0.8145$, $0.95^8 = 0.8145^2 = 0.6634$, $0.95^{16} = 0.6634^2 = 0.4401$, and $0.95^{20} = 0.95^{16} \times 0.95^4 = 0.4401 \times 0.8145 = 0.3585$. So $\text{FWER} = 1 - 0.3585 = 0.6415$ — a 64% chance that at least one of the twenty worthless signals looks significant at the 5% level, purely by luck.

Worked example 2: how fast it climbs

Redo the calculation at a few batch sizes, same $\alpha=0.05$: at $m=1$, FWER $=0.05$ (the advertised rate). At $m=5$: $1-0.95^5=0.226$. At $m=14$: $1-0.95^{14}=0.512$ — already past a coin flip. At $m=100$: $0.95^{100}\approx0.0059$, so FWER $\approx0.994$ — a false alarm is nearly certain. The climb is not linear, since the "safe" probabilities multiply rather than add, but it is steep and relentless throughout.

What this means in practice

Quant research is, structurally, a multiple-testing machine: every parameter sweep, every universe of candidate signals, every grid search over lookback windows is a batch of tests run against the same historical data. A backtest engine reporting "$p < 0.05$" for the single best result out of a thousand tried is not reporting a 5% false-alarm rate — it is close to certain that at least one fluke would have cleared that bar. Controlling the FWER, typically via Bonferroni or Holm corrections, tightens the per-test bar so the combined risk across the whole search stays at the advertised level.