False Discovery Rate (FDR)

Run one test and a small p-value is reassuring. Run ten thousand, screening every factor, feature, and parameter you can dream up, and a pile of small p-values is guaranteed even if nothing works, because at $5\%$ you expect one false alarm in twenty. The false discovery rate reframes the problem in the only way that scales: instead of asking "am I certain none of my winners is fake?", it asks "of the winners I declare, what fraction should I expect to be junk?" and lets you cap that fraction at a level you can live with.

What FDR is

Sort your tests into the ones you rejected (called discoveries) and the rest. Among the discoveries, some are real and some are false. Write $R$ for the total number you rejected and $V$ for how many of those were actually null (false discoveries). The false discovery rate is the expected share of false ones:

$$\text{FDR} = \mathbb{E}\!\left[\frac{V}{R}\right] .$$

Controlling FDR at, say, $10\%$ is a statement of research budget: "I accept that about one in ten of the signals I greenlight will turn out to be noise." That is a very different, and usually more useful, promise than the family-wise error rate (FWER), which controls the chance of even one false discovery and is punishingly strict when you test thousands of things.

The Benjamini–Hochberg procedure

The recipe that controls FDR is simple enough to do by hand. Take your $m$ p-values and sort them ascending: $p_{(1)} \le p_{(2)} \le \dots \le p_{(m)}$. Pick your budget $\alpha$. Find the largest rank $k$ for which

$$p_{(k)} \le \frac{k}{m}\,\alpha ,$$

then reject every hypothesis from rank $1$ through $k$. The genius is that the threshold adapts: when many effects are real, lots of small p-values pile up and the bar $\tfrac{k}{m}\alpha$ climbs, letting more through; when nothing is real, the bar stays tiny and BH collapses to roughly the strict Bonferroni rule. Under independence (and many kinds of positive dependence) this holds FDR at $\alpha$.

Worked example

You test $m = 10$ strategies at a budget of $\alpha = 0.10$. Sort the p-values and compute each threshold $\tfrac{k}{m}\alpha = k \times 0.01$:

Rank $k$	$p_{(k)}$	Threshold $\tfrac{k}{10}(0.10)$	Pass?
1	0.001	0.010	yes
2	0.008	0.020	yes
3	0.012	0.030	yes
4	0.021	0.040	yes
5	0.030	0.050	yes
6	0.110	0.060	no
7	0.240	0.070	no
8	0.350	0.080	no
9	0.600	0.090	no
10	0.810	0.100	no

The largest rank that passes is $k = 5$ ($0.030 \le 0.050$), so you reject the smallest five, and you may reject rank 5 even though a later individual p-value would have failed on its own line. Of those five discoveries, you expect about $0.10 \times 5 = 0.5$ to be false, i.e. probably all five are real. Compare with Bonferroni, which uses a flat cutoff of $0.10/10 = 0.010$ and would keep only the top two. BH is more powerful precisely because its threshold rises with rank.

Where it bites in quant research

• The denominator is everything you tried. The $m$ that matters is the number of attempts, including the parameters you swept, the universes you sliced, and the ideas you quietly abandoned. Reported $m$ is almost always an undercount, so even BH under-corrects. This is the same hidden-multiplicity trap behind Backtest Overfitting.
• Correlated tests. Strategy tests share market beta and overlapping signals, violating independence. Plain BH may need the more conservative Benjamini–Yekutieli variant when dependence is strong and negative.
• FDR is not FWER. A $10\%$ FDR expects false discoveries, that is the deal. If a single false positive is catastrophic (deploying real capital on a fake edge), you may want the stricter FWER control despite its loss of power.

FDR is the practical successor to the FWER story told in p-values and multiple testing; it underpins how careful shops screen experiments and factors, and it is the same spirit as the deflated Sharpe ratio, which discounts a backtest's headline number for the number of trials behind it.