Accept Some Mistakes: Benjamini and Hochberg's False Discovery Rate

You have three hundred candidate trading signals. You test each one. Forty of them come back "statistically significant" at the usual one-in-twenty threshold.

How many should you believe?

Well, if all three hundred were pure noise, you would expect about fifteen of them to clear a one-in-twenty bar by chance alone. So among your forty winners, roughly fifteen are probably garbage. You have a discovery pile that is somewhere around a third fake, and you have no idea which third.

This is the multiple testing problem, and by 1995 statisticians had a solution for it. The trouble was that the solution was so severe that nobody could use it.

The problem: the old fix was too strict to be useful

The classical answer is the Bonferroni correction, and it is beautifully simple: if you are running three hundred tests and you want to keep the chance of making even one false discovery below five percent, then require each individual test to clear a bar of five percent divided by three hundred. Which is roughly one in six thousand.

That works. It absolutely controls false positives. It also means that a genuinely good signal, a real one, with a modest effect size, will almost certainly fail to clear the bar. You have protected yourself from error by making discovery nearly impossible.

Statisticians call this a loss of power: the ability of a test to detect a real effect when one exists. Bonferroni trades away nearly all of your power in exchange for a guarantee most people do not actually need.

And here is the key observation Benjamini and Hochberg made: do you really need that guarantee?

The Bonferroni correction controls the probability of making any false discovery at all. It treats one false positive out of forty findings as exactly as catastrophic as forty false positives out of forty. But in most real research, that is not how you feel about it. If you screen three hundred signals and hand back forty, and two of them turn out to be duds, you have done a good job. If thirty-eight of them are duds, you have done a terrible job. The distinction that matters is not "was I ever wrong" but "what fraction of what I told you is wrong."

That is the entire insight, and it is one of those reframings that seems obvious the second it is stated and took the field decades to see.

The key idea via analogy: the quality of the shortlist

Imagine you are a recruiter screening ten thousand applications and forwarding a shortlist to the hiring manager.

The Bonferroni mindset says: never forward a single unqualified candidate. To guarantee that, you would have to set the bar so high that you would forward almost nobody, including most of the genuinely brilliant applicants. The hiring manager gets a shortlist of one name and complains, rightly, that you are useless.

The Benjamini-Hochberg mindset says: make sure that no more than one in twenty of the people on my shortlist is unqualified. You accept in advance that a few duds will slip through. In exchange, you can lower the bar enough to actually find the good candidates. The hiring manager gets a shortlist of two hundred names, roughly ten of which are mistakes, and is delighted, because the alternative was nothing.

That fraction, the proportion of your declared discoveries that are actually false, is what Benjamini and Hochberg named the False Discovery Rate. And their paper does two things: it defines it, and it gives you a startlingly simple procedure that provably controls it.

The procedure, which you could do on paper

The mechanics are so easy they feel like they cannot possibly be a landmark statistical result.

Run all your tests and collect the p-values.
Sort them from smallest to largest.
Walk down the sorted list. Compare each p-value against a moving threshold that gets more generous as you go down the list. The first p-value is compared against a very strict bar, the second against a slightly looser one, and so on, with the bar rising in proportion to how far down the list you are.
Find the last p-value that still beats its threshold.
Declare everything from the top of the list down to that point a discovery. Reject the rest.

The genius is in step 3. Bonferroni holds every test to the same brutally strict bar. Benjamini and Hochberg let the bar adapt to the evidence in the data itself. If you have a lot of very small p-values clustered at the top of your list, that is evidence that real effects exist in your data, and the procedure responds by relaxing, letting more discoveries through. If your p-values look like a uniform smear (exactly what you would see if nothing were real), the procedure stays strict and lets almost nothing through.

The data tells the procedure how generous to be. That is why it is so much more powerful than Bonferroni, and it is why it works.

Why it mattered

It made large-scale screening possible. Genomics arrived in the late 1990s with experiments that tested twenty thousand genes at once. Bonferroni would have found nothing. Benjamini-Hochberg is the reason that field has results at all. The same applies to neuroimaging, to clinical screening, and eventually to finance.
It reached quantitative finance through the factor zoo. When Harvey, Liu and Zhu confronted the hundreds of published return-predicting factors, the false discovery rate was one of the central tools they used to work out how many were real. If you want to screen three hundred signals and keep the ones worth trading, the FDR framework is the right instrument, precisely because it accepts that some of your keepers will be junk.
It changed how researchers think about error. The shift from "never be wrong" to "be wrong at a controlled rate" is a genuinely different philosophy of discovery, and it is the right one whenever you are screening rather than confirming.
It is trivially easy to implement. Sort, compare against a rising line, cut. This matters. A brilliant method that requires a specialist never gets used. This one is a few lines of code, which is why it is everywhere.

The honest limitations

You are knowingly accepting false discoveries. This is not a bug, it is the deal. But it means the output of a Benjamini-Hochberg screen is a pile of candidates, not a set of truths. If you treat every signal that survives an FDR screen as a real, tradeable edge, you will be allocating capital to noise, on purpose, at a known rate.
The original proof assumed independent tests. Real trading signals are wildly correlated with each other. Later work extended the procedure to handle certain kinds of dependence, but the extensions are more conservative, and heavily correlated tests remain a genuinely awkward case.
It says nothing about which discoveries are the fake ones. You know roughly what fraction of your shortlist is wrong. You have no idea which entries. In finance, where you have to actually pick which signals to fund, that is a painful gap.
The FDR level is arbitrary. Why five percent? Why not one, or twenty? The choice depends on how much a false discovery costs you relative to a missed one, and nobody in finance has a principled way to price that trade-off.
It corrects for the tests you ran, not the ones you thought about. Like every multiple-testing procedure, it can only account for the search you declare. The features you engineered because a colleague's earlier result made them seem promising are part of the search too, and no procedure can see them.

The one-line takeaway

Benjamini and Hochberg replaced the impossible goal of "never make a single false discovery" with the achievable one of "keep the fraction of my discoveries that are false below a level I choose", and gave a procedure so simple you could run it by hand that lets you screen thousands of hypotheses and still find the real ones.