Should you control FWER or FDR for 1,000 signals?

Question

A quant screens 1,000 candidate signals. Applying Bonferroni (threshold $0.05/1000 = 0.00005$ ), essentially nothing survives, even signals with genuine promise. A colleague says "just use FDR instead."

Explain the difference between controlling the family-wise error rate and the false discovery rate, and which is appropriate for a large screening problem.

Accepted Answer

Two different things to control. - Family-wise error rate (FWER): the probability of making even one false positive across the entire family. Bonferroni and Šidák control this. It's the right notion when a single false positive is costly or embarrassing, a confirmatory drug trial, a published claim you'll stake your reputation on. - False discovery rate (FDR): the expected proportion of false positives among the signals you flag. Benjamini–Hochberg controls this. It's the right notion when you're screening many candidates, expect several real ones, and can tolerate that some fraction of your …

Should you control FWER or FDR for 1,000 signals?

Your answer

More Statistics questions