What fraction of your discoveries turned out fake?

Question

A signal-research team screens a large set of candidates and flags 40 as "significant." A later out-of-sample validation reveals that 6 of those 40 were actually worthless (false discoveries).

What fraction of the team's discoveries were false, and how does this quantity differ from a per-test error rate like 5%?

Accepted Answer

Count the proportion. The false discovery proportion is simply the number of false discoveries divided by the total number of discoveries: 	ext{FDP} = \frac{	ext{false discoveries}}{	ext{total discoveries}} = \frac{6}{40} = 0.15, so 15% of the flagged signals were fake. Why this is a different animal from 5%. A per-test significance level like 5% answers "how often does a truly null test fire?" That is a rate among the nulls. The false discovery proportion answers a very different question: "of the things I actually flagged, what share are wrong?" One looks at the pool of nulls, the other …

What fraction of your discoveries turned out fake?

Your answer

More Statistics questions