Five fakes among twenty-five discoveries

Question

A research group publishes 25 "significant" findings. A replication effort later shows that 5 of them do not hold up (they were false discoveries).

What fraction of the group's findings were false, and what does this say that a per-study 5% level does not?

Accepted Answer

Compute the proportion. The false discovery proportion divides false discoveries by total discoveries: 	ext{FDP} = \frac{5}{25} = 0.20, so 20% of the published findings were false. Why 5% did not protect them. Each study may have used an honest 5% significance level, but that number only bounds how often a truly null hypothesis fires. It says nothing about what fraction of the reported findings are wrong. When many hypotheses are tested and mostly-null ones dominate, the share of false results among discoveries can be far above 5%, here it is 20%. The right control. If the group wanted to gu…

Five fakes among twenty-five discoveries

Your answer

More Statistics questions