What fraction of your "significant" results are actually true?

Question

You run tests at \alpha = 5\% with 80% power. A junior colleague reasons: "So only 5% of my significant findings are false positives." That's wrong. Compute the actual fraction of significant findings that are true (the positive predictive value) when most tested hypotheses are false, and explain the role of the base rate.

Accepted Answer

Why 5% is the wrong number. \alpha = 5\% is the false-positive rate among true nulls, the chance a worthless hypothesis trips the test. It is not the chance that a flagged result is worthless. To get that, you need the base rate: what fraction of the hypotheses you test are real to begin with. Work an example. Suppose 10% of the hypotheses you test have a real effect (a generous rate for blind data mining). Out of 1,000 tests: - 100 are true, 900 are null. - True positives: 0.80 	imes 100 = 80 (power catches 80% of the real ones). - False positives: 0.05 	imes 900 = 45 (5% of the nulls slip…

What fraction of your "significant" results are actually true?

Your answer

More Statistics questions