The false-positive chance on a single null test

Question

You run one hypothesis test at the 5% significance level. Suppose the null hypothesis is actually true (there is truly no effect).

What is the probability that this single test returns a "statistically significant" result, and why is that probability not the same as "the chance my result is a fluke"?

Accepted Answer

The probability. When the null is true, the chance of crossing the significance bar is exactly the significance level: \mathbb{P}(	ext{significant} \mid 	ext{null true}) = \alpha = 0.05 = 5\%. So a single truly-null test throws a false positive 5% of the time, or 1 in 20. Why it is not "the chance my result is a fluke." The 5% is a conditional probability: it assumes the null is true. In real life you do not know whether the null is true. Whether your particular significant result is a fluke depends on how often the things you test are real to begin with (the base rate). If almost everythin…

The false-positive chance on a single null test

Your answer

Discussion

Learn the concepts

Related questions