Eight of ten, is the p-value under 5%?

Question

A classifier makes 10 independent binary predictions and gets 8 right. Under the null that it is guessing, each prediction is correct with probability $0.5$ .

Compute the exact one-sided p-value (probability of 8 or more correct by guessing), and decide whether it is significant at the 5% level.

Accepted Answer

Set up. Let X \sim 	ext{Binomial}(10, 0.5). The one-sided p-value is p = \mathbb{P}(X \geq 8) = \frac{\binom{10}{8} + \binom{10}{9} + \binom{10}{10}}{2^{10}}. Count and compute. The coefficients are \binom{10}{8} = 45, \binom{10}{9} = 10, \binom{10}{10} = 1, summing to 56. With 2^{10} = 1024: p = \frac{56}{1024} \approx 0.0547 \approx 0.055. The verdict. At p \approx 0.055, the result is just above the 5% threshold, so it is not significant at the conventional level, despite looking strong (8 out of 10). This is a good reminder that small samples produce coarse, grainy p-values: with only 10…

Eight of ten, is the p-value under 5%?

Your answer

More Statistics questions