p

Question

An experiment on 10 million users finds a 0.01% relative increase in clicks with $p < 0.001$ . The analyst reports it as a "highly significant win."

Explain why a tiny effect can be highly statistically significant, and how to distinguish statistical significance from practical significance.

Accepted Answer

Why huge samples make everything significant. The p-value measures whether an effect is distinguishable from zero, and the standard error shrinks like 1/\sqrt{n}. With n = 10 million, the SE is minuscule, so any effect that isn't exactly zero, even a 0.01% blip with no business value, gets divided by a tiny SE and produces a giant test statistic and a microscopic p-value. Statistical significance answers "can we tell it's not zero?"; with enough data the answer is almost always yes, because nothing in the real world is exactly zero. The two questions are different. - Statistical significance:…

p < 0.001 on 10 million users, and still not worth shipping

Your answer

More Statistics questions