Are p = 0.049 and p = 0.051 really different?

Question

Signal A has p = 0.049 and signal B has p = 0.051. A manager keeps A ("significant") and kills B ("not significant"), treating them as fundamentally different results. Explain why the evidence in these two signals is almost identical, and why the sharp accept/reject split misrepresents them.

Accepted Answer

The evidence is nearly the same. A p-value is a smooth, continuous measure of surprise under the null. The z-statistics behind p = 0.049 and p = 0.051 are about 1.97 and 1.95, essentially the same number. Nothing meaningful distinguishes the underlying edges; the difference is a rounding-error wobble in the data. Why the split misrepresents them. The 0.05 threshold is a convention, not a law of nature. Chopping a continuous scale at 0.05 turns a hair's-breadth difference into a categorical "real versus fake" verdict. That is a distortion: A and B carry almost the same weight of evidence and s…

Are p = 0.049 and p = 0.051 really different?

Your answer

Discussion

Learn the concepts

Related questions