Is p = 0.049 really a discovery and p = 0.051 a dud?

Question

Experiment A yields p = 0.049 and is written up as a "significant discovery." Experiment B yields p = 0.051 and is filed as a "null result." The underlying estimates and data are almost identical. Explain why treating 0.05 as a bright line is misguided, and what to report instead.

Accepted Answer

The two results are essentially the same. A p-value is a continuous measure of how surprising the data are under the null. 0.049 and 0.051 represent almost identical evidence, the difference between them is trivial and well within the noise of any single experiment. Nothing physical changes as the p-value crosses 0.05; the threshold is a convention Fisher suggested, not a law of nature. Treating one as a triumph and the other as a failure ("dichotomania") manufactures a qualitative distinction where only a negligible quantitative one exists. Why the bright line causes harm. 1. Publication and…

Is p = 0.049 really a discovery and p = 0.051 a dud?

Your answer

More Statistics questions