"No significant effect", but the interval says nothing

Question

Two experiments both report "no significant effect." In experiment 1 the 95% CI for the effect is $[-0.3\%, +0.4\%]$ . In experiment 2 it's $[-9\%, +11\%]$ . A summary slide lists both as "null result."

Explain why these two "null results" mean opposite things, and the principle that separates them.

Accepted Answer

Same verdict, opposite meaning. Both intervals contain zero, so both tests are non-significant. But they tell completely different stories: - Experiment 1, CI [-0.3\%, +0.4\%]: the effect is pinned down tightly around zero. Even the extremes are tiny. This is genuine evidence that any real effect is small and unimportant, a meaningful null result. - Experiment 2, CI [-9\%, +11\%]: the data are consistent with a large harm, a large benefit, or nothing at all. This experiment simply didn't learn anything, it was too underpowered or noisy to distinguish an important effect from zero. "Not signif…

"No significant effect", but the interval says nothing

Your answer

More Statistics questions