Does p = 0.001 mean a big effect?

Question

An analyst backtests a signal over 5 million trades and reports " $p = 0.001$ , a highly significant edge." The estimated average edge per trade is $0.02$ basis points.

Is a smaller p-value evidence of a larger or more important effect? Explain what the tiny p-value does and does not establish here.

Accepted Answer

A small p-value is not a big effect. The p-value measures how surprising the data are under the null (no edge), and that surprise grows with both the effect size and the sample size. With 5 million trades, even a laughably small true edge produces a tiny p-value, because the standard error shrinks like 1/\sqrt{n} and a huge n makes the estimate extremely precise. So p = 0.001 here says "we are confident the edge is not exactly zero," not "the edge is large." What it does establish. With this much data, the analyst has strong evidence that the true edge differs from zero, i.e. the sign is prob…

Does p = 0.001 mean a big effect?

Your answer

More Statistics questions