What a p-value of 0.03 does not mean

Question

A backtest of a new signal returns a p-value of $0.03$ against the null "the strategy has zero mean return." A colleague concludes: "So there's a 97% chance the strategy is real and only a 3% chance the null is true."

Is the statement correct? Give the correct interpretation, and explain what you would need to make a statement about the probability the null is true.

Accepted Answer

The statement is wrong. A p-value is computed assuming the null is true: p = \mathbb{P}\big(	ext{data at least this extreme} \mid 	ext{null true}\big) = 0.03. It is a statement about the data under the null, not about the null under the data. Your colleague has flipped the conditioning and read p as \mathbb{P}(	ext{null true} \mid 	ext{data}), which it is not. In frequentist terms the null is either true or false; no probability attaches to it. Correct interpretation. "If the strategy truly had zero edge, data this favourable or more would occur only 3\% of the time." That is evidence aga…

What a p-value of 0.03 does not mean

Your answer

More Statistics questions