What a p-value is, and is not

Question

A researcher tests a new signal and reports " $p = 0.03$ , so there's a 97% chance the signal is real."

Give the correct definition of a p-value, explain what is wrong with the researcher's sentence, and show with a quick calculation why a small p-value can coexist with a low probability that the signal is real.

Accepted Answer

Definition. The p-value is the probability, computed assuming the null hypothesis is true, of observing a test statistic at least as extreme as the one actually observed: p = \mathbb{P}(	ext{data at least this extreme} \mid H0). It is a statement about the data given the null, not about the null given the data. What went wrong. The researcher inverted the conditioning: p = \mathbb{P}(	ext{data} \mid H0) was read as \mathbb{P}(H0 \mid 	ext{data}) = 0.03. Converting one into the other requires a prior via Bayes' rule, and the answer depends heavily on that prior. The base-rate calculation. S…

What a p-value is, and is not

Your answer

More Statistics questions