A very weak effect and a fourfold inflation

Question

An analyst reports only the signals that pass $p < 0.05$ (two-sided), and the true effect is just half a standard error in size.

Quantify how badly the reported effect overstates the truth, and how often it points the wrong way, when the true effect equals $0.5$ standard errors.

Accepted Answer

The setup. In standard-error units the estimate is distributed N(0.5, 1). A result is significant if \hat	heta 1.96. Power. \mathbb{P}(\hat	heta 1.96) = 1 - \Phi(1.46) \approx 0.072 and \mathbb{P}(\hat	heta < -1.96) = \Phi(-2.46) \approx 0.0069. Total power \approx 0.079, so under 8% of runs of this real effect ever reach significance. This is a badly underpowered test. Inflation. Among the significant (mostly upper-tail) results, the expected reported estimate is \mathbb{E}[\hat	heta \mid \hat	heta 1.96] = 0.5 + \frac{\varphi(1.46)}{1 - \Phi(1.46)} = 0.5 + \frac{0.137}{0.072} \approx 2.…

A very weak effect and a fourfold inflation

Your answer

More Statistics questions