Minimum detectable lift at 90% power

Question

Your A/B test allows n = 8{,}000 visitors per arm. The baseline conversion rate is a low p = 0.05. You will run a two-sided test at \alpha = 0.05, but this time you insist on 90\% power. What is the minimum detectable effect (smallest absolute lift you can reliably catch)?

Accepted Answer

The formula. \delta \approx \left(z{\alpha/2} + z\betaight) \sqrt{\frac{2\,p(1-p)}{n}}. Constants. For 90% power, z\beta = 1.28, so (z{\alpha/2} + z\beta) = 1.96 + 1.28 = 3.24; and p(1-p) = 0.05 	imes 0.95 = 0.0475. Plug in. \sqrt{\frac{2 	imes 0.0475}{8000}} = \sqrt{\frac{0.095}{8000}} = \sqrt{0.0000119} = 0.00345. \delta \approx 3.24 	imes 0.00345 \approx 0.0112. So about 1.1 percentage points, a move from 5% to roughly 6.1%. The very low base rate makes p(1-p) small, which helps, but demanding 90% instead of 80% power raised the multiplier from 2.80 to 3.24 and pushed the detectable e…

Minimum detectable lift at 90% power

Your answer

More Statistics questions