Minimum detectable lift for a one-sided test

Question

Your A/B test allows n = 1{,}500 visitors per arm. The baseline conversion rate is p = 0.30. Because you only care about improvements, you will run a one-sided test at \alpha = 0.05 with 80\% power. What is the minimum detectable effect (smallest absolute lift you can reliably catch)?

Accepted Answer

The one-sided formula. \delta \approx \left(z{\alpha} + z\betaight) \sqrt{\frac{2\,p(1-p)}{n}}, using z\alpha = 1.645 instead of z{\alpha/2} = 1.96. Constants. (z\alpha + z\beta) = 1.645 + 0.84 = 2.485; p(1-p) = 0.30 	imes 0.70 = 0.21. Plug in. \sqrt{\frac{2 	imes 0.21}{1500}} = \sqrt{\frac{0.42}{1500}} = \sqrt{0.00028} = 0.01673. \delta \approx 2.485 	imes 0.01673 \approx 0.0416. So about 4.2 percentage points, a move from 30% to roughly 34.2%. The base rate near 0.5 makes p(1-p) large (more binomial noise), and the sample is modest, so the detectable effect is fairly coarse even with t…

Minimum detectable lift for a one-sided test

Your answer

More Statistics questions