Minimum detectable lift for a fixed A/B test

Question

Your A/B test is capped at n = 2{,}000 visitors per arm. The baseline conversion rate is p = 0.20. You will run a two-sided test at \alpha = 0.05 and want 80\% power. What is the minimum detectable effect (the smallest absolute lift in conversion rate you can reliably catch)?

Accepted Answer

Invert the sample-size formula. The usual proportion sample-size rule can be solved for the effect instead of for n. The minimum detectable absolute lift is \delta \approx \left(z{\alpha/2} + z\betaight) \sqrt{\frac{2\,p(1-p)}{n}}. This is just "the effect that a sample of size n has exactly enough power to detect." Constants. (z{\alpha/2} + z\beta) = 1.96 + 0.84 = 2.80. Plug in. p(1-p) = 0.20 	imes 0.80 = 0.16, so \sqrt{\frac{2 	imes 0.16}{2000}} = \sqrt{\frac{0.32}{2000}} = \sqrt{0.00016} = 0.01265. \delta \approx 2.80 	imes 0.01265 \approx 0.0354. So the smallest lift you can reliably…

Minimum detectable lift for a fixed A/B test

Your answer

Discussion

Learn the concepts

Related questions