Minimum detectable lift from a low baseline

Question

Your A/B test allows n = 5{,}000 visitors per arm. The baseline conversion rate is p = 0.10. You will run a two-sided test at \alpha = 0.05 with 80\% power. What is the minimum detectable effect (smallest absolute lift you can reliably catch)?

Accepted Answer

The formula. \delta \approx \left(z{\alpha/2} + z\betaight) \sqrt{\frac{2\,p(1-p)}{n}}. Constants. (1.96 + 0.84) = 2.80; p(1-p) = 0.10 	imes 0.90 = 0.09. Plug in. \sqrt{\frac{2 	imes 0.09}{5000}} = \sqrt{\frac{0.18}{5000}} = \sqrt{0.000036} = 0.006. \delta \approx 2.80 	imes 0.006 = 0.0168. So about 1.7 percentage points, a move from 10% to roughly 11.7%. The lower base rate (p(1-p) is smaller than at p=0.20) and the larger sample both help shrink the detectable effect. Takeaway. A lower base rate and more traffic both lower the minimum detectable effect. Here 5,000 per arm can catch lif…

Minimum detectable lift from a low baseline

Your answer

Discussion

Learn the concepts

Related questions