How big does the A/B test need to be?

Question

Your baseline conversion rate is 10%. The team wants the experiment powered to detect a lift to 11% (a 1 percentage-point absolute improvement) with the usual settings: two-sided $\alpha = 0.05$ and 80% power.

Estimate the required sample size per arm, and explain what each ingredient contributes and how the answer moves if the target effect shrinks.

Accepted Answer

The formula. For comparing two proportions, a standard sample-size approximation per arm is n \approx \frac{\left(z{\alpha/2} + z{\beta}ight)^2 \left[p1(1-p1) + p2(1-p2)ight]}{(p2 - p1)^2}. The pieces: z{\alpha/2} = 1.96 (two-sided 5%), z{\beta} = 0.84 (80% power), so their sum is 2.80 and (2.80)^2 = 7.84. This constant is the "\alpha and power" cost, it rises if you demand more power or a stricter \alpha. Plug in. p1 = 0.10, p2 = 0.11: p1(1-p1) = 0.09, \quad p2(1-p2) = 0.11 	imes 0.89 = 0.0979, \quad 	ext{sum} = 0.1879. (p2 - p1)^2 = (0.01)^2 = 0.0001. n \approx \frac{7.84 	imes 0.187…

How big does the A/B test need to be?

Your answer

More Statistics questions