Sample size for an A/B test on order size

Question

You run an A/B test to see whether a checkout change shifts the mean order size. Order size has a standard deviation of about $\sigma = 20$ dollars in each arm. You want to detect a true mean difference of $\Delta = 5$ dollars, using a two-sided test at $\alpha = 0.05$ with $80\%$ power.

Roughly how many orders per arm do you need?

Accepted Answer

The formula. n \approx \frac{2\,\sigma^2 \left(z{\alpha/2} + z\betaight)^2}{\Delta^2}. Constants. (1.96 + 0.84)^2 = 7.84. Plug in. \sigma^2 = 400, \Delta^2 = 25: n \approx \frac{2 	imes 400 	imes 7.84}{25} = \frac{6272}{25} \approx 251. Rounding up, about 252 orders per arm, roughly 500 total. The effect is only one quarter of a standard deviation, so despite a plain-looking scenario the sample requirement is a few hundred. Takeaway. A quarter-sigma effect needs about 252 orders per arm at 80% power. The smaller the effect relative to the noise, the larger the sample, and it grows with th…

Sample size for an A/B test on order size

Your answer

More Statistics questions