Minimum detectable lift at a 50% base rate

Question

Your A/B test allows n = 1{,}000 visitors per arm. The baseline rate is p = 0.50 (a near coin-flip metric, such as click-or-not). You will run a two-sided test at \alpha = 0.05 with 80\% power. What is the minimum detectable effect (smallest absolute lift you can reliably catch)?

Accepted Answer

The formula. \delta \approx \left(z{\alpha/2} + z\betaight) \sqrt{\frac{2\,p(1-p)}{n}}. Constants. (1.96 + 0.84) = 2.80; p(1-p) = 0.50 	imes 0.50 = 0.25, which is the maximum possible value, so this is the noisiest base rate. Plug in. \sqrt{\frac{2 	imes 0.25}{1000}} = \sqrt{\frac{0.5}{1000}} = \sqrt{0.0005} = 0.02236. \delta \approx 2.80 	imes 0.02236 \approx 0.0627. So about 6.3 percentage points, a move from 50% to roughly 56.3%. Because p(1-p) peaks at 0.5, a coin-flip metric with only 1,000 per arm has the coarsest resolution of all: anything smaller than a 6-point swing tends to sl…

Minimum detectable lift at a 50% base rate

Your answer

More Statistics questions