Halve the effect, quadruple the sample

Question

An experiment powered to detect a 4\% lift needed 2{,}500 users per arm. The team now wants to detect a 2\% lift instead (half the effect), keeping the same baseline, the same \alpha, and the same power. How many users per arm are needed now?

Accepted Answer

The scaling law. For a fixed baseline, \alpha, and power, the required sample size grows as one over the square of the effect you want to detect: n \propto \frac{1}{(	ext{effect})^2}. This comes straight from the sample-size formula, where the effect sits squared in the denominator. Apply it. Halving the effect (4\% 	o 2\%) multiplies n by (1/	frac{1}{2})^2 = 2^2 = 4: n{	ext{new}} = 4 	imes 2{,}500 = 10{,}000 	ext{ per arm}. So chasing an effect half as large costs four times the data, 40,000 total instead of 10,000. This is why targeting ever-smaller lifts becomes brutally expensive. T…

Halve the effect, quadruple the sample

Your answer

More Statistics questions