Detecting a fifth of the lift

Question

An experiment powered to detect a 10\% lift needed 300 users per arm. The team now wants to detect a 2\% lift (one fifth the effect), with everything else unchanged. How many users per arm are needed now?

Accepted Answer

The scaling law. With baseline, \alpha, and power fixed, n \propto 1/(	ext{effect})^2. Apply it. Cutting the effect to one fifth multiplies n by (10/2)^2 = 5^2 = 25: n{	ext{new}} = 25 	imes 300 = 7{,}500 	ext{ per arm}. Resolving a lift one fifth the size costs twenty-five times the data. The square in the law means small reductions in the target effect translate into large jumps in sample size. Takeaway. Since n \propto 1/	ext{effect}^2, targeting a fifth of the lift multiplies the sample by 25: 300 per arm becomes 7,500.

Detecting a fifth of the lift

Your answer

Discussion

Learn the concepts

Related questions