Cutting the effect to a third

Question

An experiment powered to detect a 3\% lift needed 1{,}600 users per arm. The team now wants to detect a 1\% lift (one third the effect), with everything else unchanged. How many users per arm are needed now?

Accepted Answer

The scaling law. With baseline, \alpha, and power fixed, n \propto 1/(	ext{effect})^2. Apply it. Cutting the effect to one third multiplies n by (3/1)^2 = 9: n{	ext{new}} = 9 	imes 1{,}600 = 14{,}400 	ext{ per arm}. Detecting a lift one third the size needs nine times the data. The cost grows with the square of how much finer you want to resolve. Takeaway. Because n \propto 1/	ext{effect}^2, reducing the target effect to a third multiplies the sample by nine: 1,600 per arm becomes 14,400.

Cutting the effect to a third

Your answer

Discussion

Learn the concepts

Related questions