More data shrinks the detectable effect

Question

With 1{,}000 users per arm, an experiment can reliably detect an effect of 5 units (its minimum detectable effect). You expand it to 4{,}000 users per arm, keeping the same \alpha and power. What is the new minimum detectable effect?

Accepted Answer

The scaling law, in the other direction. The minimum detectable effect shrinks as one over the square root of the sample size: 	ext{MDE} \propto \frac{1}{\sqrt{n}}. This is the flip side of n \propto 1/	ext{effect}^2. Apply it. Quadrupling n multiplies \sqrt{n} by 2, so the MDE shrinks by a factor of 2: 	ext{MDE}{	ext{new}} = \frac{5}{2} = 2.5 	ext{ units}. Four times the data buys you only twice the resolution, a direct consequence of the square-root scaling. Takeaway. The minimum detectable effect falls like 1/\sqrt{n}, so quadrupling the sample halves it: from 5 units to 2.5.

More data shrinks the detectable effect

Your answer

Discussion

Learn the concepts

Related questions