A 25% variance cut and its sample-size payoff

Question

You apply CUPED with a moderate pre-experiment covariate (correlation about 0.5), which reduces your metric's variance by 25%, so the new variance is 75% of the original. You want the same statistical power as before.

By what percentage can you cut the required sample size (equivalently the runtime at fixed traffic)?

Accepted Answer

Sample size tracks variance. For fixed effect, significance level, and power, n \propto 	ext{variance}. Reduce the noise and you need proportionally fewer users to reach the same precision. Apply the 25% reduction. New variance is 75% of the original, so n{	ext{new}} = 0.75 	imes n{	ext{old}}. You need only 75% as many users, a 25% reduction in sample size, and roughly 25% less runtime at fixed traffic. Perspective. A quarter off the sample is a solid, worthwhile saving even from a moderate covariate: a test that took 4 weeks could finish in about 3. It is less dramatic than the halving y…

A 25% variance cut and its sample-size payoff

Your answer

More Statistics questions