If CUPED cuts variance 40%, how much shorter is the test?

Question

You apply CUPED to an A/B test and it reduces your metric's variance by 40%, so the new variance is 60% of the original. You want the same statistical power (same $\alpha$ , same power, same effect to detect) as before.

By what percentage can you cut the required sample size (equivalently, the test runtime at fixed traffic)?

Accepted Answer

Sample size scales directly with variance. The number of users needed for a given power is proportional to the variance of the metric: n \propto 	ext{variance}. This is because a smaller variance means each user's data is less noisy, so you need fewer of them to pin down the effect with the same precision. Cut the variance and the required sample falls in the same proportion. Apply the 40% reduction. New variance is 60% of the original, so n{	ext{new}} = 0.60 	imes n{	ext{old}}. You need only 60% as many users, a 40% reduction in sample size. At a fixed daily traffic rate, runtime is prop…

If CUPED cuts variance 40%, how much shorter is the test?

Your answer

Discussion

Learn the concepts

Related questions