Halving the variance to halve the sample size

Question

You apply CUPED and it reduces your metric's variance by 50%, so the new variance is half the original. You want to keep the same statistical power as before.

By what percentage can you cut the required sample size (equivalently the runtime at fixed traffic)?

Accepted Answer

Sample size is proportional to variance. For a fixed effect, significance level, and power, n \propto 	ext{variance}. A less noisy metric needs fewer users to reach the same precision, so the required sample falls in step with the variance. Apply the 50% reduction. New variance is half the original, so n{	ext{new}} = 0.50 	imes n{	ext{old}}. You need only half as many users, a 50% reduction in sample size. At fixed daily traffic, the test finishes in about half the time. Perspective. Cutting a test's duration in half is a large operational win: you can run twice as many experiments in the…

Halving the variance to halve the sample size

Your answer

Discussion

Learn the concepts

Related questions