From correlation 0.7 straight to time saved

Question

You apply CUPED with a pre-experiment covariate that correlates $\rho = 0.7$ with your metric. You want the same statistical power as before.

By what percentage can you cut the required sample size (equivalently the runtime at fixed traffic)?

Accepted Answer

Step 1: correlation to variance reduction. CUPED multiplies the variance by (1 - ho^2): 1 - ho^2 = 1 - 0.7^2 = 1 - 0.49 = 0.51. So the new variance is 51% of the original, a 49% reduction. Step 2: variance to sample size. Required sample size is proportional to variance, n \propto 	ext{variance}, so n{	ext{new}} = 0.51 	imes n{	ext{old}}. You need only 51% as many users, a 49% reduction in sample size, and about 49% less runtime at fixed daily traffic. Why the chain matters. A single covariate correlation of 0.7 turns into cutting your experiment nearly in half. That is the full value…

From correlation 0.7 straight to time saved

Your answer

More Statistics questions