From correlation 0.7 straight to time saved

Question

You apply CUPED with a pre-experiment covariate that correlates $\rho = 0.7$ with your metric. You want the same statistical power as before.

By what percentage can you cut the required sample size (equivalently the runtime at fixed traffic)?

Accepted Answer

Step 1: correlation to variance reduction. CUPED multiplies the variance by (1 - ho^2): 1 - ho^2 = 1 - 0.7^2 = 1 - 0.49 = 0.51. So the new variance is 51% of the original, a 49% reduction. Step 2: variance to sample size. Required sample size is proportional to variance, n \propto 	ext{variance}, so n{	ext{new}} = 0.51 	imes n{	ext{old}}. You need only 51% as many users, a 49% reduction in sample size, and about 49% less runtime at fixed daily traffic. Why the chain matters. A single covariate correlation of 0.7 turns into cutting your experiment nearly in half. That is the full value…

From correlation 0.7 straight to time saved

Your answer

Discussion

Learn the concepts

Related questions