A strong covariate that shrinks the test to a third

Question

You apply CUPED with a strong pre-experiment covariate that correlates $\rho = 0.8$ with your metric (behavior is very stable over time). You want the same statistical power as before.

By what percentage can you cut the required sample size (equivalently the runtime at fixed traffic)?

Accepted Answer

Step 1: correlation to variance reduction. CUPED multiplies the variance by (1 - ho^2): 1 - ho^2 = 1 - 0.8^2 = 1 - 0.64 = 0.36. So the new variance is just 36% of the original, a 64% reduction. Step 2: variance to sample size. Since required sample size is proportional to variance, n{	ext{new}} = 0.36 	imes n{	ext{old}}. You need only 36% as many users, a 64% reduction in sample size, and about 64% less runtime at fixed daily traffic. Why strong covariates are prized. A correlation of 0.8 shrinks the experiment to roughly a third of its original size. A test that would have taken 6 wee…

A strong covariate that shrinks the test to a third

Your answer

More Statistics questions