CUPED variance reduction when the covariate correlates 0.6

Question

Your team runs A/B tests on average session time. As a CUPED covariate you use each user's average session time in the two weeks before the test starts, which correlates $\rho = 0.6$ with the in-test metric.

By what percentage does CUPED reduce the variance of the metric?

Accepted Answer

How CUPED helps. Users differ a lot in baseline session time regardless of any treatment, and that pre-existing spread is pure noise for measuring a treatment effect. CUPED uses the pre-experiment covariate to subtract off the predictable part of each user's outcome, shrinking the noise while leaving the treatment effect unbiased. The formula. With correlation ho between covariate and metric, the variance is multiplied by (1 - ho^2): 	ext{new variance} = (1 - ho^2) 	imes 	ext{old variance}. Plug in ho = 0.6. ho^2 = 0.36, \qquad 1 - ho^2 = 0.64. The variance falls to 64% of its …

CUPED variance reduction when the covariate correlates 0.6

Your answer

Discussion

Learn the concepts

Related questions