When a CUPED covariate barely helps (correlation 0.3)

Question

You want to apply CUPED to an A/B test on clicks per user, but your only pre-experiment covariate is a weak one, and it correlates just $\rho = 0.3$ with the in-test metric.

By what percentage does CUPED reduce the variance, and what does this tell you about choosing covariates?

Accepted Answer

The mechanism, briefly. CUPED subtracts the part of each user's outcome that the covariate can predict, removing noise without biasing the treatment effect. The stronger the covariate predicts the outcome, the more noise it removes. The formula. Variance is multiplied by (1 - ho^2): 	ext{new variance} = (1 - ho^2) 	imes 	ext{old variance}. Plug in ho = 0.3. ho^2 = 0.09, \qquad 1 - ho^2 = 0.91. The variance falls only to 91% of its original value, a 9% reduction. The lesson about covariates. Because CUPED's benefit scales with the square of the correlation, a weak covariate barely…

When a CUPED covariate barely helps (correlation 0.3)

Your answer

Discussion

Learn the concepts

Related questions