A weak-ish CUPED covariate at correlation 0.5

Question

You are considering CUPED for an A/B test on orders per user. The only pre-experiment covariate you have, orders in the prior month, correlates $\rho = 0.5$ with the in-test metric.

By what percentage does CUPED reduce the variance, and is it worth using?

Accepted Answer

Why CUPED can still help. Even a moderate predictor removes some of the baseline spread that has nothing to do with the treatment. CUPED subtracts the predictable part of each user's outcome using the covariate, lowering the noise without biasing the treatment effect. The formula. Variance is multiplied by (1 - ho^2): 	ext{new variance} = (1 - ho^2) 	imes 	ext{old variance}. Plug in ho = 0.5. ho^2 = 0.25, \qquad 1 - ho^2 = 0.75. The variance drops to 75% of its original value, a 25% reduction. Is it worth it? Yes, usually. A 25% variance cut translates into roughly 25% fewer user…

A weak-ish CUPED covariate at correlation 0.5

Your answer

More Statistics questions