How much variance does CUPED remove at correlation 0.7?

Question

You want to speed up your A/B tests using CUPED (Controlled-experiment Using Pre-Experiment Data). Your outcome metric is revenue per user during the test. You have a pre-experiment covariate, each user's revenue in the 4 weeks before the test, which correlates $\rho = 0.7$ with the in-test metric.

By what percentage does CUPED reduce the variance of your metric, and why does that help?

Accepted Answer

The core idea of CUPED. Much of the variation in your metric is baggage users bring with them: some people just spend more than others, before the test even starts. That predictable part adds noise to your comparison without carrying any information about the treatment. CUPED subtracts off the part of each user's outcome that you could have predicted from their pre-experiment behavior, leaving a cleaner, lower-variance metric that still measures the treatment effect without bias. The formula. If the pre-experiment covariate correlates \rho with the metric, CUPED reduces the variance by a fact…

How much variance does CUPED remove at correlation 0.7?

Your answer

Discussion

Learn the concepts

Related questions