The standard error trap in "clicks per session"

Question

Your key A/B metric is clicks-per-session: total clicks divided by total sessions, pooled across all users in an arm. But sessions belong to users, users generate different numbers of sessions, and sessions within a user are correlated.

Explain why a naive standard error on this ratio metric is wrong, and how the delta method (with clustering) fixes it.

Accepted Answer

Why it's tricky. The metric R = \frac{\sum 	ext{clicks}}{\sum 	ext{sessions}} = \bar{X}/\bar{Y} is a ratio of two random totals, and the randomization unit is the user, not the session. Two problems compound: 1. It's a ratio, not a mean. The variance of \bar{X}/\bar{Y} has no elementary formula because both numerator and denominator are random and correlated (a user with more sessions contributes more to both). 2. Sessions are clustered within users. Treating each session as an independent observation ignores within-user correlation and understates the variance, so you get false significanc…

The standard error trap in "clicks per session"

Your answer

More Statistics questions