The feature spread from treated users to their control friends

Question

A social app randomly gives some users a new group-chat feature and keeps others as controls. Treated users become more active, as expected, but the control users also become noticeably more active. The team is confused: the control was supposed to stay flat.

Explain what happened, why it biases the A/B estimate, and how you would design the experiment.

Accepted Answer

The feature travels through the social network. Group chat is inherently social: a treated user who starts a group pulls their friends into it, and some of those friends are in the control group. Once a control user is added to a lively group chat, they use the app more too. So the treatment leaked from treated users to their control friends. This is spillover through a network: one user's treatment changes a connected user's outcome. Why it biases the estimate. A clean A/B test needs the control group to represent the no-feature world. But here controls were partly exposed to the feature thr…

The feature spread from treated users to their control friends

Your answer

More Statistics questions