A slow-loading treatment page that drops users

Question

An A/B test intended a 50/50 split, but treatment ended up with 6% fewer users than control, and the SRM check fails decisively. Investigation shows the treatment page loads a heavier script and takes about a second longer to become interactive; some users on slow connections leave before the page (and their assignment) is logged. Treatment shows a +5% lift on conversion.

Explain how this specific SRM biases the measured lift, and what the team should do.

Accepted Answer

Who got dropped is the key. The users who abandon a slow-loading page are not a random sample: they are disproportionately people on poor connections or slower devices, who also tend to convert at lower rates. Because those low-converting users bounce out of treatment before being counted, treatment's remaining users are a healthier, higher-converting slice than control's. So even if the treatment change did nothing, treatment's average conversion would look higher purely because its weak users were filtered out. The +5% lift is at least partly, and possibly entirely, this selection effect ra…

A slow-loading treatment page that drops users

Your answer

More Statistics questions