When the two arms log events differently

Question

An A/B test intended a 50/50 split fails the SRM check: treatment records fewer users than control. The team discovers the cause is instrumentation. In control, a user is logged as soon as the page loads; in treatment, the new code only logs a user after a component finishes rendering, so users who leave during that brief window never get counted in treatment. Treatment shows a +2% lift.

Explain why this logging mismatch invalidates the result, and what the team should do.

Accepted Answer

Different counting rules make the arms non-comparable. In a valid A/B test the only difference between arms should be the treatment itself, including how users are measured. Here the arms count users at different moments: control counts everyone who arrives, treatment only counts users who stay long enough for a component to render. That means treatment quietly excludes the quickest-leaving users, exactly the people least likely to convert or engage. Treatment's counted population is therefore a filtered, higher-quality subset, and its metric is inflated for reasons that have nothing to do wi…

When the two arms log events differently

Your answer

More Statistics questions