"The imbalance is tiny, just trust the lift", why not?

Question

An experiment intended a 50/50 split, but the SRM check fails badly (chi-square far above the threshold): control got noticeably more users than treatment. The dashboard still shows treatment beating control by a clean, "significant" +4% on the primary metric. A colleague argues: "The imbalance is only a percent or two, and the lift is big and significant, let's just ship it."

Explain why a sample-ratio mismatch invalidates the result even when the lift looks strong, what typically causes SRM, and what you should do.

Accepted Answer

SRM breaks the one thing an A/B test relies on. The entire value of a randomized test is that the two groups are identical on average except for the treatment, so any difference in the metric can be attributed to the treatment. A sample-ratio mismatch is direct evidence that assignment did not work as designed: something systematically decided who ended up in which arm. Once that is true, the groups may differ in ways that have nothing to do with the treatment, and your +4% could be that hidden difference rather than the treatment effect. The size of the count imbalance is not the point; the …

"The imbalance is tiny, just trust the lift", why not?

Your answer

Discussion

Learn the concepts

Related questions