Your 50/50 split came out 50.2/49.3, is that a problem?

Question

You designed an A/B test for a 50/50 split. When the data comes in, control has 50,200 users and treatment has 49,300, close, but not exactly even. Before reading the results, you check the split.

Test whether this imbalance is consistent with a fair 50/50 assignment, and explain why a "sample ratio mismatch" can invalidate the experiment even if the results look great.

Accepted Answer

Run the check. Total users N = 50{,}200 + 49{,}300 = 99{,}500; expected count per arm under 50/50 is 49{,}750. A chi-square goodness-of-fit test: \chi^2 = \frac{(50{,}200 - 49{,}750)^2}{49{,}750} + \frac{(49{,}300 - 49{,}750)^2}{49{,}750} = \frac{450^2}{49{,}750} 	imes 2 = \frac{202{,}500}{49{,}750}	imes 2 \approx 8.14. With 1 degree of freedom the critical value is 3.84 (p \approx 0.004). So this imbalance is not consistent with a fair coin, it's a statistically significant sample ratio mismatch (SRM). What looks like "close enough" to the eye is, at this scale, a red flag. Why SRM is a ki…

Your 50/50 split came out 50.2/49.3, is that a problem?

Your answer

More Statistics questions