Is your 50/50 split actually 50/50? A sample-ratio check

Question

You designed an experiment to split traffic evenly, 50/50, between control and treatment. At the end you count 10,200 users in control and 9,800 in treatment, out of 20,000 total. A teammate shrugs: "close enough." You want to check whether this imbalance is more than chance.

Compute the sample-ratio-mismatch (SRM) chi-square statistic and state whether there is an SRM.

Accepted Answer

What SRM tests. If randomization is working, the counts should match the intended ratio up to normal sampling noise. A sample-ratio mismatch means the observed split is statistically inconsistent with the intended one, a red flag that something in assignment, logging, or filtering is broken. The standard check is a chi-square goodness-of-fit test on the counts. Set up expected counts. Intended split 50/50 on 20,000 users gives expected E = 10{,}000 in each arm. Observed are O1 = 10{,}200 and O2 = 9{,}800, so each arm is off by 200. Chi-square statistic. \chi^2 = \sum \frac{(O - E)^2}{E} = \fr…

Is your 50/50 split actually 50/50? A sample-ratio check

Your answer

Discussion

Learn the concepts

Related questions