A 0.5% imbalance that still fails the SRM test

Question

A large-scale experiment was set to split traffic 50/50. It ended with 100,500 users in control and 99,500 in treatment, out of 200,000 total, a split of 50.25% versus 49.75%.

Compute the SRM chi-square statistic and decide whether this half-percent imbalance is a mismatch.

Accepted Answer

Expected counts. A 50/50 split of 200,000 users gives E = 100{,}000 per arm. Observed are 100,500 and 99,500, so each arm is off by 500. Chi-square statistic. \chi^2 = \frac{500^2}{100{,}000} + \frac{500^2}{100{,}000} = \frac{250{,}000}{100{,}000} + \frac{250{,}000}{100{,}000} = 2.5 + 2.5 = 5.0. Compare to the threshold. With 1 degree of freedom the 5% critical value is 3.84. Since 5.0 3.84 (p-value about 0.025), there is a sample-ratio mismatch, even though the split is only 50.25/49.75. The scale effect. This is the central lesson of SRM: the bigger the experiment, the smaller the percentag…

A 0.5% imbalance that still fails the SRM test

Your answer

Discussion

Learn the concepts

Related questions