SRM on an uneven 90/10 holdback split

Question

A cautious feature rollout uses a 90/10 split: 90% of users stay on the control experience and 10% receive the new feature. Out of 100,000 users, you observe 89,700 in control and 10,300 in treatment.

Compute the SRM chi-square statistic for this uneven split and decide whether there is a mismatch.

Accepted Answer

Expected counts follow the intended ratio. The 90/10 design on 100,000 users gives expected counts E{	ext{control}} = 90{,}000 and E{	ext{treatment}} = 10{,}000. Observed are 89,700 and 10,300, so each arm is off by 300 (control low by 300, treatment high by 300). Chi-square statistic, using each arm's own expected count. \chi^2 = \frac{300^2}{90{,}000} + \frac{300^2}{10{,}000} = \frac{90{,}000}{90{,}000} + \frac{90{,}000}{10{,}000} = 1 + 9 = 10. Compare to the threshold. With 1 degree of freedom the 5% critical value is 3.84. Since 10 3.84 (p-value about 0.0016), there is a sample-ratio mi…

SRM on an uneven 90/10 holdback split

Your answer

Discussion

Learn the concepts

Related questions