A 24,600 vs 25,400 split on 50,000 users, real or noise?

Question

A search-ranking experiment was set to split traffic 50/50. The final counts are 24,600 in control and 25,400 in treatment, out of 50,000 users. By eye the arms look balanced.

Compute the SRM chi-square statistic and decide whether there is a sample-ratio mismatch.

Accepted Answer

Expected counts. A 50/50 split of 50,000 users gives E = 25{,}000 per arm. Observed are 24,600 and 25,400, so each arm is off by 400. Chi-square statistic. \chi^2 = \frac{400^2}{25{,}000} + \frac{400^2}{25{,}000} = \frac{160{,}000}{25{,}000} + \frac{160{,}000}{25{,}000} = 6.4 + 6.4 = 12.8. Compare to the threshold. With 1 degree of freedom the 5% critical value is 3.84. Since 12.8 \gg 3.84 (p-value about 0.0003), the imbalance is nowhere near chance. There is a sample-ratio mismatch. The intuition trap. A 49.2/50.8 split looks trivially balanced, but with 50,000 users the expected wobble arou…

A 24,600 vs 25,400 split on 50,000 users, real or noise?

Your answer

Discussion

Learn the concepts

Related questions