When a 5,050 vs 4,950 split is perfectly fine

Question

A pricing experiment was set up as a 50/50 split. It ended with 5,050 users in control and 4,950 in treatment, out of 10,000 total. An analyst worries the arms are unbalanced and wants to check for a sample-ratio mismatch.

Compute the SRM chi-square statistic and decide whether there is a mismatch.

Accepted Answer

Expected counts. A 50/50 split of 10,000 users gives E = 5{,}000 per arm. Observed are 5,050 and 4,950, so each arm is off by 50. Chi-square statistic. \chi^2 = \frac{50^2}{5{,}000} + \frac{50^2}{5{,}000} = \frac{2{,}500}{5{,}000} + \frac{2{,}500}{5{,}000} = 0.5 + 0.5 = 1.0. Compare to the threshold. With 1 degree of freedom the 5% critical value is 3.84. Since 1.0 < 3.84 (p-value about 0.32), the imbalance is well within what randomization produces by chance. There is no sample-ratio mismatch. The point. A 50/50 split will almost never land at exactly 5,000/5,000; small deviations are expect…

When a 5,050 vs 4,950 split is perfectly fine

Your answer

Discussion

Learn the concepts

Related questions