Qm
Statistics

An SRM that shows up in just one country

ab-testingsample-ratio-mismatchsegmentation

An A/B test passes the overall SRM check, so at first glance the 50/50 split looks healthy. But when the team slices by country, one large market has a severe SRM (treatment there got far fewer users than control), while other markets look fine. That one market happens to have very different conversion behavior from the rest. The overall result shows treatment up +1.5%.

Explain why a segment-level SRM matters even when the overall check passes, and what the team should do.

Your answer

Solving needs a free account

Answers, streaks and solutions unlock when you are signed in. Reading the question and the hint stays free.

Create a free accountSign in

Discussion

Sign in to join the discussion · reading is open to everyone

💡 Discussion rules

  1. No full solutions here. Hints and approaches only.
  2. Complexity, edge cases and intuition are the point.
  3. Interview experiences are welcome. Respect your NDAs.

Loading discussion…

Learn the concepts

The theory behind this question.

A/B TestingCluster-Randomized and Switchback ExperimentsExperimental Design: Blocking and Factorial DesignsOnline Experimentation for Model RolloutA Benchmarking Harness for New Signals

Related questions

Your 50/50 split came out 50.2/49.3, is that a problem?Is your 50/50 split actually 50/50? A sample-ratio checkA 24,600 vs 25,400 split on 50,000 users, real or noise?When a 5,050 vs 4,950 split is perfectly fineSRM on an uneven 90/10 holdback split
All questions →