Qm
Statistics

Bot traffic that lands in only one arm

ab-testingsample-ratio-mismatchdata-quality

An A/B test intended a 50/50 split, but control ended up with many more users than treatment, and the SRM check fails badly. Digging in, the team finds that an automated scraper was assigned to control (its requests all fell into one bucket) and generated tens of thousands of low-value pageviews with almost no conversions. Treatment shows a +3% lift on the primary metric.

Explain how bot traffic in one arm biases the comparison, and what the team should do.

Your answer

Solving needs a free account

Answers, streaks and solutions unlock when you are signed in. Reading the question and the hint stays free.

Create a free accountSign in

Discussion

Sign in to join the discussion · reading is open to everyone

💡 Discussion rules

  1. No full solutions here. Hints and approaches only.
  2. Complexity, edge cases and intuition are the point.
  3. Interview experiences are welcome. Respect your NDAs.

Loading discussion…

Learn the concepts

The theory behind this question.

A/B TestingAlternative Data Coverage BiasAnnotation Guidelines and Inter-Annotator AgreementBerkson's ParadoxCluster-Randomized and Switchback Experiments

Related questions

Your 50/50 split came out 50.2/49.3, is that a problem?Is your 50/50 split actually 50/50? A sample-ratio checkA 24,600 vs 25,400 split on 50,000 users, real or noise?When a 5,050 vs 4,950 split is perfectly fineSRM on an uneven 90/10 holdback splitA 40% lift appeared overnight. Be suspicious, not thrilled
All questions →