Qm
Statistics

The grader knew which website version was new

blindingobserver-biasab-testing

A company runs an A/B test of a website redesign. Because the winning metric is subjective "page quality", an analyst hand-scores a sample of sessions from each version. The analyst, who championed the redesign, can see which version each session came from.

What bias does this introduce, and how should the scoring be done instead?

Your answer

Solving needs a free account

Answers, streaks and solutions unlock when you are signed in. Reading the question and the hint stays free.

Create a free accountSign in

Discussion

Sign in to join the discussion · reading is open to everyone

💡 Discussion rules

  1. No full solutions here. Hints and approaches only.
  2. Complexity, edge cases and intuition are the point.
  3. Interview experiences are welcome. Respect your NDAs.

Loading discussion…

Learn the concepts

The theory behind this question.

A/B TestingCluster-Randomized and Switchback ExperimentsExperimental Design: Blocking and Factorial DesignsOnline Experimentation for Model RolloutA Benchmarking Harness for New Signals

Related questions

Why you run an A/B test for whole weeks, not "until it's ready"How big does the A/B test need to be?Why you can't stop an A/B test when it "hits significance"One guardrail metric out of twenty went redThe test isn't significant, can we just run it longer?
All questions →