What does flipping a coin actually buy you?

Question

You want to test a new onboarding flow. One engineer suggests giving it to everyone who signs up on odd-numbered days and using even-day sign-ups as the control, since that is easy to implement. Another insists on assigning each user by a coin flip.

Explain what randomization buys you that a rule like "odd versus even days" cannot, especially regarding things you never measured.

Accepted Answer

The goal is comparable groups. A clean experiment needs the treatment and control groups to be alike in every way except the treatment, so that any outcome difference can only be caused by the treatment. The danger is a confounder: some background factor that differs between the groups and also affects the outcome. Why odd versus even days can fail. The assignment rule is tied to the calendar, and the calendar is not neutral. Odd and even days can line up differently with weekends, paydays, marketing pushes, or product launches. If, say, more odd days fall on weekends, your treatment group is…

What does flipping a coin actually buy you?

Your answer

Discussion

Learn the concepts

Related questions