Give the trial to the first 100 volunteers?

Question

A company wants to test a new premium feature. A manager suggests enrolling the first 100 users who volunteer as the treatment group and using a later batch of users as the control, because that is the fastest way to launch.

Why does "first 100 volunteers as treatment" undermine the comparison, and what does randomization fix?

Accepted Answer

Early volunteers are not typical. People who sign up first for something new tend to be the most engaged, most tech-forward, and most enthusiastic users. That eagerness is a confounder: it predicts higher activity and spending all on its own, quite apart from the feature. So a treatment group made of early volunteers is stacked with people who would have looked better than average anyway. Any advantage you measure mixes the feature's effect with the fact that keen users got it. Adjusting cannot fully rescue it. You might try to control for observed differences like past activity, but eagernes…

Give the trial to the first 100 volunteers?

Your answer

Discussion

Learn the concepts

Related questions