When treating one seller quietly hurts the control seller

Question

A marketplace runs an A/B test: treated sellers get boosted higher in search results, control sellers do not. Treated sellers see more sales, control sellers see fewer, and the team reports a large positive effect from the boost.

Explain why this A/B comparison overstates the true effect, and how you would design the experiment instead.

Accepted Answer

The two groups are not independent. Search results are a shared, roughly fixed shelf: if treated sellers are pushed up, control sellers are pushed down on the very same result pages that shoppers see. The control sellers did not get worse in any absolute sense; they lost sales to the treated sellers who leapfrogged them. Buyers have only so much to spend, so a sale gained by treatment is partly a sale stolen from control. This is interference (also called spillover): one unit's treatment changes another unit's outcome. Why it inflates the measured effect. A clean A/B test assumes the control …

When treating one seller quietly hurts the control seller

Your answer

Discussion

Learn the concepts

Related questions