Simpson's paradox: which ad creative truly converts better?

Question

You ran two ad creatives and measured conversion rate (conversions / impressions), split by device:

	Mobile	Desktop	Overall
Creative A	100/1000 (10%)	40/100 (40%)	140/1100 (12.7%)
Creative B	8/100 (8%)	380/1000 (38%)	388/1100 (35.3%)

Creative A converts better on mobile (10% vs 8%) and on desktop (40% vs 38%), yet B's overall rate is far higher (35.3% vs 12.7%).

Which creative should you serve, and why does the aggregate disagree with both segments?

Accepted Answer

This is Simpson's paradox: a comparison that holds inside every subgroup reverses when the groups are pooled. The driver is a confounder correlated with which creative got the impressions. The confounder is device. Desktop converts about four times better than mobile for both creatives (roughly 38 to 40 percent vs 8 to 10 percent). And the two creatives were shown to very different device mixes: - A got 1000 of its 1100 impressions on mobile, the low-converting device. - B got 1000 of its 1100 impressions on desktop, the high-converting device. So B's headline rate mostly reflects where it wa…

Simpson's paradox: which ad creative truly converts better?

Your answer

More Statistics questions