The A/B test that reverses

Question

A website runs an A/B test of two page versions, tracking conversions split by device.

Desktop: version A converts 800 of 1000 visitors (80%), version B converts 85 of 100 visitors (85%).
Mobile: version A converts 10 of 100 visitors (10%), version B converts 150 of 1000 visitors (15%).

Version B has the higher conversion rate on desktop and on mobile. Yet version A has the higher overall conversion rate.

Show that version A has the higher overall conversion rate, and give it.

Accepted Answer

Within each device, B wins: desktop 85\% to 80\%, mobile 15\% to 10\%. Now pool each version's own traffic. 	ext{Version A overall} = \frac{800+10}{1000+100} = \frac{810}{1100} = \frac{81}{110} \approx 73.6\%. 	ext{Version B overall} = \frac{85+150}{100+1000} = \frac{235}{1100} \approx 21.4\%. Version A wins overall, 73.6\% to 21.4\%, despite losing on both devices. The reversal is a traffic-mix artifact. Version A was shown mostly to desktop visitors (1000 of 1100), who convert well, so its overall number rides high. Version B was shown mostly to mobile visitors (1000 of 1100), who convert…

The A/B test that reverses

Your answer

Discussion

Learn the concepts

Related questions