Testing an A/B/C experiment the right way

Question

An A/B/C test has three checkout variants and records, for each, how many visitors converted. A colleague immediately runs three pairwise two-proportion z-tests (A vs B, A vs C, B vs C) and reports whichever is significant.

Explain the right sequence: an omnibus chi-square test of independence first, then careful pairwise follow-up, and why order matters.

Accepted Answer

Step 1: the omnibus test. With three or more groups, first ask a single question: "are the conversion rates all equal, or does at least one differ?" Lay the data out as a 3 	imes 2 contingency table (variant × converted/not) and run a chi-square test of independence: \chi^2 = \sum{	ext{cells}} \frac{(O - E)^2}{E}, \qquad E{ij} = \frac{(	ext{row total}i)(	ext{col total}j)}{N}, with degrees of freedom (	ext{rows}-1)(	ext{cols}-1) = (3-1)(2-1) = 2. A significant result means the variants are not all the same, without yet saying which. This single gatekeeper test controls the overall false-…

Testing an A/B/C experiment the right way

Your answer

More Statistics questions