Testing a 2x2 table when the counts are tiny

Question

A pilot compares two order-routing configs. Config A has 6 rejected orders out of 12; config B has 1 rejection out of 12. You want to test whether rejection rates differ. A colleague reaches for the chi-square test of independence.

Explain why the chi-square test is unreliable on counts this small, what Fisher's exact test does instead, and how its p-value is defined.

Accepted Answer

Why chi-square struggles here. The chi-square test of independence relies on a large-sample approximation: the test statistic is only approximately chi-square distributed, and that approximation degrades when expected cell counts are small (the common rule of thumb is every expected count \geq 5). With 12 per group and a combined rejection rate of 7/24 \approx 0.29, some expected counts hover near or below 5, so the nominal p-value can be well off. Yates' continuity correction patches this partially but crudely. What Fisher's exact test does. Instead of approximating, it computes the probabil…

Testing a 2x2 table when the counts are tiny

Your answer

More Statistics questions