Can a paired sign-flip test with four pairs reach significance?

Question

You run an exact paired permutation test on 4 matched pairs. The null says each pair's difference is equally likely to carry a + or - sign, so the permutation distribution is generated by flipping the signs of the differences in all possible ways. You want a two-sided test. What is the smallest two-sided p-value this test can produce, and can it ever reach p < 0.05?

Accepted Answer

For a paired (sign-flip) permutation test, each of the 4 differences independently keeps or flips its sign, giving 2^{4} = 16 equally likely sign patterns. The most extreme observed pattern (all differences pointing the same way) is matched by its mirror image (all flipped) at the opposite tail. So the smallest two-sided p-value counts those two extreme patterns out of 16: p{\min} = \frac{2}{16} = 0.125. Since 0.125 0.05, a paired permutation test with only four pairs cannot reach p < 0.05, however large the differences. You need more pairs (e.g. 5 pairs give 2/32 = 0.0625, still short; 6 pai…

Can a paired sign-flip test with four pairs reach significance?

Your answer

More Statistics questions