The smallest p-value a permutation test can produce

Question

You run an exact two-sample permutation test comparing a group of $5$ units against a group of $4$ units ( $n = 9$ total). The p-value is the fraction of all group relabelings whose test statistic is at least as extreme as observed.

What is the smallest one-sided p-value this test could ever return, no matter how extreme the data?

Accepted Answer

An exact permutation test's null distribution has exactly one entry per distinct way of assigning the labels. Here that is the number of ways to choose which 4 of the 9 units form the smaller group: \binom{9}{4} = 126. The most extreme possible outcome is when the observed labeling is the single most extreme of all 126 arrangements. Then the one-sided p-value is the fraction "1 out of 126": p{\min} = \frac{1}{126} \approx 0.0079. No data can push it below that, because the p-value is a count of arrangements divided by 126. Since 0.0079 < 0.05, this design can reach significance, but its resol…

The smallest p-value a permutation test can produce

Your answer

More Statistics questions