Permutation Tests

You backtest a new signal on two baskets of stocks — one where the signal fires and one where it doesn't — and the signal basket returns 1.8% more over the test period. Is that a real edge, or could a random split of the same stocks into two groups of that size produce an 1.8% gap just by chance? A textbook t-test answers this by assuming returns are normally distributed and using a formula. But returns aren't normal — they're skewed and fat-tailed — so the formula's assumptions are shaky exactly where it matters most, in the tails, which is where "is this significant?" questions live. A permutation test sidesteps the assumption entirely: it doesn't ask "what would a normal distribution predict," it asks "what actually happens if I reshuffle these exact numbers at random, many times, and recompute the gap."

An analogy: shuffling a deck of report cards

Imagine two classrooms took the same test, and Classroom A's average score is 4 points higher than Classroom B's. You suspect this happened because Classroom A had a better teacher, but it's also possible the two classrooms just got, by luck of enrollment, a slightly stronger set of students. To check, take all the report cards from both classrooms, throw them in one pile, and deal them back out into two random groups of the same sizes as the real classrooms. Compute the average gap between these two random groups. Do this shuffle-and-recompute thousands of times. If a 4-point gap (or bigger) shows up constantly in these random reshuffles, the real 4-point gap isn't special — random noise alone produces gaps that size all the time. If a 4-point gap almost never shows up in the reshuffles, the real gap is unusual, and "better teacher" becomes a credible explanation.

The test, one symbol at a time

Let there be $n$ observations total, split into a group of size $n_1$ (signal fires) and $n_2$ (signal doesn't), with $n_1 + n_2 = n$. Let $T$ be the test statistic — here, the difference in group means, $T = \bar{X}_1 - \bar{X}_2$. Compute the observed value, $T_{obs}$, from the real group labels.

Now generate the null distribution by permutation: randomly reassign the $n$ observations into a group of size $n_1$ and a group of size $n_2$ — pretending the labels "fires" / "doesn't fire" are meaningless and just randomly relabeling — and recompute the statistic, $T^{(b)}$, for each of $B$ such reshuffles ($b = 1, \dots, B$). The permutation p-value is the fraction of reshuffled statistics at least as extreme as the real one:

$$p = \frac{1}{B} \sum_{b=1}^{B} \mathbf{1}\big( |T^{(b)}| \geq |T_{obs}| \big).$$

In plain English: out of all the random reshuffles you tried, what fraction produced a gap as big as or bigger than the gap you actually observed? If that fraction is small (conventionally below 5%), the observed gap is unusual under the "labels don't matter" hypothesis, and you have evidence the grouping is real. Crucially, this p-value is built entirely from your own data reshuffled — it makes no assumption that returns are normal, or symmetric, or anything else about their shape.

Worked example 1: six numbers by hand

Signal-fires returns: $\{2, 3, 4\}$ (mean 3). Signal-doesn't returns: $\{-1, 0, 1\}$ (mean 0). Observed statistic:

$$T_{obs} = 3 - 0 = 3 .$$

Pool all six values: $\{2, 3, 4, -1, 0, 1\}$. There are $\binom{6}{3} = 20$ ways to split six numbers into two groups of three; enumerate a few: split $\{2,3,4\}$ vs $\{-1,0,1\}$ is the real split ($T=3$, matches observed). Split $\{4, 3, -1\}$ vs $\{2, 0, 1\}$: means $2.0$ vs $1.0$, $T = 1.0$. Split $\{4, 1, 0\}$ vs $\{2, 3, -1\}$: means $1.67$ vs $1.33$, $T=0.33$. Split $\{4, -1, 0\}$ vs $\{2, 3, 1\}$: means $1.0$ vs $2.0$, $T=-1.0$. Carrying this out for all 20 splits (by symmetry, the extreme splits are the ones that keep the three biggest or three smallest values together), only the original split and its mirror image ($\{-1,0,1\}$ vs $\{2,3,4\}$, which gives $T=-3$) reach $|T|=3$ — none of the other 18 mixed splits get close. So

$$p = \frac{2}{20} = 0.10 .$$

With this tiny sample, a gap this size happens in 2 of the 20 possible relabelings — not overwhelming evidence, but the observed split is already the most extreme one possible, which is the best six numbers this unbalanced can do.

Worked example 2: using a subset of shuffles

With even modestly larger samples, checking every possible split is infeasible ($\binom{40}{20}$ is astronomical), so in practice you draw $B$ random reshuffles instead of all of them. Suppose with $n_1=n_2=20$ stocks, $T_{obs} = 1.8\%$, and after $B = 1{,}000$ random reshuffles, only 23 of them produce $|T^{(b)}| \geq 1.8\%$. Then

$$p = \frac{23}{1000} = 0.023 .$$

This says: if the "signal fires" label were meaningless noise, a gap as large as 1.8% would arise from pure random grouping only about 2.3% of the time. That's below the conventional 5% threshold, so the observed gap is judged unlikely to be a fluke of which stocks happened to land in which group — evidence the signal carries real information, obtained without ever assuming returns are normally distributed.

What this means in practice

Permutation tests are the standard way to sanity-check any signal or strategy split without leaning on distributional assumptions that don't hold for real returns — comparing signal-on versus signal-off periods, testing whether a factor's long leg genuinely outperforms its short leg, or checking whether a backtest's Sharpe ratio is distinguishable from what random trade timing would produce. They're computationally heavier than a closed-form t-test (you need many reshuffles), but they're exact for whatever data you actually have, fat tails and all, rather than approximately correct under an assumption you haven't verified. They also generalize immediately to any statistic — median, Sharpe ratio, correlation — not just the mean, which classical tests often can't handle cleanly.