Randomization Inference in Experiments

You run an A/B test: 10 accounts get a new fee schedule, 10 keep the old one, and you observe a $40 average difference in monthly trading revenue. Is that difference real, or could it plausibly have arisen just from which accounts happened to land in which group? A standard t-test answers this by assuming the underlying data is roughly normal and using a known formula for how the test statistic behaves under the null. Randomization inference throws that assumption away and answers the question more directly, using only one fact you actually know for certain: which accounts got treatment was decided by a coin flip you controlled.

An analogy: could the deal have gone the other way?

Imagine dealing two poker hands and one player wins a suspiciously large pot. Before accusing anyone of cheating, ask: if I reshuffled this exact deck and redealt many times, how often would a hand this lopsided occur purely by the luck of the shuffle? If lopsided results are common under random reshuffling, the actual outcome isn't surprising; if they're rare, something beyond chance is likely going on. Randomization inference does exactly this with your experiment's treatment labels: it reshuffles who was labeled treatment and who was labeled control, recomputes the outcome gap each time, and asks how often a reshuffle produces a gap as extreme as the one actually observed.

The idea, one symbol at a time

Let $\hat{\delta}$ be the observed difference in average outcomes between treatment and control groups. Under the sharp null hypothesis — that treatment had zero effect for every single unit, not just on average — each unit's outcome would have been exactly the same no matter which group it was assigned to. That means the labels "treatment" and "control" could be shuffled among the $n$ units without changing any of the underlying outcome values, and every one of the $\binom{n}{n_1}$ possible label assignments (choosing $n_1$ units to be "treatment" out of $n$) is equally likely under this null. Computing $\hat{\delta}$ for every possible reshuffling builds the randomization distribution. The p-value is

$$p = \frac{1}{\binom{n}{n_1}} \sum_{\text{permutations}} \mathbb{1}\{|\hat{\delta}_{\text{perm}}| \geq |\hat{\delta}_{\text{obs}}|\},$$

which in plain English is: the fraction of all possible ways to reassign the labels that would have produced a gap at least as extreme as the one you actually saw. No assumption about the shape of the outcome distribution is used anywhere in this calculation — the randomness comes entirely from the experiment's own design.

Worked example 1: a tiny experiment by hand

Suppose 4 accounts, 2 treated and 2 control, with observed revenues $120, $100 (treated) and $90, $95 (control). Observed difference: $\hat{\delta} = \frac{120+100}{2} - \frac{90+95}{2} = 110 - 92.5 = 17.5$. There are $\binom{4}{2} = 6$ ways to choose 2 of the 4 accounts as "treatment." Checking all 6 relabelings, only the actual assignment and its mirror image (which flips the sign) produce a gap of magnitude 17.5 — the largest possible split, since it's genuinely the two highest values against the two lowest. So 2 of 6 permutations achieve $|\hat{\delta}| \geq 17.5$, giving $p = 2/6 \approx 0.33$ — not significant, despite the eye-catching $17.50 gap, because with only 4 units any clean split already looks large.

Worked example 2: why it doesn't need normality

Suppose outcomes are heavily skewed — most accounts earn near $0 but a few earn $1,000+ (typical of trading revenue). A t-test's validity depends on the sampling distribution of the mean being roughly normal, which skewed data with a small sample can violate badly. Randomization inference sidesteps this: it never assumes a distribution for the outcomes, only that the assignment was random. Permuting labels 10,000 times by computer and counting how often the simulated gap exceeds the observed one gives a valid p-value regardless of skew, because the permutation distribution is built from the actual data, not a theoretical curve.

Note how a t-test's reference curve above is a fixed, assumed normal shape — permutation inference instead builds its own reference distribution directly from your data's actual permutations, which is why it stays valid even when the real distribution looks nothing like this bell curve.

What this means in practice

Randomization inference is the right default for small or oddly-distributed experiments — A/B tests on a handful of accounts, portfolio managers, or trading desks — exactly the settings where classical formulas' normality assumptions are most likely to fail. It requires only that assignment was genuinely random, which connects it directly to good experimental design: if randomization wasn't clean, the permutation logic breaks along with everything else.