Monte Carlo Permutation Tests for Strategies

A trend-following strategy on crude oil futures shows a Sharpe ratio of 1.1 over ten years. Is that skill, or do trending markets simply reward any rule that stays with recent direction, insight or not? A t-test on the Sharpe assumes returns are independent and roughly normal — an assumption daily trading returns routinely violate. A permutation test sidesteps the assumption: it asks what your exact rule would have scored on data where any genuine structure has been destroyed but the raw statistical texture survives.

The idea, before the mechanics

Take the real return series or the real trading signal, and scramble it thousands of different ways so that any true directional relationship between signal and future return is erased, while the rest of the data's character survives. Run your exact strategy logic on each scrambled version and record its Sharpe. If your real Sharpe beats 990 of 1,000 scrambled versions, that is meaningfully different from beating only 550 of 1,000 — the latter is what you'd expect from a strategy with no edge at all.

Building the null distribution

The right thing to scramble depends on what null hypothesis you're testing. Two common choices:

• Shuffle trade order or block-bootstrap the return series to destroy serial dependence while keeping the marginal return distribution intact — tests "is the sequencing of trades adding value, beyond just being long a risky asset."
• Shuffle the entry-signal timestamps against the return series (keep returns fixed, randomly relabel which days were "signal on") — tests "does the signal pick informative days, or would any random subset of days of the same count have done as well."

For each of $M$ scrambles (typically 1,000–10,000), rerun the identical backtest — same costs, same sizing, same rebalance rule — and store the resulting Sharpe. The p-value is the fraction of scrambled Sharpes at or above the real one:

$$p = \frac{1 + \#\{m : SR_m \geq SR_{\text{real}}\}}{M+1}.$$

The plus-ones avoid ever reporting exactly zero, since you can never be certain nothing scrambled could beat you, only that none of your $M$ draws did.

Worked example: crude oil trend-follower

Real strategy: 20/100-day moving-average crossover on WTI futures, 10 years, Sharpe $1.10$. Shuffle the entry/exit timestamps 2,000 times, keeping the actual daily return series fixed, and rerun the identical position-sizing and cost model on each shuffled signal. Suppose 46 of the 2,000 scrambled runs produce a Sharpe of 1.10 or higher.

$$p = \frac{1 + 46}{2001} \approx 0.0235.$$

About 2.3% of random signal placements do as well as the real one — better than a typical 5% significance bar, so there's something here, but note it is far from the near-zero p-value a naive Sharpe-1.1-must-be-great read would suggest.

Worked example: a strategy that fails the test

A short-vol overlay shows Sharpe $0.85$ over the same window. Shuffling its entries 2,000 times, 340 scrambled runs beat 0.85.

$$p = \frac{1+340}{2001} \approx 0.170.$$

Roughly one in six random signal placements does as well. That 0.85 headline Sharpe is consistent with noise — plausible because short-vol strategies harvest a background risk premium that shows up in almost any systematically-short positioning, real signal or not, so the bar the permutation test sets is deliberately high for exactly this strategy family.

Switch the parent shape above between skewed and normal: the sampling distribution stays roughly bell-shaped, which is why a Monte Carlo p-value works even when returns are far from normal — the test never assumes normality, it builds the comparison distribution empirically.

What this means in practice

Run a permutation test before trusting any headline Sharpe, especially where the edge could plausibly be "any exposure to a known risk premium" rather than genuine skill. It costs only compute, needs no distributional assumptions, and directly answers the only question that matters: could randomness alone have produced this result.