Combinatorially Symmetric Cross-Validation and PBO

You have run 200 variants of a strategy and one of them has a Sharpe of 2.4. The usual next step is to test that one variant on a hold-out and see what happens. But a single hold-out gives you a single number, and a single number cannot answer the question you actually have, which is not "is this variant good?" but "is my selection procedure any good?" — if I pick the in-sample winner from this family, how often does that pick turn out to be a dud?

Combinatorially symmetric cross-validation (CSCV) answers exactly that, and its output, the probability of backtest overfitting (PBO), is a number you can put next to a Sharpe ratio.

The recipe, in five plain steps

1. Build a matrix. Rows are time periods, columns are your $N$ strategy variants. Each cell is that variant's return in that period. Every variant must be run over the same history.
2. Chop time into $S$ equal blocks. $S = 16$ is the common choice.
3. Take every balanced split. Choose any 8 of the 16 blocks as the training half; the other 8 become the test half. There are $\binom{16}{8} = 12{,}870$ such splits. It is called symmetric because each block serves as training exactly as often as it serves as test, so no period is privileged.
4. In each split, do what you would really do. Pick the variant with the best in-sample Sharpe on the training half. Then look up where that same variant ranks among all $N$ variants on the test half.
5. Score the rank. If the winner lands in the bottom half out-of-sample, that split is a failure. PBO is the fraction of the 12,870 splits that failed.

The rank is converted to a number with the logit, $\lambda = \log\!\big(\omega / (1-\omega)\big)$, where $\omega$ is the winner's relative rank out-of-sample on a 0-to-1 scale. In words: $\lambda$ is positive when the pick beat the median, negative when it lost to the median, and zero exactly at the median. PBO is the share of the distribution sitting at or below zero.

Worked example: 200 crossover variants

Eight years of daily returns on one futures contract, and 200 moving-average crossover variants (fast lookback 5 to 50, slow 20 to 200). Split into $S = 16$ half-year blocks, giving 12,870 combinations.

Across those splits, the in-sample winner averaged a Sharpe of 2.3 — and the very same variant averaged 0.05 on its matching test half. Of the 12,870 splits, 7,850 put the winner below the out-of-sample median:

$$\text{PBO} = 7850 / 12870 = 0.61 .$$

Read that in plain English: if you pick the best-looking variant from this family, there is a 61% chance it will underperform the median member of the same family going forward. Your selection procedure is worse than picking at random, because the search is now systematically drawn to the variants that fit the noise hardest.

Contrast a second family: the same contract, but only 12 variants chosen for economic reasons rather than by grid search. Best in-sample Sharpe 1.1, matching out-of-sample Sharpe 0.6, PBO 8%. Lower headline number, real strategy.

Worked example: scoring one split by hand

Take one split with $N = 200$ variants. Its in-sample winner comes 150th out of 200 on the test half, counting 1 as worst. The relative rank is $\omega = 150/201 = 0.746$, so

$$\lambda = \log\!\frac{0.746}{0.254} = \log 2.94 = +1.08 ,$$

a comfortable pass. Another split's winner comes 62nd: $\omega = 62/201 = 0.309$ and $\lambda = \log(0.309/0.691) = -0.81$, a failure. PBO simply counts how many of the 12,870 $\lambda$ values came out negative.

Where it sits

CSCV is the cheap, model-free member of a family. The Deflated Sharpe Ratio adjusts the headline number for the number of trials and the non-normality of returns; Combinatorial Purged Cross-Validation builds many out-of-sample paths while purging overlapping labels; Data-Snooping Bias is the underlying disease all of them treat. Use CSCV first, because it needs nothing but the matrix of returns you already produced when you ran the parameter sweep — and because a PBO above one half is a stop sign no amount of further tuning will remove.