Putting a Number on Self-Deception: the Probability of Backtest Overfitting

Saying "your backtest is probably overfit" is easy. It is also useless. It is the kind of criticism that can be levelled at any result, which means it discriminates between nothing and helps you decide nothing.

The same team behind the "pseudo-mathematics" broadside, Bailey, Borwein, Lopez de Prado and Zhu, clearly felt this, because their follow-up work is an attempt to move from accusation to measurement. Their question: can we compute the probability that a given research process is producing overfit results? Not a vague warning. An actual number, between zero and one, that you can put next to your strategy.

The answer is yes, and the machinery is called the Probability of Backtest Overfitting, or PBO.

The problem: standard cross-validation does not fit the crime

In ordinary machine learning, the defence against overfitting is cross-validation. Chop your data into pieces, train on some pieces, test on the others, rotate, and average. If your model looks good on data it did not learn from, you can breathe a little easier.

This does not transfer cleanly to strategy backtesting, for two reasons.

First, financial data has an arrow of time. Standard cross-validation happily trains on 2019 and tests on 2015, which is a form of time travel that no real trader gets to do. Any result built that way is contaminated.

Second, and more subtly, the thing being overfit is not the model, it is the selection. In a backtest, you typically do not fit one flexible model. You generate many candidate strategies (a grid of parameters, a family of signals) and then you pick the best one. The overfitting lives in the act of picking. Cross-validating each individual candidate misses the point entirely, because the sin was committed at the moment of selection.

So the authors needed a procedure that tests the research process, not the individual strategy.

The key idea via analogy: does your best in-sample horse win the second race?

Here is the whole idea, stripped of machinery.

Imagine you have a hundred candidate strategies and a long history of daily returns for each one. Now do this:

1. Cut the timeline into a number of equal blocks. Say sixteen chunks.
2. Deal those chunks into two halves: half go into a pretend "in-sample" set, half into a pretend "out-of-sample" set. Do this in every possible combination, and do it symmetrically, so that each half is the same size and every chunk gets to play both roles.
3. In each split, find the strategy that scored best in the in-sample half. This is you, in miniature, doing what you always do: picking the winner.
4. Now look up how that same strategy ranked in the out-of-sample half.
5. Repeat over all the splits, and count.

The question you are asking is beautifully simple: when I pick the winner using one set of data, does it tend to be a winner on the other set, or does it collapse into mediocrity?

If your in-sample champion routinely finishes in the bottom half out of sample, then your selection procedure is not finding skill, it is finding luck. PBO is literally the fraction of splits in which the in-sample champion underperforms the median out of sample. That is it. A PBO of 0.6 means: sixty percent of the time, the strategy my process would have chosen turned out to be a below-average performer on fresh data.

The authors call this Combinatorially Symmetric Cross-Validation. The "combinatorially symmetric" part matters: by using all the balanced splits rather than one arbitrary split, you stop yourself from cherry-picking a flattering train/test division, which would just push the overfitting up one level.

The analogy: you are running a stable of horses, and you always enter the one that won its practice runs. PBO asks how often your practice-run winner finishes in the back half of the actual race. If the answer is "usually," your practice runs are measuring nothing but track conditions on the day.

Why it mattered

• It converts a scolding into a diagnostic. You can compute PBO for your own research pipeline, today, and get a number. Numbers can be compared, tracked, and argued with. Warnings cannot.
• It evaluates the process rather than the product. This is the real conceptual jump. It accepts that you are going to search over many strategies (of course you are, that is research) and asks whether the searching itself is reliable. That is the right unit of analysis.
• It produces a genuinely useful side effect. Running the procedure also shows you the relationship between in-sample and out-of-sample performance across your candidates. A healthy process shows a positive relationship: better in-sample tends to mean better out-of-sample. An overfit process often shows a flat or negative relationship, meaning your in-sample rankings are actively misleading. Seeing that negative slope with your own eyes is a memorable experience.
• It gave allocators a question with teeth. "What is the PBO of your research process?" is much harder to bluff past than "are you sure this isn't overfit?"

The honest limitations

• It only measures what you show it. PBO evaluates the set of candidate strategies you hand it. If you already threw away four hundred ideas last year before building this grid, that history is invisible to the calculation, and the true overfitting is worse than PBO reports. It is a lower bound on your sins, not a full accounting.
• Chopping time into shuffled blocks is not innocent. By reassembling blocks from different periods into pretend in-sample and out-of-sample sets, you break the natural time ordering and can disrupt genuine long-horizon dependence like regimes and trends. The authors accept this trade for the sake of getting many splits, but it is a real cost, and it makes PBO a better tool for high-frequency and mean-reverting strategies than for slow, regime-dependent ones.
• There is no magic threshold. Is a PBO of 0.35 acceptable? Nobody can tell you. The number is meaningful mostly in comparison: against other processes, or against the same process before and after you cleaned it up.
• It can itself be gamed. Anything that becomes a reported metric becomes a target. A researcher who tweaks their candidate set until the PBO looks good has simply added another layer of overfitting, and this one is invisible.
• A low PBO is not a promise of profit. It says your selection process is not obviously self-deceiving. It says nothing about transaction costs, capacity, crowding, or whether the market regime that generated your data still exists.

The one-line takeaway

The Probability of Backtest Overfitting takes the vague fear that "my best strategy might just be my luckiest one" and turns it into a computable number: how often does the strategy my process picks as the winner turn out to be a below-average performer on data it has not seen?