LLM Data Contamination in Backtests

A backtest is supposed to simulate a strategy as if it only knew what was actually knowable at each historical date. Feed a large language model a 2019 earnings call and ask it to predict what happens next, and there's a subtle problem: that model may have been pretrained on internet text from 2023 or later, which already includes news articles, analyst commentary, and even the actual subsequent stock-price outcome discussing that exact 2019 call. The model isn't reasoning about the call in a vacuum — it may already "know" what happened next, and a backtest built on its output can look far better than any strategy could have actually performed live.

The analogy: grading a student who's already seen the answer key

Imagine testing a student's ability to predict exam outcomes by giving them last year's practice exam — except this student happens to have already seen last year's actual results published online. Their "prediction" isn't a prediction; it's recall dressed up as foresight. A backtest using an LLM pretrained on data after the historical period it's testing has the same problem, just harder to notice because the leakage is buried inside billions of pretraining tokens rather than an obvious document.

Why this is worse than ordinary look-ahead bias

Ordinary look-ahead bias in a backtest is usually a traceable coding mistake — a feature computed using a future data revision, or a filing date that doesn't match when the filing was actually public. LLM contamination is harder to catch because:

• The leakage sits inside the model's pretrained knowledge, not the researcher's pipeline code, so reviewing the backtest script alone won't reveal it.
• Pretraining cutoff dates are often approximate or undisclosed, so it's not always obvious which historical period is actually "safe" to test against a given model.
• The model never admits it's drawing on knowledge of the actual outcome — it doesn't say "I recall this happened," it just outputs a confident answer indistinguishable in style from genuine forward reasoning.

Worked example: an inflated backtest from contamination

A researcher backtests an LLM-based strategy that reads 2018–2020 earnings-call transcripts and predicts next-quarter guidance direction, using a model with a pretraining cutoff in 2023. Backtested "accuracy" comes out at 78% — impressively high. Suspicious of the number, the researcher reruns the identical test using a version of the model with a pretraining cutoff before 2018, so it could not possibly have seen any post-call outcome data. Accuracy on the same transcripts drops to 54%, barely better than a coin flip on the specific guidance-direction task. The 24-point gap is a direct, quantified estimate of how much of the original result was leakage rather than genuine predictive signal.

Worked example: a contamination check on a single date

A backtest claims to predict a stock's reaction to a specific 2021 earnings call. Prompting the model with only the transcript text and asking "what happened to this stock in the following week" and it answers with a specific, correct percentage move — a red flag, since nothing in the transcript itself states the subsequent price move; a genuinely uncontaminated model reasoning only from the transcript's content should express uncertainty about a number it has no way to have derived from the given text alone.

What this means in practice

Any strategy backtest that relies on an LLM to reason about historical events needs a contamination check — comparing performance on periods before versus after the model's pretraining cutoff, ideally with an older model whose cutoff genuinely predates the entire backtest window. A backtest that only works on periods the model could plausibly "remember" the outcome of is not evidence of a working strategy; it's evidence of leakage dressed up as one.