Spotting Overfitting in a Published Result

A paper shows a strategy with a beautiful, steadily rising equity curve and a Sharpe ratio north of 2. Before getting excited, a working researcher's first question isn't "does this work?" — it's "how many things did the author try before landing on this one?" Most published overfitting isn't fraud; it's the ordinary result of testing dozens of variants and reporting only the best one, without disclosing the search.

A few tells are worth checking every time. First, count the free parameters relative to the amount of data: a strategy with eight tunable thresholds backtested on fifteen years of daily data has enormous room to fit noise, no matter how principled each threshold sounds. Second, look for an equity curve that is suspiciously smooth — real strategies have drawdowns and flat stretches; a curve that climbs in an almost straight line usually means the parameters were chosen to produce exactly that curve. Third, check whether the paper reports any true out-of-sample test: a holdout period the author never touched while designing the rule, or better, a later time period that didn't exist when the research was done. Its absence isn't proof of overfitting, but it removes the one piece of evidence that would rule it out. Fourth, watch for parameter values that look hand-picked rather than principled — a 47-day lookback with no stated reason to prefer 47 over 40 or 50 is a flag that it was chosen because it happened to work best in-sample.

A useful habit: ask what the result would look like with the parameters perturbed slightly. If a strategy's Sharpe ratio collapses when a 20-day moving average becomes 22 days, the original number was likely a lucky draw from a noisy search rather than a robust pattern the market actually contains.