Designing an Ablation Study

A model with fifteen features and three architectural tricks beats the baseline. Which of those eighteen things actually mattered? Without an answer, the team can't tell whether they've built something real or whether twelve of the eighteen changes are dead weight that happened to survive because nobody checked. An ablation study answers this directly: remove one component at a time, retrain, and measure how much performance drops. A component whose removal barely changes anything wasn't doing the work its presence implied.

The logic: one change, holding everything else fixed

The idea is borrowed from controlled experiments generally — change one variable, hold the rest constant, and attribute the resulting difference to that one variable. Start from the full model and its measured performance. Then produce a series of variants, each missing exactly one component — one feature, one regularization term, one architectural piece — retrained and evaluated under identical conditions (same data split, same random seed policy, same evaluation metric) as the full model. The drop in performance from removing component $X$, relative to the full model, is an estimate of how much $X$ contributes given everything else is present — which is an important caveat, because a feature's contribution can depend heavily on what else is in the model.

Worked example: ablating a five-feature signal

A model predicting next-week stock returns uses five features and achieves a validation information coefficient (IC) of 0.048. An ablation study removes each feature one at a time and retrains:

Removed feature	IC without it	Drop from full (0.048)
none (full model)	0.048	—
earnings surprise	0.031	0.017
short interest	0.044	0.004
sector momentum	0.046	0.002
analyst revisions	0.021	0.027
price reversal (5d)	0.047	0.001

Analyst revisions and earnings surprise are clearly load-bearing — removing either causes a large drop. Price reversal, sector momentum, and short interest barely move the number when removed individually; each is contributing very little on its own, and a team optimizing for a simpler, more robust model has a strong, evidence-based case for dropping some of them, rather than a guess based on which features "sound" important.

What this means in practice

Ablation is standard practice before publishing or productionizing any model with more than a couple of moving parts, precisely because it's easy to accumulate features and tricks that survived a single backtest by chance rather than by genuine contribution — see benchmark design and leaderboard overfitting for the related failure mode of tuning against the same test set too many times. A well-run ablation study also needs a baseline row: the full model's performance, and ideally a naive baseline (no model at all, or the simplest possible model) for scale, so a 0.002 IC drop can be judged against how large an IC even is on this problem, rather than in a vacuum. Ablations should also be re-run periodically rather than trusted forever — a feature that mattered a lot in one regime can become nearly redundant once the market environment shifts, and a component that looked useless in isolation last year can become the one propping up performance in a new regime.

Practice

1. In the worked example, if removing "sector momentum" together with "price reversal" caused IC to drop to 0.038 (a bigger drop than either alone), what would that tell you about the two features that ablating them one at a time missed?
2. Why is it important to keep the data split and random seed identical across every variant in an ablation study, rather than re-splitting for each one?