Pre-Empting the Obvious Objections
Why a good research write-up names the criticisms a skeptical reader will raise, and answers them directly, instead of hoping nobody asks.
Every backtest write-up has a handful of predictable objections: is this overfit? Does it survive realistic transaction costs? Would it still work at real trading size, given the strategy's capacity? A researcher who leaves these unaddressed isn't being neutral — a skeptical reader (a portfolio manager, an allocator, a risk committee) will assume the worst about whatever wasn't discussed, and rightly so, since the easiest objections to skip are usually the ones the result is weakest against.
Pre-empting objections means stating the obvious critique yourself, in the researcher's own words, before presenting the answer: "the reader might reasonably worry this is overfit to the 2015-2019 sample — here is the out-of-sample performance from 2020-2023, and here's how the parameter choices were made without looking at that later period." This does two things a defensive or silent write-up can't: it demonstrates the researcher actually considered the weakness rather than being blind to it, and it lets the reader evaluate the actual answer instead of just noticing the gap and losing trust in everything else in the report.
The habit generalizes beyond backtests: any claim strong enough to change a decision — sizing a strategy up, greenlighting a live deployment — will attract scrutiny, and naming the objection first is both more persuasive and more honest than waiting to be asked.
State the reader's obvious objection yourself, in your own write-up, before they raise it — silence on a predictable weak point reads as either not having thought of it or having something to hide, while addressing it directly builds the credibility needed for the rest of the report to be trusted.
Further reading
- Chan, Quantitative Trading, ch. 6