Independent Replication by a Second Researcher
Rebuilding a strategy from the written spec alone, without looking at the original code, catches the bugs and hidden assumptions that reviewing the original code never will.
A researcher's backtest reports a Sharpe of 1.8 on a volatility carry strategy. Their code runs, the logic reads sensibly, a colleague reviews the pull request and approves it. Six months later, live performance is flat. The postmortem finds the original backtest joined option data using the trade date rather than the settlement date, silently pulling in a day of look-ahead on roughly 8% of trades — enough by itself to add most of that 1.8 Sharpe. Code review missed it because reviewing code means reading the same logic the author already believed was correct; it does not independently ask "does this number reappear if I build it a different way?"
Replication versus review
Code review checks whether the implementation matches what the author intended. Independent replication checks whether the result is real, by having a second researcher, working only from the strategy's written specification — the signal definition, the universe, the rebalance rule, not the original codebase — build it again from scratch and see whether the same performance shows up. The two exercises catch different bugs. Review catches things that look wrong in the code. Replication catches things that look right in the code but encode an assumption the author never noticed making, because a second implementation built independently has no reason to repeat the same silent choice.
Worked example
A momentum strategy in single-name equity options, original researcher reports Sharpe 1.6 over a four-year backtest. A second researcher, given only the written spec, rebuilds it independently over three weeks.
| Original | Replication | |
|---|---|---|
| Sharpe reported | 1.6 | 0.7 |
| Data join | Trade date | Settlement date (correctly lagged) |
| Universe | All listed names | All listed names, minus 40 later found delisted mid-sample |
| Result | Overstated by look-ahead + survivorship | Closer to what live trading would see |
The gap wasn't one bug, it was two independent silent assumptions the original researcher made without realising either was a choice — join timing and survivorship in the universe construction. Neither would have been caught by that same researcher rereading their own code, because both assumptions were invisible to them precisely because they'd made them without noticing. A second implementation, built blind to the first, had no reason to inherit either one, and the disagreement between the two Sharpe ratios is what surfaced the problem.
Replication means a second person builds the strategy again from the spec, blind to the original implementation — not a second person reading the first person's code. The value comes entirely from the independence; showing the replicator the original code defeats the purpose.
The common shortcut is having the second researcher review the code instead of rebuilding it, because it's faster and feels like the same due diligence. It isn't. Code review inherits every assumption baked into the original implementation and can only catch things that look syntactically or logically wrong — not a data join that runs without error but silently leaks information.
If a full independent rebuild isn't affordable for every strategy, reserve it for anything crossing the size or risk threshold that would trigger real capital allocation — the cost of a three-week replication is trivial next to the cost of a live Sharpe collapsing from 1.6 to flat after the fact.
Replication is expensive in researcher time, which is exactly why it should be reserved for strategies about to be funded rather than run on every idea in the pipeline — pair it with the cheaper adversarial stress tests in Adversarial Backtest Design as an earlier filter, and lean on Point-in-Time Data discipline so that at least one of the two most common replication failures, look-ahead in the data itself, is structurally prevented rather than caught after the fact.
Related concepts
Practice in interviews
Further reading
- Harvey, Liu & Zhu, ...and the Cross-Section of Expected Returns
- López de Prado, Advances in Financial Machine Learning (ch. 8)