How Close Is Close Enough in a Replication?

When a researcher tries to reproduce a published result — say, a factor's average return — small differences from the original are almost guaranteed, even with no error at all: slightly different data vendors, rebalancing dates, or rounding conventions all shift the number a little. The practical question isn't "did I get the exact same figure," but "is the difference small enough to still call this the same finding, or large enough that something is actually wrong?"

There's no single universal threshold, but researchers generally look at whether the replicated result is in the same direction, of a similar order of magnitude, and statistically indistinguishable from the original given the noise inherent in the estimate — for example, a replicated factor return should generally fall within the original's confidence interval, or close to it, not merely have the same sign. A result that flips sign, shrinks by an order of magnitude, or falls far outside a reasonable margin should be treated as a failed replication worth investigating, not a rounding difference to wave away.

Consider a published anomaly reporting a 6% annualized long-short return. A replication using a different but comparable dataset that finds 5.5% is close enough to call a successful replication; one that finds 1% (or a negative return) points to a genuine discrepancy — perhaps the original relied on a data-snooping artifact, a look-ahead bias, or a sample period that no longer holds.