Duplicate Records and Deduplication

A duplicate record is a row (a trade, a price observation, a fundamental data point) that appears more than once in a dataset when it should appear once. In market data this happens more often than it seems: a data vendor re-sends a corrected version of a day's prices without removing the original, two different feeds get merged and overlap on a shared date range, or a corporate action (a stock split, ticker change, or merger) causes the same underlying security to be recorded under two identifiers that both get pulled in.

Duplicates rarely announce themselves — a backtest engine will happily process two identical rows for the same date as two independent observations. The practical damage is that any statistic computed by counting or averaging over rows gets silently distorted: a duplicated high-return day is counted twice, inflating both the apparent frequency of good days and, if position sizing reacts to recent history, the realized exposure to that day's move.

The fix is a deduplication step run before any research, using a key that uniquely identifies a real-world record — for prices, that's usually (security identifier, date, source) — and explicit rules for which copy to keep when duplicates disagree (typically the most recently corrected vendor record wins, not an arbitrary first-seen row). This check belongs in the automated data pipeline, run and logged every time new data lands, not as a one-off manual pass.