Cleaning High-Frequency Tick Data

A day of raw tick-by-tick data for a single liquid stock can be millions of rows, and a meaningful fraction of them are not usable as-is: a trade printed at half the correct price because of a fat-finger or a data-vendor glitch, a quote update that arrived with a timestamp earlier than the trade it's supposed to have preceded, the same trade reported twice by two different feeds, or a "trade" that's actually an exchange-reported correction or an off-exchange print that shouldn't be treated as normal market activity. Any statistic computed on the raw feed — a volatility estimate, a spread measurement, a signed order flow number — inherits every one of these errors, and because the errors are often extreme values, they distort averages and especially variances far more than their small count would suggest.

The standard cleaning pipeline runs several checks in sequence. Price filters flag trades that deviate from a rolling local median by more than some threshold (say, 10 standard deviations of recent tick-to-tick moves), catching fat-finger and bad-tick errors. Timestamp checks enforce that events arrive in non-decreasing time order per feed and per instrument, and reconcile trade and quote timestamps that may come from different systems with different clock synchronization. Duplicate removal drops exact repeats that come from redundant feed connections. Condition-code filtering excludes trades tagged as corrections, out-of-sequence reports, or off-exchange prints that don't reflect the continuous market (many exchanges publish a trade condition code exactly for this purpose). Crossed or locked quote handling deals with brief moments where the bid exceeds the ask across venues — real but not something most single-venue analysis should treat as a tradeable state.

Worked example. A stock trading around $50.00 has a raw tick sequence: 50.01, 50.02, 5.00, 50.03, 50.02. The 5.00 print is almost certainly a fat-finger (decimal misplaced) or bad feed tick — it's a 90% single-tick move followed immediately by a return to the $50 range, which a rolling-median filter with a reasonable deviation threshold (e.g., more than 20 times the recent typical tick size) catches and removes before any statistic is computed. Left in, that one bad tick alone would make a naive volatility estimate on this handful of ticks enormous — its squared deviation from the local mean dwarfs every other tick combined, even though it represents zero real trading activity.

What this means in practice

No microstructure research or live signal is trustworthy without documenting its cleaning pipeline, because different reasonable choices of threshold and filter can materially change downstream numbers like realized volatility or effective spread. Production trading systems typically apply a lightweight version of this filtering in real time (reject obviously bad ticks before they touch a signal) and a heavier, backward-looking version for research, where more context is available to distinguish a genuine large move from a bad print.

See Outlier Detection and Winsorization for the general statistical toolkit behind the filtering thresholds, and Tick, Volume And Dollar Bars for how cleaned tick data gets aggregated into usable bars.