Data Quality Checks and Validation

A vendor's feed occasionally sends a stock price with a decimal point in the wrong place, a duplicate row, or a batch of trades for a day the market was actually closed. None of these errors crash a pipeline — the file parses fine, the numbers are numeric, the columns are the right type — so without an explicit check, bad data sails straight through into a backtest or a live position calculation and produces a wrong answer that looks completely normal. Data quality checks are the automated rules that catch this class of problem: not "does the code run," but "is the data actually plausible."

Checks generally fall into a few recurring categories. Range checks catch values outside what's physically possible or economically sane — a stock price of zero or negative, a trading volume larger than the total shares outstanding. Consistency checks compare related fields against each other — a bid price higher than the ask price, a high-of-day lower than the close. Completeness checks catch missing data — a symbol that normally reports every minute going silent for an hour, a corporate action that should have produced an adjustment but didn't. Referential checks catch orphaned or mismatched records — a trade referencing a symbol that doesn't exist in the reference-data table for that date.

The critical design decision is what to do when a check fails: halt the pipeline and alert a human, or drop the bad record and continue, or flag it and pass it through anyway for downstream code to handle. Silently dropping bad records is convenient but dangerous if it happens often enough to bias results — dropping every crossed-quote record, for instance, might systematically remove exactly the volatile moments a risk model most needs to see. The safer default for anything feeding a backtest or live strategy is to fail loudly on anything unexpected, because a pipeline that crashes is annoying but a pipeline that quietly produces wrong numbers is expensive.

Worked example: catching a decimal error before it costs money

A price feed reports a stock trading at $4,250 instead of the correct $42.50 — a vendor-side decimal placement bug. A range check comparing each new price against the stock's prior-day close, flagging anything more than, say, 20% away without a matching news event, would catch this instantly: $4,250 is roughly 100x the prior close, far outside any plausible single-day move. Without that check, the bad price flows into a backtest as a real 100x return, or into a live system as a real trade decision based on a price that never actually existed.

What this means in practice

Data quality checks should run as close to ingestion as possible, so bad data never reaches a system where it could influence a decision — catching it after a backtest has already run means throwing away the whole result once discovered. The set of checks worth writing grows over time as new failure modes are discovered in production; a mature pipeline's validation suite is effectively a running list of every way the data has been wrong before.