Data Validation and Schema Checks

Every model is fed by a pipeline that pulls in data from somewhere — a market data feed, a vendor file, an upstream database — and any of those sources can quietly change without warning: a column gets renamed, a currency field starts arriving in cents instead of dollars, a date format flips. Schema checks are automated rules that catch this before it reaches the model, by verifying that each incoming batch of data matches an expected structure: the right columns, the right data types, values inside plausible ranges, and no unexpected nulls.

A schema is typically written once, describing each field's type (integer, string, timestamp), whether it can be missing, and reasonable bounds (a price should never be negative, a probability should stay between 0 and 1). Every new batch of data is checked against that schema automatically, and a violation either blocks the pipeline or raises an alert, rather than letting silently corrupted data flow into a model that has no way of knowing something changed.

Worked example. A firm's daily equity price feed is expected to contain a close_price column of positive floats. One day an upstream vendor changes their export format and starts sending price in cents rather than dollars for a subset of tickers, so a stock that closed at $52.10 arrives as 5210. A schema check with a plausibility rule — say, flag any close_price outside a stock's trailing 30-day range by more than 5x — would catch this immediately and quarantine the affected rows, instead of letting a trading model see a fake 100x price jump and react to it.