Missing Data and Imputation

A NaN is a message. In most fields the message is "somebody forgot". In finance it is usually "something happened": the stock was halted, the company was acquired, the filing was late, or nobody wanted to trade it at any price all day.

That distinction matters because the standard reflexes — forward-fill, drop the row, fill with the column mean — all destroy the message and replace it with something reassuring. And the reassurance points the same way every time, because the reasons data goes missing in markets are overwhelmingly bad news.

Diagnose before you fill

Why it is missing	What it actually means	Reasonable handling
Illiquid, did not trade today	No new information arrived	Carry the last price with a staleness flag; do not treat the 0% as an observation
Trading halted	Material news pending; a large move is coming	Exclude from the tradeable universe until it reopens; never fill
Delisted, acquired, bankrupt	The position ended, usually badly	Apply an explicit Delisting Returns value, then stop the series
Company not yet listed	Nothing is missing	Leave empty. Never backfill
Feed outage or vendor gap	The value existed; you lost it	Cross-check a second source; if unresolved, mark the whole window unusable
Field not reported (no R&D line)	Often a genuine zero, sometimes not	Depends on the filing convention — check, do not guess
Fundamentals between reports	Constant by construction	Forward-fill from the arrival date, which is correct here

The last row is the exception that shows the rule. Forward-filling quarterly fundamentals across the days between filings is right, because the underlying quantity really does not change. Forward-filling a price across a halt is wrong, because the underlying quantity is changing violently and you are recording that it did not.

Worked example: a minimum-variance book that was not

A minimum-variance portfolio over 500 US names. Two years of daily returns, sample covariance with mild shrinkage, long-only, 5% position cap, target volatility 9%.

The data has gaps — halts, thinly traded small caps, and a handful of names that stopped trading months before being formally delisted. The pipeline forward-fills prices, which turns every gap into a run of exact-zero returns. Thirty-eight names are affected.

Look at what the optimiser sees for those 38. Measured daily volatility 0.6% against a universe median of 1.6%. Measured correlation to the market 0.18 against a median of 0.55. On both criteria — low variance, low covariance — they are the best assets in the universe. They are, of course, the worst.

The optimiser puts 31% of the book into those 38 names, several at the 5% cap. Realised volatility of the resulting "minimum variance" portfolio: 19%, against the 9% target, with the overshoot arriving in a handful of days when the halted names reopened.

Verdict: nothing was wrong with the optimiser. It solved the problem it was handed exactly. The forward-fill wrote a false covariance matrix and the optimiser, doing its job, found and maximised the falsehood. This is the general shape of the failure: an optimiser is a machine for locating your worst data.

The other classic fills, and what each one costs

Column-mean or sector-median imputation. Filling a missing book-to-price with the sector median makes the stock look average. Companies with missing fundamentals are systematically not average — they are late filers, recent IPOs, or firms in distress. Worse, if the median is computed over the full sample you have also introduced Look-Ahead Bias, because the 2005 fill used 2020 data.

Dropping incomplete rows. Fine when missingness is unrelated to the outcome, which here it never is. Dropping every name with a gap systematically removes the failures, which is Survivorship Bias arriving through the back door of a dropna() call.

Interpolation. Linear interpolation across a gap uses the value at the end of the gap to fill its middle. That is a time machine, and it is the most common accidental look-ahead in time-series work.