Testing Machine-Learning Code

A feature-generation function can pass every unit test — correct types, correct shapes, no exceptions — and still silently leak tomorrow's price into today's row. Ordinary software testing checks that code does what you told it to do. Machine-learning testing has to additionally check that what you told it to do is not quietly wrong, because a model degrades gracefully: it keeps producing numbers, just worse ones, and nothing crashes to tell you.

That's the core difficulty. A null pointer exception announces itself. A subtly leaked feature, a mismatched train/serve transform, or a label that got shifted by one day does not — it just makes the backtest look better than it should, and you find out in production, at a cost.

The three layers

Unit tests cover the deterministic parts: a function that computes a rolling z-score should give the exact expected number on a hand-built input of five rows. This is ordinary software testing and it's necessary but not sufficient — it says the arithmetic is right, not that the arithmetic is being fed the right thing.

Data tests check the pipeline's assumptions about its own input: no nulls where nulls shouldn't appear, a price column that's always positive, a date column strictly increasing per instrument, a feature's distribution not silently shifting between training runs. These catch the failure mode where the code is correct but the world changed underneath it — a vendor renamed a column, split-adjusted a price differently, or backfilled a value that wasn't available at the time.

Behavioural tests probe the model itself, not the code around it: build a handful of synthetic inputs where you know what the answer should be — a stock with textbook momentum, a spread that should mean-revert — and assert the model's output moves in the right direction. This is the layer that catches a model trained on leaked data, because a leaked model often gets behavioural tests wrong in ways accuracy metrics don't reveal, since the leak inflates historical accuracy without producing sensible reasoning.

Worked example: a leak that unit tests miss

Suppose a feature computes each stock's rank versus its sector average return, intended to use only information available at the start of the trading day. The function is unit-tested against a small hand-built table and returns the right ranks — it passes.

The bug: the sector average was computed over the same day's closing prices, not the prior day's. Every unit test built from a static snapshot passes, because the test data doesn't distinguish "average computed from data available at 9:30am" from "average computed from data available at 4:00pm" — both produce the same numbers on a fixed table. Only a data test that checks the feature's timestamp against the label's timestamp — asserting the feature uses no row with a date greater than or equal to the prediction date — catches it. Add that assertion as an automated check and the leak fails loudly on every future pipeline run, instead of only showing up as an unexplained live/backtest gap months later.

Where this bites in practice

Run data tests on every scheduled pipeline execution, not just at development time — vendors change formats without announcing it. Keep a small, hand-curated set of behavioural test cases per model and re-run them after every retrain; a model that starts failing a case it used to pass is a retrain regression, and without the test you'd only notice once it costs money. Version the test data alongside the code, since a data test is only as good as the fixture it's checked against.