Unit Tests for Feature Transforms

A feature-transform function computes a 20-day rolling z-score, gets refactored slightly for performance six months after it was written, and the refactor accidentally introduces an off-by-one error in the rolling window. Nothing crashes. The model keeps training, keeps producing predictions, and performance quietly degrades over the following weeks — degradation that gets blamed, at first, on "the market regime changing" rather than on the actual cause: a silent bug in one feature's code. Unit tests exist precisely to catch this class of bug at the moment it's introduced, not months later through a vague performance complaint.

The idea: test the function, not the whole pipeline

A unit test checks one small piece of code — a single feature-transform function — against a known, hand-computed correct answer, run automatically on every code change. Instead of "does the whole pipeline produce reasonable-looking output," it asks a sharper question: "given this tiny, made-up input, does this exact function produce this exact, hand-verified output?" That specificity isolates a bug immediately: if the z-score test fails, the bug is in the z-score function, not somewhere among the other several hundred features that also ran.

The tests are cheap to write and run precisely because they don't need real market data — a handful of made-up numbers, small enough to compute by hand, is enough to pin down behavior on ordinary cases and on edge cases that are easy to get subtly wrong (insufficient history at the start of a series, a division that could hit zero, a missing input value).

Worked example: a test that catches the off-by-one bug directly

Suppose the rolling z-score function computes, for each day, the current value minus the trailing-20-day mean, divided by the trailing-20-day standard deviation. A unit test constructs a tiny synthetic series — say, 25 known values — hand-computes what day 21's z-score should be using a 20-day window ending the day before, and asserts the function's output matches to a tight tolerance.

If the refactor introduces an off-by-one error — the window ends on the current day, silently including a value the feature shouldn't see yet — the test fails immediately, since the output on the synthetic series no longer matches the hand-computed value. The failure points directly at the z-score function, caught in code review before it ever reaches training, let alone production. Compare that to the no-test world: the bug ships silently, degraded performance shows up weeks later, and someone works backward through hundreds of features to find which one broke — days of debugging instead of minutes.

Read that curve as "cost to fix a bug" against "time since it was introduced": a unit test catches the bug at the leftmost point on that curve, where the fix is cheapest — a code review comment — instead of somewhere far to the right, where the same bug now requires untangling weeks of degraded live performance.

What this means in practice

The highest-value tests target the parts of a transform most likely to be subtly wrong: boundary conditions (the first rows of a series, before a window has enough history), unusual inputs (missing data, zero variance, ties), and any logic recently touched by a refactor. Small, deliberately constructed synthetic inputs with a hand-verified answer are what make a test fast, deterministic, and genuinely diagnostic when it fails.