Architecture of a Quant ML Stack

Ask a researcher what a quant ML stack needs and the first answers are usually about models: which gradient-boosting library, whether to use a neural network, how to tune hyperparameters. In practice, model choice is rarely where a working system succeeds or fails. It's the layers underneath the model — how raw data becomes a feature, how a training set is assembled and preserved, how the trained model gets from a researcher's notebook into a system that trades real money — where correctness actually lives or dies. A stack can use a perfectly reasonable model and still produce leaked, unreproducible, or silently-wrong results because one of these layers was built casually.

The layers, and what each one is actually for

Feature computation. Raw data (prices, fundamentals, alternative data) is transformed into model-ready features. The critical property here is point-in-time correctness: a feature computed "as of" a given date must only use information genuinely available by that date (see Point-in-Time Feature Computation Engines). This layer is where look-ahead bias is born or avoided, and it is worth building once, carefully, rather than re-implemented per signal.

Training data assembly. Features and labels are joined into a training set for a specific model run. This snapshot needs to be immutable and versioned — if the underlying feature tables are later corrected or backfilled, a model trained last month should still be exactly reproducible against the data as it existed then, not silently re-computed against revised history (see Immutable Training-Set Snapshots).

Model training and evaluation. This is the layer researchers spend the most attention on — architecture, cross-validation scheme, hyperparameters — and it deserves that attention, but it is only as trustworthy as the two layers beneath it.

Serving and inference. The trained model has to produce predictions in production, ideally using a feature-computation path that is provably consistent with the one used in training — if training used a batch job computing a feature one way and production computes the "same" feature slightly differently, the model sees inputs at inference time it never actually learned from (see Offline/Online Feature Parity).

A worked example

A desk's stack computes a 20-day trailing volatility feature two different ways: a nightly batch job for training data, which uses a vectorised calculation over a stored price history table, and a live feature service for production inference, which recomputes it incrementally from a streaming price feed as the trading day progresses.

During a period of unusually choppy trading, the model's live predictions start diverging noticeably from what backtesting on the same dates would have produced. Investigation traces the gap to the volatility feature: the batch job uses close-to-close prices exclusively, while the streaming feature service, built independently by a different team, incorporates the current day's intraday high alongside the previous close — a reasonable-sounding choice in isolation, but one that makes the live feature systematically different from what the model was trained on, especially on volatile days when intraday and close-to-close volatility diverge most.

The fix isn't a better model — it's collapsing the two feature-computation code paths into one, so training and inference provably use the identical calculation, verified by a test that feeds the same historical inputs through both training-time and inference-time code and asserts the outputs match exactly.

Think of small inconsistencies between layers as compounding rather than staying isolated: a feature discrepancy that seems minor on a calm day gets amplified precisely on the volatile days a strategy most needs to get right, similar to how small rate differences compound into large gaps over time on this curve.

Where stacks break in practice

Most breakage doesn't happen at launch — it happens months later, when a well-intentioned change to one layer (a faster feature-computation rewrite, a schema migration, a new data vendor) is made without re-verifying consistency with the layers around it. Because each layer usually has a different owner (data engineering, research, trading infrastructure), a change that's locally correct for its own layer can silently break a downstream layer's assumption, and nothing catches it until performance degrades unexplainably.