Designing Feature Pipeline DAGs

A mid-sized quant shop might compute a few hundred features from raw prices, fundamentals, and alternative data — a 20-day moving average, a sector-relative earnings surprise, a momentum z-score that itself depends on the moving average. If you write this as one long script, every small change (a bug fix in the moving average, a new data vendor) forces you to rerun the entire thing from scratch, and it becomes nearly impossible to know which downstream features are even affected by an upstream change. The fix used across the industry is to stop thinking of feature computation as a script and start thinking of it as a graph.

The idea: features as nodes, dependencies as edges

A DAG (directed acyclic graph) represents each feature as a node and each "feature A needs feature B" relationship as an arrow from B to A. "Directed" means the arrows have direction (raw price feeds the moving average, not the reverse); "acyclic" means there are no loops — nothing can depend on itself, even indirectly, or the computation would never finish. Once your pipeline is expressed this way, a scheduler can look at the graph and figure out, automatically, the correct order to compute things in, and — critically — exactly which nodes need to rerun when one upstream node changes.

Think of it like a spreadsheet with formulas: change one cell, and only the cells whose formulas reference it recalculate. A feature pipeline DAG is that same idea applied to hundreds of "cells" running on a schedule across a cluster instead of instantly on your desktop.

Worked example: tracing an update through a small DAG

Suppose the graph is: raw_price → daily_return → 20d_momentum → momentum_zscore, and separately raw_price → 20d_volatility, with momentum_zscore also depending on sector_average_momentum, which depends on 20d_momentum for every stock in the sector.

If a data vendor sends a corrected raw_price for one ticker on one date, the DAG tells you the blast radius: daily_return for that ticker must recompute, then 20d_momentum, then sector_average_momentum for the whole sector (it aggregates across tickers), then momentum_zscore for every ticker in that sector — even ones whose own raw price never changed. 20d_volatility is untouched, since it isn't downstream of daily_return on any path. Without an explicit graph, a team either reruns everything (slow) or reruns by guesswork (risky — someone forgets momentum_zscore touches the whole sector, and a bug survives in production for months).

Picture that curve as "cost to recompute" against "number of downstream features affected" — a DAG scheduler is what lets you stay near the bottom-left of that curve, touching only what changed, instead of always paying for a full recompute regardless of how small the input change was.

What this means in practice

Tools like Airflow, Dagster, and dedicated feature stores exist largely to make this graph explicit and machine-readable rather than living only in an engineer's head. Writing the DAG down also documents the pipeline: a new team member can see, in one picture, which raw inputs ultimately feed a given signal. The discipline that matters most is keeping edges honest — if a feature secretly reads an undeclared input, the scheduler won't know to recompute it when that input changes, and you get silently stale features that look fine until a backtest disagrees with production.