Pipeline Orchestration and DAGs

A model-training pipeline is really a chain of dependent steps: pull raw data, clean it, compute features, split into train and validation sets, train the model, evaluate it, and only then deploy. Each step needs the previous one to have finished successfully, and some steps (like pulling price data and pulling fundamentals data) don't depend on each other and can run in parallel. Representing this as a directed acyclic graph (DAG) — nodes are tasks, directed edges are "must finish before" dependencies, and no cycles are allowed, since a task can't depend on its own future output — gives an orchestrator exactly the information it needs to run the pipeline correctly and efficiently.

What the orchestrator actually does

Given the DAG, an orchestration tool (Airflow, Dagster, Prefect and similar systems are common choices) figures out a valid execution order automatically, runs independent branches in parallel to save time, retries a failed task a configured number of times before giving up, and — critically — knows exactly which downstream tasks need to be rerun if an upstream task's output changes, because that's just everything reachable from that node in the graph. This last property is what separates a real orchestrator from a plain script: if the feature-computation step is rerun with corrected logic, the orchestrator knows the training and evaluation steps downstream are now stale and need rerunning too, without a human having to trace that dependency manually.

Worked example

A weekly retraining pipeline has this dependency structure: "pull prices" and "pull fundamentals" run in parallel with no dependency on each other (roughly 12 minutes each); both must finish before "compute features" (8 minutes), which must finish before "train model" (25 minutes), which must finish before "evaluate" (5 minutes) and "canary deploy" (3 minutes). Run naively step by step, total time is 53 minutes even though the two pulls could overlap; an orchestrator that recognizes their independence runs them in parallel instead, cutting wall-clock time to 41 minutes. One week, the fundamentals-pull task fails twice on a vendor timeout; the orchestrator retries with a backoff delay, succeeds on the third attempt, and because "compute features" depends on that task, correctly waits rather than running on stale or partial fundamentals data — a mistake a simple cron-scheduled script without dependency awareness could easily make.

What this means in practice

Designing a pipeline as an explicit DAG rather than a linear script pays off the first time something fails at 3 a.m. — the orchestrator retries only the failed task and everything depending on it, not the whole pipeline from scratch, and a human checking the dashboard can see precisely which step broke and what it's blocking, rather than reading through a monolithic log file to reconstruct the same information.