Experiment Tracking and Model Registries

A researcher tries forty variants of a signal over a few weeks — different lookback windows, different smoothing parameters, different feature sets — and the best one has a backtested Sharpe of 2.1. Three months later, someone asks: which exact parameters produced that number, on which version of the code, on which slice of data? If the answer lives only in the researcher's memory or a scattered pile of notebook cells that have since been overwritten, it's effectively lost. This is the problem experiment tracking solves: a tool (MLflow, Weights & Biases, and similar) that automatically logs, for every run, the parameters used, the code version (usually a git commit hash), the data version, and the resulting metrics — searchable and comparable later without anyone having to remember or re-derive anything.

A model registry is a related but distinct idea: a catalog of trained models — not just the code that produces them, but the actual fitted artifact, weights and all — tagged with a status like "staging," "production," or "archived." Where experiment tracking answers "what did we try and what happened," the registry answers "which specific trained object is the one currently running live, and what's the audit trail of every version that preceded it." A model that scored well in an experiment-tracking log still has to be explicitly promoted into the registry before it's the one actually making trading decisions.

Worked example. A team trains twelve candidate short-horizon volatility forecasters over a month, varying the feature window and regularization strength, each run auto-logged with its parameters, git commit, and out-of-sample RMSE. The tracking tool's dashboard sorts by RMSE and shows candidate #7 (a 20-day window, moderate regularization) as the best, with a one-click link back to the exact commit and config that produced it. The team promotes model #7's serialized artifact into the registry with a "staging" tag, runs it in shadow mode alongside the live model for two weeks to confirm it behaves as expected on fresh data, then flips its tag to "production" — at which point the registry records exactly when that swap happened and which model it replaced, so a later regression can be traced to a specific promotion event rather than a vague "sometime last month."

What this means in practice

Without experiment tracking, comparing forty variants means either trusting memory or re-running all forty from scratch to find the winner again — expensive and error-prone. Without a model registry, "what's running in production right now" is a question someone has to answer by checking a deploy script or asking around, rather than looking it up. Both tools turn tribal knowledge into a queryable record, which matters most exactly when something goes wrong and the first question is "what changed, and when."