Experiment Tracking for Quant Research

A quant analyst presents a strategy with a backtested Sharpe of 2.1. Nobody in the room can answer how many other variants were tried before this one — different lookbacks, different universes, different entry filters — because the failed attempts were never logged, only deleted or overwritten. Without that count, the 2.1 can't be evaluated at all. A Sharpe of 2.1 found on the first try means something completely different from a Sharpe of 2.1 that was the best of 200 tries, and the difference is entirely in a record nobody kept.

What tracking actually captures

Experiment tracking means every backtest run — successful or not, published or discarded — gets logged automatically with its exact parameters, the code version and data snapshot it used, and its full result, not just the headline number the researcher chose to highlight. The point isn't bookkeeping for its own sake. It's that the count and distribution of all attempts, including the failures nobody wanted to write down, is the only way to correct a final result for how much search went into finding it — see The False Strategy Theorem for the statistics of exactly how much that correction matters.

Worked example

Two desks each report a strategy with a backtested Sharpe of 2.0.

Desk A has full experiment tracking: the log shows this was variant 4 out of 6 tried, all reasonably similar and all clearing Sharpe 1.2 or better. With only 6 correlated attempts, the multiple-testing correction to the reported Sharpe is small.

Desk B has no tracking, but informal recollection suggests "a good number of things were tried" before landing here — nobody can say whether that was 10 or 400. Absent a real count, the only defensible move is to assume the worst case consistent with how the research process actually runs, which for an unconstrained search over that much history is often in the hundreds. Applying a multiple-testing correction at even a conservative 150 independent trials collapses the effective Sharpe for Desk B's result to under 0.5.

	Logged attempts	Correction basis	Adjusted Sharpe
Desk A	6 (all recorded)	Known, small	~1.7
Desk B	Unknown	Assumed worst case	<0.5

Identical headline number, entirely different conclusion, and the only thing separating them is whether a log exists.

What good tracking looks like day to day

The mechanics matter less than the automation. A researcher manually pasting results into a spreadsheet will, without any intent to deceive, paste the interesting ones more reliably than the boring ones — a failed run with a Sharpe of 0.1 doesn't feel worth the two minutes it takes to log by hand, and that small daily bias compounds into a badly incomplete record over a quarter. The fix is making logging a side effect of running the backtest at all, not a separate step: the harness itself writes the parameters, the code commit, the data snapshot reference, and the result to an append-only store the instant the run finishes, with no researcher action required and no way to skip it. Tools built for this (MLflow, Weights & Biases, or an in-house table with the same shape) all do the same three things — capture inputs, capture outputs, make deletion hard — and any of them beats a spreadsheet, because the spreadsheet's core failure mode is that it only remembers what a human decided was worth remembering.

Tracking tooling (MLflow-style experiment logs, or a simple database table written to automatically) is a prerequisite for the deflated Sharpe ratio and similar multiple-testing corrections described in The Deflated Sharpe Ratio — those corrections need a trial count as an input, and experiment tracking is where that count has to come from. See Researcher Degrees of Freedom for the fuller catalogue of unlogged choices this practice is meant to close off.