Sweep Result Databases and Model Leaderboards

After running a 200-configuration hyperparameter sweep, the immediate question is "which one won, and by how much?" — but a close second question, one that matters far more over the following months, is "six weeks from now, will anyone remember we already tried this exact combination and it didn't work?" If sweep results live only as scattered log files or a researcher's memory, both questions get harder to answer every week, and teams quietly waste compute re-running experiments whose outcome was already known.

The idea: treat experiments like data, not like exhaust

A sweep result database records every training run's hyperparameters, final metrics, and enough metadata (data snapshot used, code version, random seed) to make each row self-contained and comparable to every other row, structured so it can be queried rather than read one log file at a time. Instead of "ctrl-F through log files from last month," the question "show me every run with dropout above 0.3 that beat our current production model" becomes an ordinary database query.

A leaderboard is the natural view on top of that database: runs ranked by the metric that matters, with the production model's score marked as the bar to beat. Its value isn't just showing the winner — it's making the history of what's been tried visible to the whole team, so a new hire can see at a glance that "wider hidden layer, same learning rate" was already tried and made things worse.

Worked example: a leaderboard preventing wasted effort

Suppose a validation Sharpe ratio of 1.2 is the current production score. A junior researcher wants to try doubling hidden layer width, suspecting more capacity helps. Querying the database for "hidden layer width ≥ current × 2" turns up three prior runs, all with validation Sharpe between 0.9 and 1.05 — worse than production, logged eight months ago by someone checking for underfitting, not chasing performance. That's a five-minute query that saves a full GPU-day of retraining and redirects attention toward a direction not already shown not to work.

Without the database, that run either goes unfound (redundant work happens) or turns up by luck in a Slack thread, with no guarantee its hyperparameters, data snapshot, or code version actually match what's proposed now. The database's structure — one row per run, uniform columns, queryable — is what makes the comparison trustworthy rather than anecdotal.

Read that curve loosely as "cumulative useful sweep coverage" against "number of runs logged": every run adds to a shared, growing map of what's been tried, and that curve only keeps climbing if runs are recorded somewhere queryable — a run that only exists in someone's terminal history contributes nothing to that shared map.

What this means in practice

The database is only as useful as its discipline: every run, including the "obviously bad" ones, should be logged with the same fields — hyperparameters, metric, data snapshot reference, code version, seed — since it's precisely the unremarkable runs someone will want to search for later. Teams that only log winners lose exactly the information (what doesn't work, and why) that would save the most redundant compute.