Running Hyperparameter Sweeps at Scale

A model has a learning rate, a regularization strength, a number of hidden layers, and a dropout rate — settings that aren't learned by training but have to be chosen beforehand, and that can each make or break performance. Trying every combination of even five values for each of those four settings is $5^4 = 625$ full training runs. At an hour per run on a GPU, that's 625 GPU-hours for a single sweep — a real budget decision, which is why how you search matters as much as what you search over.

The idea: search smarter than brute force

Grid search — trying every combination on a fixed grid — wastes compute in a specific way: if only two of four hyperparameters actually matter, grid search still spends equal effort on variations of the two that don't, treating every combination as equally worth trying.

Random search samples combinations randomly from a range instead of a fixed grid, and for the same compute budget explores the important dimensions more thoroughly, since it isn't wasting grid points on dimensions that don't matter. Bayesian optimization goes further, using earlier runs' results to choose the next combination to try — spending less time in regions that already looked bad and more refining promising ones, the way you'd narrow a search after seeing a few data points.

A separate and often bigger lever is early stopping within the sweep: killing a training run partway through if its validation curve is already clearly worse than the best run so far, rather than letting every configuration run to completion. Since most bad hyperparameter combinations reveal themselves as bad well before training finishes, this alone can cut total sweep compute dramatically.

Worked example: budgeting a sweep

A team has a budget of 100 GPU-hours and wants to tune two hyperparameters: learning rate (continuous, roughly log-scale from 0.0001 to 0.1) and dropout rate (continuous, 0 to 0.5). A grid search with 10 values per dimension needs $10 \times 10 = 100$ full runs; if each run takes one hour, that exactly exhausts the budget with zero room for early stopping, mistakes, or a second sweep on a different architecture choice.

Switching to random search with early stopping changes the arithmetic: run 200 configurations, but kill any run whose validation loss after 20% of training is already worse than the current best. Most random draws land in mediocre regions and get killed early — say 150 of 200 runs stop at 0.2 hours instead of 1, while 50 promising ones run to completion. Total compute: $150 \times 0.2 + 50 \times 1 = 80$ GPU-hours — twice as many configurations explored, for less total compute than the exhaustive grid.

Treat that curve loosely as "best validation performance found so far" against "number of configurations tried": it rises quickly at first as random search finds decent regions, then flattens — which is exactly the signal that tells a team when a sweep has hit diminishing returns and further runs aren't worth the compute.

What this means in practice

The practical skill isn't picking the fanciest search algorithm — random search with early stopping already captures most of the benefit over grid search — it's setting a sensible search range for each hyperparameter (too narrow and you miss the good region, too wide and you waste runs) and staying disciplined about killing bad runs early rather than burning the budget on runs already clearly losing.