Random Search vs Grid Search

Choosing hyperparameters like learning rate and regularization strength means searching over combinations you can't evaluate without actually training the model each time — an expensive way to explore. Grid search picks a handful of values for each hyperparameter and tries every combination: a natural-feeling, exhaustive-looking approach. Random search instead samples hyperparameter combinations randomly from the same ranges, the same number of times. The surprising finding, both theoretical and empirical, is that random search usually finds a better setting for the same evaluation budget — because most hyperparameters don't matter nearly as much as one or two of them do, and grid search wastes its budget spreading evenly across all of them.

The analogy: searching for a signal along the important street

Imagine two hyperparameters as two streets crossing at right angles, and somewhere along the north-south street (and nowhere in particular along the east-west one) is where a hidden signal is strongest. A grid search with 3 values per street tests only 3 distinct positions along the important north-south street, no matter how many points you add to the unimportant east-west one — because every one of the grid's points shares those same 3 north-south values, just repeated at different east-west positions. A random search with the same total number of tries places its points at many different positions along both streets, so it tests far more distinct positions along whichever street actually matters — without needing to know in advance which one that is.

Why the effective resolution differs

Suppose a search budget allows $N$ total trials over $d$ hyperparameters, and (as is common in practice) only $k$ of those $d$ hyperparameters actually affect performance much. Grid search with $N$ trials and $d$ dimensions typically uses roughly $N^{1/d}$ distinct values per dimension — so as $d$ grows, the number of distinct values tested per important dimension shrinks fast. Random search, by contrast, always tests $N$ distinct values along every dimension, important or not, because each trial's coordinate along each axis is drawn independently:

$$\text{grid: distinct values per axis} \approx N^{1/d}, \qquad \text{random: distinct values per axis} = N$$

In words: grid search's resolution along any one axis degrades as more hyperparameters are added, because the same trial budget has to be spread across every combination; random search's resolution along any one axis stays at the full trial count $N$ regardless of how many other hyperparameters are being searched alongside it.

Worked example 1: grid search's shrinking resolution

Budget of $N=64$ trials, searching $d=3$ hyperparameters with a grid. An even grid uses $64^{1/3}=4$ values per hyperparameter ($4\times4\times4=64$ combinations). If only 1 of those 3 hyperparameters actually matters, grid search has only tested 4 distinct values of the one that matters — the other 48 combinations tested three different, irrelevant hyperparameters at those same 4 values.

Worked example 2: random search's full resolution

Same budget, $N=64$ random trials over the same 3 hyperparameters, each drawn independently from its range. Along the one hyperparameter that matters, random search has effectively tested 64 distinct values (with high probability, since continuous draws rarely repeat exactly) — sixteen times finer resolution along the dimension that actually determines performance, for the identical total compute budget of 64 trials.

Choosing what shape to sample each hyperparameter from — uniform, log-uniform, normal — is itself part of designing a search; the explorer above shows how a distribution's shape controls where samples concentrate, the same choice you make when deciding how to draw random hyperparameter values.

What this means in practice

Random search is now a common default baseline for hyperparameter tuning specifically because it's simple, parallelizes trivially (every trial is independent), and empirically tends to beat grid search at the same budget, especially once you're searching more than two or three hyperparameters at once. It is itself often outperformed by more adaptive methods like Bayesian optimization or successive halving, but as a baseline it is a large step up from grid search at essentially no extra implementation cost.

Practice

1. With a budget of 100 trials over 4 hyperparameters, how many distinct values per hyperparameter does an even grid test?
2. Why does random search's per-axis resolution not degrade as more hyperparameters are added to the search?
3. Name one hyperparameter search scenario where a small, evenly-spaced grid might actually be preferable.