Designing a Hyperparameter Search Space

Any hyperparameter search — grid, random, or something more adaptive — only ever explores the region you told it to explore. If the true best learning rate is $0.0003$ but your search space runs from $0.01$ to $0.1$, no algorithm will ever find it, no matter how clever. Search-space design is the unglamorous decision that happens before any searching: what range should each hyperparameter span, and on what scale should values be drawn — because for many common hyperparameters, "evenly spaced" and "evenly useful" are not the same thing at all.

The analogy: a map that isn't drawn to a useful scale

Imagine using a single ruler to measure both the width of a room and the width of a continent — a scale useful for the room is useless for the continent, and vice versa. Learning rates behave similarly: whether a rate is $0.001$ or $0.01$ matters enormously (a factor of 10 in effective step size), but whether it's $0.501$ or $0.510$ barely matters at all, even though both pairs differ by roughly the same absolute amount ($0.009$ versus $0.009$). A search space that spaces its samples evenly in absolute terms — a linear scale — wastes most of its samples on tiny, unimportant absolute differences at the large end of the range, and starves the small end, where a factor of 10 can live in a sliver of absolute space.

Linear versus log scale

For a hyperparameter like learning rate or weight decay, where what matters is the ratio between values rather than their difference, sample on a log scale: draw $u$ uniformly from some range, then set the hyperparameter to $10^u$ (or $e^u$).

$$\eta = 10^{u}, \qquad u \sim \text{Uniform}(a, b)$$

In words: instead of sampling the learning rate $\eta$ directly and evenly, sample its exponent $u$ evenly over a range $(a,b)$, then exponentiate — this spreads samples evenly across orders of magnitude ($0.0001$, $0.001$, $0.01$, $0.1$ all get similar sampling density) rather than evenly across raw values (which would put almost every sample between $0.01$ and $0.1$ if the range were $(0.0001, 0.1)$ sampled linearly). Hyperparameters like dropout rate or a mixing coefficient bounded between $0$ and $1$, where the actual magnitude (not the ratio) is what matters and the whole range is roughly equally interesting, are usually fine sampled on a plain linear scale instead.

Worked example 1: linear sampling wastes the small end

Learning rate range $(0.0001, 0.1)$, 10 samples drawn uniformly on a linear scale. Roughly 9 of the 10 samples will land above $0.01$ (since $0.01$ to $0.1$ is 90% of that linear range), and perhaps 0 or 1 will land below $0.001$ — even though the region between $0.0001$ and $0.001$, a whole order of magnitude, might contain the actual best value. Linear sampling here systematically starves the small-value region of any real search attention.

Worked example 2: log sampling spreads evenly across scales

Same range $(0.0001, 0.1)$, sampled on a log scale: $u \sim \text{Uniform}(-4, -1)$ (since $10^{-4}=0.0001$ and $10^{-1}=0.1$), then $\eta = 10^u$. Ten samples of $u$ spread evenly across $(-4,-1)$ land, roughly, two to three per unit interval — meaning about 2–3 samples fall in $(0.0001, 0.001)$, another 2–3 in $(0.001, 0.01)$, and another 2–3 in $(0.01, 0.1)$: every order of magnitude gets comparable search attention, instead of nearly all of it going to the top decade as in the linear case.

Sampling a hyperparameter's exponent from a uniform distribution and exponentiating it is one specific case of the general idea the explorer above shows: the shape of the distribution you draw from determines where your samples actually land.

What this means in practice

Learning rate, weight decay, and most regularization strengths are conventionally searched on a log scale; dropout probability, momentum coefficients close to but below 1 (often reparameterized as $1-\text{momentum}$ on a log scale for the same reason), and mixing ratios are searched linearly or with their own natural reparameterization. Getting this wrong doesn't cause an error — it silently wastes most of a search budget on a region of the space that was never going to matter, which is why it's worth deciding deliberately rather than defaulting to "just sample uniformly" for every hyperparameter without checking whether that hyperparameter behaves multiplicatively or additively.

Practice

1. You want to search weight decay between $10^{-6}$ and $10^{-2}$. What uniform range for the exponent $u$ would you sample from?
2. Why is dropout probability (bounded between 0 and 1, no notion of "orders of magnitude") usually fine on a linear scale?
3. In one sentence, why does a badly chosen search space undermine even a very good search algorithm?