Bayesian Optimization for Model Search

Tuning hyperparameters — a learning rate, a tree depth, a regularization strength — is like drilling for oil across a field with no map, where every test well costs a fortune (here, a full training run that might take hours). Grid search drills a fixed pattern regardless of what earlier wells found. Random search drills at random. Neither uses what it just learned to decide where to drill next. Bayesian optimization does: after every well it updates a belief about where the oil likely is, and picks the next well based on that belief, balancing "drill where I already found oil" against "drill somewhere unknown, in case it's better."

The two ingredients

The surrogate model is a cheap stand-in for the real, expensive experiment. It is a probabilistic model — most commonly a Gaussian process — fitted to the (hyperparameter setting, validation score) pairs tried so far. Crucially, it does not just predict a single expected score at an untested setting; it predicts a range of plausible scores, wide where few nearby points have been tried and narrow where many have. Write the surrogate's belief at a candidate setting $x$ as a mean $\mu(x)$ and an uncertainty $\sigma(x)$.

The acquisition function turns that belief into a decision about where to test next. A widely used one is Expected Improvement:

$$\text{EI}(x) = \mathbb{E}\big[\max(0,\ f(x) - f^{*})\big]$$

In words: at candidate setting $x$, how much better does the surrogate expect the score to be than $f^{*}$, the best score found so far — averaged over all the outcomes the surrogate currently thinks are plausible, and counting only the outcomes where $x$ actually beats the current best. A setting can score high on EI either because $\mu(x)$ is already high (looks good) or because $\sigma(x)$ is large (unknown, so it might be great) — this is what balances exploiting known good regions against exploring uncertain ones.

Think of the curve here as one slice of the surrogate's mean prediction as a hyperparameter varies — the optimizer is not just chasing the current peak, it is also weighing the width of uncertainty around untested regions of the curve, which a single mean line like this cannot show but which the acquisition function accounts for underneath.

Worked example 1: choosing the next point by hand

Three settings have been tried: learning rate $0.01$ scored $0.80$, $0.10$ scored $0.85$ (the current best, $f^{*}=0.85$), $0.50$ scored $0.60$. The surrogate, having seen these three points, produces beliefs for two untested candidates:

• $x = 0.05$: near a good point, so the surrogate is fairly confident, $\mu = 0.86$, $\sigma = 0.03$.
• $x = 0.25$: far from any tested point, so the surrogate is unsure, $\mu = 0.75$, $\sigma = 0.15$.

Naive "exploit only" search picks $x=0.05$ because $0.86 > 0.75$. But Expected Improvement weighs the chance of beating $0.85$, not just the mean. For $x=0.05$, the mean is barely above $f^*$ and the spread is narrow, so there's decent but modest probability of improvement, roughly landing around $\text{EI} \approx 0.02$. For $x=0.25$, the mean is below $f^*$, but the spread is five times wider — there's real probability mass above $0.85$ in that wide range even though the average sits below it — giving something like $\text{EI} \approx 0.04$. Bayesian optimization tests $x = 0.25$ next, precisely because it is the one grid or random search would also try, but for a principled reason: not blind luck, but "we genuinely don't know, and not-knowing here carries more upside than a small guaranteed gain."

Worked example 2: why it beats grid search on a budget

Suppose the true best learning rate sits at $0.03$, and a grid search is budgeted 5 evaluations across $\{0.01, 0.1, 0.2, 0.3, 0.5\}$ — none land near $0.03$, so grid search reports $0.1$ as best (score $0.85$) and never discovers $0.03$ exists (true score there might be $0.93$). Bayesian optimization with the same budget starts the same way, but after seeing $0.01\to0.80$ and $0.1\to0.85$, its surrogate's mean rises going from $0.01$ toward $0.1$, so the acquisition function pulls the 4th evaluation between them, say $0.05$, scoring $0.90$ — already better than grid search's best, and it closes in further with its 5th evaluation. The advantage isn't magic; every evaluation is chosen using everything learned from every prior one, instead of a schedule fixed in advance.

What this means in practice

Quant teams use Bayesian optimization to tune anything where one "evaluation" means retraining a model on years of data — tree depth and learning rate, network architecture, regularization strength — because each evaluation costs real wall-clock time and each wasted grid point is a sunk run. It's far less useful for a single ridge penalty, where evaluating a hundred candidates costs milliseconds and random search finds the answer just as fast, with none of the surrogate-fitting overhead.