Gaussian Process Regression

Most regression models spit out one number per input and stay silent about how much to trust it. A model interpolating between two well-observed points and a model extrapolating far past the edge of its data can produce identical-looking point predictions while deserving wildly different confidence. Gaussian process regression is built to answer both questions at once: what's the best guess, and how sure should you be.

The analogy: a rope pinned at known points

Picture an infinitely flexible, slightly stiff rope draped so it must pass exactly through a handful of known data points — the observed prices, the known measurements. Between two pins close together, the rope has very little freedom; it's almost pinned down, so you can predict its height there with confidence. Far from any pin, out past the edge of the data, the rope could sag or rise almost anywhere, and your confidence collapses. A Gaussian process is the mathematical version of "every possible rope consistent with the pins," averaged and measured for spread at every point.

The maths: a distribution over functions

A Gaussian process is defined by a mean function (usually taken as zero, for simplicity) and a covariance function $k(x, x')$ that says how correlated the curve's height is at two input points $x$ and $x'$:

$$f \sim \mathcal{GP}\big(0, \, k(x, x')\big)$$

In words: instead of picking one curve, we describe an entire cloud of plausible curves, and $k$ controls how "wiggly" they're allowed to be — points close together in $x$ have highly correlated heights (the curve can't jump), points far apart are nearly independent. Given observed data, prediction at a new point $x_*$ produces a mean and a variance:

$$\mu_* = k_*^\top K^{-1} y, \qquad \sigma_*^2 = k(x_*,x_*) - k_*^\top K^{-1} k_*$$

Here $y$ is the vector of observed outputs, $K$ is the matrix of covariances between all pairs of observed points, and $k_*$ is the vector of covariances between the new point and each observed point. In words: the predicted mean $\mu_*$ is a weighted blend of the observed values, weighted by how correlated they are with the new point; the predicted variance $\sigma_*^2$ starts at the curve's natural variance and shrinks by however much the observed data pins it down. Far from any data, $k_*$ is near zero, nothing shrinks the variance, and the model correctly reports "I don't know."

Worked example 1: two points, predict the middle

Suppose we've observed $y=10$ at $x=0$ and $y=14$ at $x=2$, using an RBF-style covariance where nearby points correlate strongly. Predicting at $x=1$, exactly halfway, symmetry makes both training points equally relevant, so the predicted mean is close to their average: $\mu_* \approx 12$. The predicted variance at $x=1$ is small, because $x=1$ sits between two observations that strongly constrain it — the rope can barely move there.

Worked example 2: extrapolation blows up the uncertainty

Same two points, but now predict at $x=10$, far outside the observed range $[0,2]$. With an RBF covariance, correlation decays fast with distance, so $k_*$ for both training points is now nearly zero — neither observation says much about $x=10$. The predicted mean drifts back toward the prior mean (often zero, or flat), and critically the predicted variance $\sigma_*^2$ grows back up toward $k(x_*,x_*)$, its unconstrained value. Concretely, if the interpolated variance at $x=1$ was, say, $0.3$, the extrapolated variance at $x=10$ might be $10\times$ larger — the model is explicitly telling you "trust this prediction far less."

Watch how an estimate narrows as more data accumulates in the explorer above — a Gaussian process does the same thing locally: its uncertainty band narrows wherever observations cluster and stays wide everywhere else.

The RBF covariance's exponential decay with distance (shown in shape here) is exactly what controls how fast a Gaussian process's confidence collapses as you move away from a data point.

What this means in practice

Gaussian processes are prized wherever knowing "how confident should I be" matters as much as the point estimate itself: pricing illiquid instruments by interpolating from nearby liquid ones, Bayesian optimisation of trading-strategy hyperparameters, or building a smooth, uncertainty-aware yield curve from sparse market quotes. The catch is cost — the $K^{-1}$ in the formulas above is a matrix inversion that scales roughly with the cube of the number of data points, which makes vanilla Gaussian processes impractical much past a few thousand observations without approximations.