Designing Gaussian Process Kernels

A Gaussian process (GP) makes predictions by assuming that outputs at nearby inputs are correlated, but "nearby" and "correlated" have to be defined somehow — that definition is the kernel, and it is the single choice that decides everything the model can and cannot express. Pick a kernel that only allows smooth wiggles, and the GP will never model a sharp jump. Pick one that repeats, and it will assume next January looks like last January before seeing any data.

The analogy: a rule for "how much does yesterday tell you about today"

Think of forecasting tomorrow's temperature. You would naturally trust that today's temperature is a strong hint about tomorrow's, a weaker hint about next week's, and almost no hint at all about six months from now — unless you also know it's a seasonal effect, in which case today tells you a lot about the same date next year. A kernel is exactly that intuition written as a formula: it takes two input points and outputs a number saying how strongly their function values should move together, purely as a function of how far apart (or how similarly-timed) they are.

The kernel, symbol by symbol

The workhorse choice is the squared-exponential (RBF) kernel:

$$k(x, x') = \sigma^2 \exp\!\left(-\frac{(x - x')^2}{2\ell^2}\right)$$

Here $\sigma^2$ (the amplitude) sets the overall scale of variation — how far function values typically swing above or below their mean. $\ell$ (the lengthscale) sets how quickly correlation fades with distance: points closer than about $\ell$ apart are treated as strongly linked, points much farther apart as nearly independent. In plain English: two inputs that are close (relative to $\ell$) are assumed to produce similar outputs; far-apart inputs are assumed to know almost nothing about each other. A separate periodic kernel replaces the raw distance with a repeating one, so that inputs exactly one period apart are treated as if they were right next to each other, encoding "this repeats every $p$ units" directly into the covariance.

Worked example 1: correlation from the RBF kernel

Let $\sigma^2 = 1$, $\ell = 2$. Compute the kernel between three points: $x=0$, $x=1$, $x=10$.

$$k(0,1) = \exp\!\left(-\frac{1}{8}\right) = 0.882, \qquad k(0,10) = \exp\!\left(-\frac{100}{8}\right) = 0.0000037$$

In words: a point one unit away keeps 88% of the correlation of being at the same spot, but a point ten units away — five lengthscales — is treated as essentially unrelated. This 3×3 covariance matrix,

$$K = \begin{pmatrix} 1 & 0.882 & 0.0000037 \\ 0.882 & 1 & 0.0000079 \\ 0.0000037 & 0.0000079 & 1 \end{pmatrix},$$

is exactly what the GP uses to decide how much the value at $x=10$ should be pulled by observations at $x=0$ or $x=1$: almost not at all.

Worked example 2: a sum of kernels for trend plus seasonality

A price series is modeled with $k = k_{\text{RBF}}(\ell=30\text{ days}) + k_{\text{periodic}}(p=252\text{ days})$. At a lag of 1 day, the RBF term dominates and gives high correlation (close in time). At a lag of exactly 252 trading days (roughly one year), the RBF term has decayed to near zero, but the periodic term reactivates full correlation because the phase lines up again — the sum lets the GP say "next year, same day, is informative, even though last month is not." Neither kernel alone can express both effects; the sum can, because covariance functions add.

Drag the rate parameter here and watch how quickly the curve decays — that decay shape is exactly what the RBF kernel's lengthscale controls, just applied to distance between two inputs rather than to a single input.

What this means in practice

Kernel choice is model choice for a GP in the same way architecture choice is model choice for a neural network — an RBF-only kernel on seasonal financial data will smooth right through the seasonal pattern it was never told to expect, producing confidently wrong forecasts a year out. Quants building GP-based forecasting or interpolation models spend more time designing and combining kernels (RBF for smooth trend, periodic for known cycles, linear for drift) than they spend on the inference machinery itself.