Kernel Ridge Regression

Plain ridge regression fits a straight line, penalizing large coefficients to avoid overfitting — reliable, fast, closed-form, but limited to linear relationships. Kernel ridge regression keeps every one of ridge regression's good properties — the same closed-form solution, the same well-understood regularization — while letting the fitted relationship curve arbitrarily, by applying the kernel trick: replace every dot product in ridge regression's math with a kernel evaluation, and the whole machine now fits nonlinear functions without ever explicitly constructing the (possibly infinite-dimensional) features the kernel implicitly represents.

The analogy: the same recipe, richer ingredients

Ordinary ridge regression is a recipe that combines a small set of ingredients (the raw features) in fixed, linear proportions, with a penalty discouraging any one ingredient from dominating. Kernel ridge regression uses the exact same recipe — the same "combine and penalize dominance" structure — but swaps in a vastly richer, effectively unlimited pantry of implicit ingredients (all the nonlinear feature combinations the kernel represents), while the recipe's logic never has to change. You get the flexibility of a much bigger ingredient list without ever needing to write out what all those ingredients actually are.

The model

Standard ridge regression solves $\hat\beta = (X^TX + \lambda I)^{-1}X^Ty$. The representer theorem guarantees the same optimization, written entirely in terms of a kernel $k$, has the equivalent closed form

$$\alpha = (K + \lambda I)^{-1} y, \qquad \hat f(x) = \sum_{i=1}^n \alpha_i\, k(x, x_i)$$

Plain English: instead of solving for a coefficient per feature (as in plain ridge), kernel ridge regression solves for a coefficient per training point, using the Gram matrix $K$ (all pairwise kernel evaluations) in place of $X^TX$. This is still a single matrix inversion — no iterative optimization, no local minima — just performed in a different, $n$-dimensional coordinate system instead of the original feature space.

Worked example 1: linear kernel recovers plain ridge

With $k(x,x')=x\cdot x'$ (the linear kernel), kernel ridge regression is mathematically identical to plain ridge regression — just computed via the dual formula instead of the primal one. On 2 points $x_1=1,x_2=2$, $y=(3,5)$, $\lambda=0.5$: $K=\begin{pmatrix}1&2\\2&4\end{pmatrix}$, $K+0.5I=\begin{pmatrix}1.5&2\\2&4.5\end{pmatrix}$, determinant $=1.5(4.5)-2(2)=6.75-4=2.75$. Inverting and multiplying by $y$ gives $\alpha \approx (0.36, 0.945)$ (solving the $2\times2$ system directly). Predicting at $x^*=1.5$: $\hat f(1.5) = 0.36\,k(1.5,1)+0.945\,k(1.5,2) = 0.36(1.5)+0.945(3)=0.54+2.835=3.375$ — the same answer plain ridge regression would give solving directly for a single slope coefficient, just reached via the $n$-dimensional dual route.

Worked example 2: RBF kernel fits a curve plain ridge cannot

Now use the RBF kernel $k(x,x')=e^{-\gamma(x-x')^2}$ with $\gamma=1$ on the same setup but a nonlinear target: $x=(0,1,2)$, $y=(0, 1, 0)$ — a bump shape no straight line can fit well. Plain linear ridge regression would fit a compromise line with substantial error at every point (predicting something like a flat-ish line near 0.33 everywhere, missing the bump entirely). Kernel ridge regression with the RBF kernel, by contrast, can place weight $\alpha$ concentrated near $x_2=1$ so predictions rise near $x=1$ and fall back toward zero away from it — reproducing the bump shape, something achievable only because the RBF kernel implicitly represents infinitely many nonlinear basis functions, none of which needed to be written down explicitly.

The RBF kernel's shape is itself a decaying exponential in squared distance — dragging this curve's parameters mirrors how a wider or narrower kernel bandwidth $\gamma$ changes how sharply kernel ridge regression's fit can bend around individual training points.

What this means in practice

Kernel ridge regression is a natural choice when the true relationship is smoothly nonlinear, the dataset is small to moderate (its cost scales with $n^3$ for the matrix inversion, so it becomes impractical much past a few thousand points without approximations), and you want a fully closed-form, deterministic fit rather than an iteratively-optimized one like a neural network or gradient-boosted trees. It shows up in fitting smooth yield curves, implied volatility surfaces, and other settings where a smooth, kernel-controlled curve is preferable to a rigid parametric form, and where you have few enough points that the cubic cost of the matrix solve stays manageable.