The RBF Kernel and Bandwidth Selection

A linear model can only draw a straight boundary. The obvious workaround — invent squared terms, cubed terms and cross-products, then fit a line in that bigger space — collapses under its own weight: forty features already generate 820 pairwise products, and you have to guess which degree to stop at. The kernel approach sidesteps the whole construction. Instead of writing down new features, you write down a similarity function between two rows and let the algorithm work entirely through that. The radial basis function kernel is by far the most used of these, and the reason it works or fails almost always comes down to one number.

The analogy: a scent that fades with distance

Picture each training example as a small perfume bottle sitting somewhere on a map. When a new point arrives, it sniffs. Bottles right next to it smell strong; bottles across the map smell of nothing. The model's answer at that point is a vote among the bottles, weighted by how strongly each one registers.

Now ask the only question that matters: how far does the scent carry? If it carries for miles, every bottle smells the same everywhere, the vote is identical at every location, and the model predicts one constant. If it carries a centimetre, a new point smells nothing at all unless it is sitting on top of a training example, and the model has become a lookup table that memorises its training set and shrugs at everything else. That distance is the bandwidth, and choosing it well is the whole job.

The formula

For two rows $x$ and $x'$, the RBF kernel is

$$k(x, x') \;=\; \exp\!\left(-\gamma \, \lVert x - x' \rVert^2\right)$$

In plain English: measure the straight-line distance between the two rows, square it, multiply by a positive number $\gamma$ (gamma), and exponentiate the negative of that. The result is 1 when the two rows are identical and slides toward 0 as they separate — a similarity score between 0 and 1.

The same kernel is often written with a length scale $\sigma$ instead:

$$k(x, x') \;=\; \exp\!\left(-\frac{\lVert x - x' \rVert^2}{2\sigma^2}\right), \qquad \gamma = \frac{1}{2\sigma^2}$$

In plain English: $\sigma$ is the bandwidth measured in the same units as your features — the distance over which similarity meaningfully decays. It is the reciprocal of $\gamma$ up to a factor of two, which is the source of endless confusion, so fix the direction now: large $\gamma$ means small $\sigma$ means a scent that dies fast means a wiggly model.

Read the horizontal axis of the explorer below as squared distance between two rows and the curve as their similarity. The rate knob is exactly $-\gamma$: push it toward $-1$ and watch similarity collapse to nothing almost immediately; ease it toward zero and everything starts looking like everything else.

Worked example 1: what gamma does to two actual rows

Two standardised feature vectors: $x = (1.0, -0.4)$ and $x' = (0.2, 0.5)$. Their difference is $(0.8, -0.9)$, so

$$\lVert x - x' \rVert^2 \;=\; 0.8^2 + (-0.9)^2 \;=\; 0.64 + 0.81 \;=\; 1.45$$

Now evaluate the kernel at three settings of $\gamma$:

$\gamma$	exponent $-\gamma \lVert x-x' \rVert^2$	$k(x,x')$	what the model sees
0.1	$-0.145$	$0.865$	near-duplicates
1	$-1.45$	$0.235$	related but distinct
10	$-14.5$	$0.0000005$	complete strangers

Two rows that are a moderate distance apart go from 87 percent similar to effectively unrelated across a hundredfold change in one parameter. Now extend that to the whole kernel matrix and both failure modes become obvious:

• $\gamma$ far too large. Every off-diagonal entry is essentially zero, so the matrix is the identity. Each training point is similar only to itself, the fitted model puts a tiny spike on each one, training error goes to zero, and out-of-sample predictions revert to the default class. This is memorisation, and it looks fantastic until you test it.
• $\gamma$ far too small. Every entry is essentially 1, so the matrix has one repeated row and carries a single degree of freedom. Every point looks the same, so the model predicts the same thing everywhere.

Slide the flexibility control below from one extreme to the other; $\gamma$ moves the RBF model along exactly this arc.

Worked example 2: picking a bandwidth with the median heuristic

You need a starting value, and the standard one is the median heuristic: set the bandwidth so that a typical pair of points has a meaningful but not overwhelming similarity. Concretely, take the median of all pairwise squared distances and set $\gamma$ to its reciprocal.

Five standardised observations on one feature: $-1.6,\; -0.5,\; 0.1,\; 0.7,\; 1.9$. The ten pairwise squared distances are

$$1.21,\; 2.89,\; 5.29,\; 12.25,\; 0.36,\; 1.44,\; 5.76,\; 0.36,\; 3.24,\; 1.44$$

Sorted, they run $0.36,\, 0.36,\, 1.21,\, 1.44,\, 1.44,\, 2.89,\, 3.24,\, 5.29,\, 5.76,\, 12.25$. With ten values the median is the average of the fifth and sixth:

$$\text{median} \;=\; \frac{1.44 + 2.89}{2} \;=\; 2.165, \qquad \gamma \;=\; \frac{1}{2.165} \;=\; 0.462$$

Check what that buys you. At the median distance the kernel evaluates to $\exp(-0.462 \times 2.165) = \exp(-1) = 0.368$. That is the whole idea in one line: the median heuristic sets the bandwidth so that an average pair of your points scores $1/e$ in similarity — not 1, not 0, but somewhere the kernel can still tell things apart.

Compare with the other common default. Scikit-learn's gamma="scale" uses one over the number of features times their variance. Those five points have mean $0.12$, squared deviations summing to $6.848$, and hence variance $6.848 / 5 = 1.370$, so $\gamma = 1 / (1 \times 1.370) = 0.730$. The two recipes land within a factor of 1.6 of each other, which is the honest verdict on both: they get you to the right order of magnitude, and then you search.

What this means in practice

Standardise before you fit, without exception. The kernel only ever sees $\lVert x - x' \rVert^2$, a raw sum of squared feature differences. Leave one column in raw prices and another in basis points and the price column supplies essentially all the distance — you have silently thrown the rest of your features away. Fit the scaler on training data only, or you have leaked.

Search gamma and C together on a log grid. Both control flexibility, so a good $\gamma$ at one $C$ can be a terrible one at another. The standard sweep is a two-dimensional grid in powers of ten, roughly $\gamma \in [10^{-4}, 10^{1}]$ against $C \in [10^{-2}, 10^{3}]$, scored by cross-validation. On time-ordered data use a time-aware split, never a shuffled one.

Watch out in high dimensions. With many features, all pairwise distances concentrate around a similar value — the The Curse of Dimensionality in its most practical form. Every kernel entry then collapses toward the same number and the kernel loses its ability to discriminate. This is why RBF SVMs are far more useful on ten well-chosen features than on four hundred raw ones.

Mind the cost. The kernel matrix is $n \times n$, so training scales somewhere between $n^2$ and $n^3$ and prediction costs one kernel evaluation per support vector. Past roughly fifty thousand rows this stops being practical, and the usual answer is to approximate the kernel with Nystrom Approximation and Random Fourier Features and fit a linear model in the approximate feature space.