Kernel PCA

Ordinary PCA finds the straight-line directions along which data varies most, which is exactly the wrong tool when the real structure is curved — data sitting on a bent sheet, a spiral, or a horseshoe shape. PCA drawn through a horseshoe finds a straight axis that cuts across the curve, mixing points that are actually far apart along the true shape. Kernel PCA fixes this the same way the kernel trick fixes support vector machines: implicitly lift the data somewhere the structure straightens out, then do ordinary PCA there.

The analogy: flattening a folded map

Imagine a paper map that's been folded in half and creased, then photographed from directly above so the fold is invisible in the photo — two points that are actually far apart on the unfolded map (one on each side of the crease, at the outer edges) can appear close together in the photo, because the fold brought them physically near each other. Measuring "distance" straight off the photo gives wrong answers about the map's true layout. Unfolding the map first — a warp, not a straight-line adjustment — restores the correct distances, and only then does "which direction has the most spread" mean what you want. Kernel PCA is the mathematical unfolding step, done implicitly.

The maths: PCA, but on a kernel matrix

Ordinary PCA finds the eigenvectors of the data's covariance matrix. Kernel PCA instead computes the eigenvectors of the kernel matrix $K$, whose entries are pairwise kernel similarities between all $n$ data points (as in the kernel trick, $K_{ij} = \phi(x_i)\cdot\phi(x_j)$, computed without ever forming $\phi$ explicitly):

$$K v = n\lambda v$$

In words: this looks just like an eigenvalue problem for ordinary PCA, except the matrix being decomposed is built from kernel similarities instead of raw covariances — so its eigenvectors describe the directions of most variation in the implicitly-lifted space, not the original one. Before this step, $K$ must be centred (its rows and columns shifted so the implicit lifted data has zero mean), a bookkeeping detail that's easy to skip and quietly gives wrong answers if you do. Projecting a point onto the $k$-th kernel principal component is then a weighted sum of kernel similarities to every training point, weighted by that eigenvector.

Worked example 1: a horseshoe, straight PCA versus kernel PCA

Consider data shaped like a horseshoe: points trace a U starting at $(0,0)$, curving up and over, ending at $(2,0)$. The two endpoints are close in raw Euclidean distance (both near $y=0$) but very far apart along the true curve. Ordinary PCA's first component would run roughly horizontally through the middle of the U, projecting the two endpoints to nearly the same value — exactly backwards, since they're the two extremes of the shape. An RBF-kernel PCA instead measures similarity by closeness along local neighbourhoods; nearby points along the curve get high kernel similarity, distant points along the curve (even if geometrically close) get low similarity because no short path connects them through dense data. The resulting first kernel component tends to track position along the curve instead, correctly separating the two endpoints.

Worked example 2: a tiny 3-point kernel matrix

Three points where kernel similarity (already centred) gives $K = \begin{pmatrix} 2 & 1 & -1 \\ 1 & 2 & -1 \\ -1 & -1 & 2\end{pmatrix}$. Points 1 and 2 are highly similar to each other ($K_{12}=1$) and both dissimilar to point 3 ($K_{13}=K_{23}=-1$). The dominant eigenvector of this matrix (by inspection, since rows 1 and 2 are symmetric) is proportional to $(1, 1, -2)$ — projecting point 3 to a very different value from points 1 and 2, which sit close together. That single kernel principal component alone separates $\{1,2\}$ from $\{3\}$, which is exactly the "curved cluster structure" a linear PCA on raw coordinates might have blurred if points 1, 2, 3 happened to look collinear in the original space.

The explorer above shows how a linear map stretches a circle along its eigenvector directions — kernel PCA finds the analogous "directions of most spread," but after an implicit nonlinear warp, which is why it can separate shapes a linear stretch never could.

The steep decay of an RBF-style kernel with distance, shaped like the curve here, is what lets kernel PCA respect a curved manifold's local neighbourhoods instead of being fooled by raw straight-line distance.

What this means in practice

Kernel PCA is useful as a preprocessing step for nonlinear structure — untangling regime clusters that live on a curved manifold in factor space, or as a nonlinear alternative to standard dimensionality reduction before clustering or visualisation. Unlike ordinary PCA, its components don't correspond to simple weighted combinations of the original features, so "component 1 means X" explanations that work for linear PCA don't translate directly.