The Curse of Dimensionality in Smoothing

A kernel smoother or nearest-neighbor estimator predicts a point's value by averaging nearby observations. With one input variable, "nearby" is easy to find: on a number line, your ten closest points are genuinely close. The curse of dimensionality is the fact that this stops being true as you add more input variables — the same ten "nearest" points end up scattered far away, and the smoother quietly stops smoothing anything meaningful.

The reason is geometric. To cover the same fraction of the data in a higher-dimensional space, a neighborhood has to stretch out along every added axis simultaneously. Doubling the number of input variables doesn't double the space you need to search — the volume needed to capture, say, 10% of the data grows so fast that even a huge dataset leaves most of that region nearly empty.

As a rough illustration: to capture 10% of the data by volume along a single dimension, you need a neighborhood covering 10% of that axis's range. To capture the same 10% of the data spread evenly over 10 dimensions, the neighborhood must extend across roughly 0.1^(1/10) ≈ 80% of the range on every one of those ten axes — hardly "local" anymore. The neighbors your smoother finds are no longer close in any meaningful sense; the estimate becomes either too noisy (if you insist on staying local) or too biased (if you widen the neighborhood enough to find real neighbors).

This is why nonparametric smoothing — kernel regression, local averaging, k-nearest-neighbors — is a small-input-count tool. Beyond a handful of predictors, practitioners either fall back on parametric models (which impose structure instead of estimating it from local data), restrict smoothing to a few dimensions at a time (additive models), or use methods designed for high dimensions, like tree ensembles, that don't rely on literal spatial closeness.