Choosing k and Distance Weighting in k-NN

k-NN predicts by looking at the $k$ closest training points to a new query and combining their answers. Two choices decide almost everything: how many neighbors, $k$, to consult, and whether all $k$ get an equal say or the closest ones are trusted more. Get these wrong and k-NN either chases noise or smooths away the local structure that made it worth using.

The analogy: how many neighbors to ask, and how loudly

Ask one neighbor about a new restaurant and you get a fast, specific answer — but if they have unusual taste, you're stuck with their quirk. Ask fifty and you get a stable average — but if you live on a particularly foodie block, that average gets diluted by opinions from people who aren't really representative. Distance weighting is the compromise: ask fifty, but let the ones next door speak louder than the ones three streets over.

The mechanics: k controls smoothness, weighting controls fairness by distance

With $k=1$, the prediction is just the single nearest point's label — low bias, high variance. As $k$ grows, the prediction averages over more points, smoothing the boundary but risking bias once the neighborhood includes points that aren't really local. A distance-weighted vote replaces the plain average:

$$\hat y(x) = \frac{\sum_{i \in N_k(x)} w_i \, y_i}{\sum_{i \in N_k(x)} w_i}, \qquad w_i = \frac{1}{d(x, x_i)} .$$

In plain English: each neighbor's vote is divided by its distance from the query, so a neighbor right next to the query dominates while one at the far edge of the $k$-neighborhood barely counts. This lets you safely use a larger $k$ — cutting noise — without losing sensitivity to the truly closest points.

Worked example: five neighbors, weighted and unweighted

Five nearest historical days to today, next-day returns $+1.2\%, +0.8\%, -0.3\%, +2.0\%, +0.5\%$, distances $0.1, 0.2, 0.4, 1.5, 1.8$. Unweighted average: $0.84\%$ — treating the two distant, less-relevant days the same as the two close ones. Weighting by $1/d_i$ gives weights $10, 5, 2.5, 0.67, 0.56$, and a weighted average of

$$\frac{10(1.2)+5(0.8)+2.5(-0.3)+0.67(2.0)+0.56(0.5)}{10+5+2.5+0.67+0.56} \approx 1.10\% ,$$

leaning more heavily toward the two very close analogues and discounting the two distant, weaker ones.

What this means in practice

Choose $k$ (and whether to weight) by cross-validation, not a rule of thumb. Larger $k$ with distance weighting is usually safer than small unweighted $k$: it protects against one noisy neighbor dominating while still letting the closest points carry most of the weight. Always tune $k$ jointly with the distance metric — a poorly-scaled metric can put irrelevant "near" neighbors ahead of genuinely relevant ones regardless of $k$.