Choosing k in k-nearest-neighbors as a bias–variance dial

Question

In k-nearest-neighbors, a prediction at a point is the average of the responses of its k closest training points. Explain how the choice of k trades bias against variance, what happens at the two extremes, and how you would choose it.

Accepted Answer

Small k (e.g., k = 1). The prediction follows the single nearest neighbor, so it hugs local structure, low bias. But it is at the mercy of that one point's noise, and moving to a slightly different training sample would change predictions a lot, high variance. The fitted surface is jagged and overfits. Large k. The prediction averages many neighbors, some of them far away in feature space, washing out real local structure, high bias. But averaging many points cancels noise, so predictions are stable across training samples, low variance. The surface is over-smoothed and underfits. At the extr…

Choosing k in k-nearest-neighbors as a bias–variance dial

Your answer

More Statistics questions