The SVM Dual and Support Vectors

A support vector machine finds the boundary separating two classes with the widest possible margin — sounds like every training point should matter. It doesn't: solving the SVM's optimization problem in its dual form reveals the boundary depends only on a small subset of points — the support vectors — sitting on or inside the margin. Every other point could be deleted without changing the answer.

The analogy: the boundary is set by whoever's closest to the fence

Two crowds separated by a rope, placed as far from both as possible: to decide exactly where the rope goes, you don't survey everyone — you find whoever in each crowd stands closest to the other crowd and place the rope between those few people. Everyone standing further back could walk away and the rope wouldn't move. Those few closest people are the support vectors.

The mechanics: from the primal problem to the dual

The primal problem minimizes $\|w\|^2$ subject to every point being correctly classified with margin at least 1. Using Lagrange multipliers $\alpha_i \ge 0$ per point, the problem rewrites entirely in terms of the $\alpha_i$'s:

$$\max_{\alpha} \ \sum_i \alpha_i - \frac{1}{2}\sum_{i,j} \alpha_i \alpha_j y_i y_j (x_i^\top x_j), \qquad \text{s.t. } 0 \le \alpha_i \le C, \ \sum_i \alpha_i y_i = 0 .$$

In plain English: only points on or inside the margin end up with $\alpha_i > 0$; every point comfortably on the correct side drops out with $\alpha_i=0$. The boundary is $w = \sum_i \alpha_i y_i x_i$, summed only over support vectors. The dual only ever needs dot products $x_i^\top x_j$ — never raw features directly — which is exactly what lets the kernel trick swap in a similarity function for the dot product and get a nonlinear boundary for free.

Worked example: which of five points are support vectors

Five 1-D points separating "trending" (class +1, signal values $3, 4, 8$) from "mean-reverting" (class −1, values $-2,-1,1$). The widest-margin boundary sits between the closest pair across classes — here $3$ and $1$, since they're nearer each other than any other cross-class pair. Solving the dual, only $3$ and $1$ get $\alpha_i>0$; $4$, $8$, $-2$, $-1$ all get $\alpha_i=0$. Delete those four points and refit: the boundary lands in exactly the same place.

What this means in practice

Support vectors are why SVMs can be memory-efficient at prediction time — store only the support vectors, not the whole training set — and why the model ignores points far from the boundary, unlike least-squares, where every point tugs at the fit. The flip side: the decision only "listens" to the hardest, most boundary-adjacent cases, so a mislabeled point near the boundary can distort the whole surface.