Sequential Minimal Optimization

Training a support vector machine means solving a quadratic optimization problem over all the Lagrange multipliers at once — one per training point — subject to a linear equality constraint. For thousands of training points, solving that directly with a general quadratic-programming solver is slow and memory-hungry. Sequential minimal optimization (SMO) sidesteps this by never solving for more than two multipliers at a time.

The trick

Because the SVM constraint ties all the multipliers together through one linear equation, you can't update just a single multiplier while holding the others fixed — that would violate the constraint. But updating exactly two multipliers together while holding the rest fixed keeps the constraint satisfied, and with only two free variables the subproblem has a closed-form analytic solution: no inner quadratic-programming solver is needed at all. SMO repeatedly picks a pair of multipliers that violates the optimality (KKT) conditions the most, solves that pair's tiny subproblem exactly, and repeats until every pair satisfies the conditions.

Worked example

With multipliers $\alpha_1$ and $\alpha_2$ selected for update and everything else fixed, the constraint $\alpha_1 y_1 + \alpha_2 y_2 = \text{const}$ pins $\alpha_1$ as a linear function of $\alpha_2$. Substituting that relationship into the objective turns it into a one-dimensional quadratic in $\alpha_2$ alone, which is minimized by a single formula and then clipped to stay within the box constraints $[0, C]$. This closed-form update, repeated over millions of pairs, is what let SVMs scale to datasets far larger than a general QP solver could handle directly.