Sequential Minimal Optimization

Training a support vector machine means solving a quadratic optimization problem over all the Lagrange multipliers at once, one per training point, subject to a linear equality constraint. For thousands of training points, solving that directly with a general quadratic-programming solver is slow and memory-hungry. Sequential minimal optimization (SMO) sidesteps this by never solving for more than two multipliers at a time.

The trick

Because the SVM constraint ties all the multipliers together through one linear equation, you can't update just a single multiplier while holding the others fixed, that would violate the constraint. But updating exactly two multipliers together while holding the rest fixed keeps the constraint satisfied, and with only two free variables the subproblem has a closed-form analytic solution: no inner quadratic-programming solver is needed at all. SMO repeatedly picks a pair of multipliers that violates the optimality (KKT) conditions the most, solves that pair's tiny subproblem exactly, and repeats until every pair satisfies the conditions.

Worked example

With multipliers $\alpha_1$ and $\alpha_2$ selected for update and everything else fixed, the constraint $\alpha_1 y_1 + \alpha_2 y_2 = \text{const}$ pins $\alpha_1$ as a linear function of $\alpha_2$. Substituting that relationship into the objective turns it into a one-dimensional quadratic in $\alpha_2$ alone, which is minimized by a single formula and then clipped to stay within the box constraints $[0, C]$. This closed-form update, repeated over millions of pairs, is what let SVMs scale to datasets far larger than a general QP solver could handle directly.