Soft-Margin SVMs and the C Parameter

The original support vector machine demands that every training point stay outside a corridor of a fixed width. On paper that is elegant. On real data it is unusable, because a single mislabelled row on the far side of the gap makes the whole problem infeasible — there is no line at all that satisfies the constraints, and the optimiser returns nothing. Even when the data is separable in principle, one noisy point can drag the boundary halfway across the plane to accommodate itself.

The soft margin fixes this by letting points trespass and charging them for it. The knob that sets the charge is called $C$, and it is the single most consequential number you will pick when fitting an SVM.

The mental picture is a parking fine. The corridor is a no-parking zone down the middle of the road. A low fine means the council leaves the zone wide and tolerates a few cars in it. A high fine means nobody dares park there, so the council narrows the zone until it fits between the parked cars. Neither is right or wrong; it is a policy choice about which nuisance you mind more.

The maths, one symbol at a time

Each training point $i$ gets its own slack $\xi_i \ge 0$, measuring how far inside the corridor it has strayed. Zero means it is safely outside; a value between 0 and 1 means it is inside the corridor but still on the correct side; anything above 1 means it is misclassified outright. The constraints relax from "score at least 1" to "score at least 1 minus your slack", and the objective picks up a bill:

$$\min_{w,\,b,\,\xi} \;\; \tfrac{1}{2}\lVert w \rVert^2 \;+\; C\sum_{i} \xi_i \qquad \text{s.t.} \quad y_i\,(w \cdot x_i + b) \ge 1 - \xi_i, \quad \xi_i \ge 0$$

In plain English: keep the weights small — which is the same as keeping the corridor wide, since its width is $2/\lVert w \rVert$ — and pay $C$ for every unit of trespass. The two terms pull in opposite directions and $C$ sets the exchange rate between them.

Written the other way round, the same problem is ordinary regularised loss minimisation:

$$\min_{w,\,b} \;\; \sum_i \max\big(0,\; 1 - y_i(w\cdot x_i + b)\big) \;+\; \frac{1}{2C}\lVert w \rVert^2$$

In plain English: minimise hinge loss plus a penalty on the size of the weights. That makes $1/(2C)$ the familiar ridge-style shrinkage parameter, so $C$ is inverse regularisation strength. Large $C$ means almost no shrinkage.

Worked example 1: what one outlier costs

Five points on a single feature axis. Four are well behaved: $x = 1$ and $x = 2$ are class $-1$; $x = 4$ and $x = 5$ are class $+1$. The fifth, at $x = 2.5$, is labelled $+1$ but sits deep in negative territory. The classifier is $f(x) = wx + b$.

Option A — keep the wide corridor and pay the fine. Fit on the four clean points. The corridor edges land on $x = 2$ and $x = 4$, giving $2w + b = -1$ and $4w + b = +1$. Subtracting, $2w = 2$, so $w = 1$ and $b = -3$. Check the odd point: $f(2.5) = -0.5$, and since $y = +1$ it needs a score of at least $+1$, so its slack is $\xi = 1 - (-0.5) = 1.5$. The bill is

$$\tfrac{1}{2}(1)^2 + C(1.5) \;=\; 0.5 + 1.5\,C$$

The corridor width is $2/\lvert w \rvert = 2$.

Option B — squeeze the corridor so nobody trespasses. Make the odd point a corridor edge alongside $x = 2$: now $2.5w + b = +1$ and $2w + b = -1$, so $0.5w = 2$, giving $w = 4$ and $b = -9$. Every point is now outside, all slacks are zero, and the bill is $\tfrac{1}{2}(16) = 8$. The corridor width has collapsed to $2/4 = 0.5$.

Now compare at two settings:

• $C = 0.1$: Option A costs $0.5 + 0.15 = 0.65$, Option B costs 8. The SVM shrugs and keeps the wide street.
• $C = 10$: Option A costs $0.5 + 15 = 15.5$, Option B costs 8. The SVM contorts for one point.

The switch happens at $0.5 + 1.5C = 8$, so $C = 5$. Below that the outlier is ignored; above it, the outlier rewrites the model and the corridor gets four times narrower. Drag the flexibility control below through the same arc — the boundary goes from too rigid to sensible to fenced around individual points.

Worked example 2: C does not survive a change in sample size

Here is a trap that bites people who tune $C$ on a subsample. The penalty term $\tfrac{1}{2}\lVert w \rVert^2$ does not grow with the number of rows. The slack term $C\sum_i \xi_i$ does — every extra row adds another slack to the pile. So the same $C$ means less regularisation on a bigger dataset.

Take a dataset of 500 rows and two candidate solutions:

solution	$\tfrac{1}{2}\lVert w \rVert^2$	total slack at $n = 500$	cost at $C = 1$
smooth $w$	8	60	$8 + 60 = 68$
tight $w'$	32	42	$32 + 42 = 74$

The smooth solution wins. Now double the data with more rows from exactly the same distribution. Each solution's total slack doubles, while its weight penalty is unchanged:

• smooth: $8 + 1 \times 120 = 128$
• tight: $32 + 1 \times 84 = 116$

The tight solution now wins, and nothing about the problem changed except the row count. To recover the earlier answer you must halve the fine: at $C = 0.5$ the smooth solution costs $8 + 60 = 68$ and the tight one $32 + 42 = 74$, restoring the original ranking. The practical rule is that the useful $C$ scales roughly like $1/n$, so a value tuned on a two-year pilot will be far too aggressive when you refit on twenty years.

What this means in practice

Tune $C$ on a logarithmic grid — something like $10^{-3}$ to $10^{3}$ in powers of ten — because the interesting range spans orders of magnitude and a linear grid wastes almost all its points. With a curved kernel you must search $C$ and the bandwidth jointly (see The RBF Kernel and Bandwidth Selection), since a narrow kernel and a large $C$ overfit in the same direction and can partly cancel each other during a one-at-a-time search.

The solution tells you where each point stands, through the dual weights $\alpha_i$ described in The SVM Dual and Support Vectors. Points outside the corridor get $\alpha_i = 0$ and could be deleted without changing anything. Points exactly on the edge get $0 < \alpha_i < C$. Points that trespassed are pinned at $\alpha_i = C$. Counting how many sit at the cap is a fast diagnostic: if most of your training set is pinned there, $C$ is too small and the model has essentially given up.

On financial data the honest default is a small $C$. Signal-to-noise is low, labels are frequently ambiguous near the boundary, and a model that insists on classifying every training day correctly is fitting the noise. Start small, raise it only if validation error genuinely improves.