Support Vector Regression

Ordinary least squares regression penalizes every deviation from the fitted line, no matter how small — a prediction off by $0.01 contributes to the loss just as a prediction off by $100 does, only less. Support vector regression (SVR) starts from a different premise: define a tolerance band of width $\epsilon$ around the fitted line, and treat any error inside that band as free — zero penalty. Only errors that spill outside the tube get penalized, and only in proportion to how far outside they land. The result is a fit that's deliberately insensitive to small noise and shaped almost entirely by the points that genuinely can't be squeezed inside the tube.

The analogy: a garden hose laid across uneven ground

Imagine laying a flexible garden hose across a bumpy lawn so that as much of the lawn's surface as possible stays within a fixed distance of the hose — small bumps within that tolerance need no attention at all, the hose just passes near them without complaint. Only the spots where the ground rises or falls by more than the tolerance actually force the hose to bend. SVR fits a regression line (or curve, with a kernel) exactly the same way: it's shaped by the handful of stubborn points that can't be squeezed inside the $\epsilon$-tube, while the many points that already sit comfortably inside contribute nothing to the final shape.

The loss function

SVR minimizes the $\epsilon$-insensitive loss:

$$L_\epsilon(y, \hat y) = \max(0, |y - \hat y| - \epsilon)$$

Plain English: if the prediction is within $\epsilon$ of the true value, the loss is exactly zero; beyond that, the loss grows linearly with how far outside the tube the point sits. The full optimization (like the classification SVM) balances this loss against a regularization term controlling the fitted function's flatness, with a hyperparameter $C$ trading off tube violations against model simplicity — and, as with classification, a kernel can be swapped in to fit a curved rather than linear tube.

Worked example 1: which points get penalized

With $\epsilon = 0.5$, consider four residuals (true minus predicted): $0.2$, $-0.4$, $0.9$, $-1.3$. Loss for each: $\max(0, 0.2-0.5)=0$; $\max(0, 0.4-0.5)=0$; $\max(0, 0.9-0.5)=0.4$; $\max(0, 1.3-0.5)=0.8$. Two of the four residuals, both comfortably inside the $0.5 tolerance band, contribute exactly zero to the loss and therefore exert zero pull on where the fitted line sits — only the two points that spilled outside the tube (contributing 0.4 and 0.8) are the ones actually shaping the fit.

Worked example 2: widening the tube changes which points matter

Same four residuals, but widen $\epsilon$ to $1.0$: losses become $\max(0,0.2-1)=0$; $\max(0,0.4-1)=0$; $\max(0,0.9-1)=0$; $\max(0,1.3-1)=0.3$. Now three of four points are inside the tube and contribute nothing — only the $-1.3$ residual still matters, and even that one's penalty shrank from 0.8 to 0.3. Widening $\epsilon$ doesn't just shrink penalties uniformly; it can flip a point from "shapes the fit" to "ignored entirely," which is why $\epsilon$ is the single most consequential hyperparameter in SVR — it directly controls how many points the model treats as noise versus signal.

What this means in practice

SVR is useful precisely where you believe small prediction errors are just noise that shouldn't influence the model at all — fitting an implied-volatility curve where quoted prices carry bid-ask noise below a certain size, or a fair-value curve where sub-tick deviations are meaningless. It's less commonly reached for than ridge or gradient-boosted trees in most quant pipelines, mainly because tuning both $\epsilon$ and $C$ (and a kernel bandwidth, if nonlinear) well requires more careful cross-validation than a single-hyperparameter method, and because, like the classification SVM, the raw fitted function doesn't come with built-in uncertainty estimates the way a Bayesian or tree-ensemble model can provide.