Linear SVM vs Logistic Regression

Given two classes of points that can roughly be separated by a straight line, both a linear support vector machine (SVM) and logistic regression will draw very similar-looking lines. But they get there by optimizing genuinely different things, and that difference shows up clearly once points sit far from the boundary or near it. Logistic regression asks every single point, "given where the boundary is, how probable is your class?" and adjusts the boundary to make all of those probability judgments as accurate as possible. The linear SVM asks a narrower question: "are you on the correct side, with enough margin?" and mostly ignores points that already comfortably satisfy that.

The analogy: a strict bouncer versus a probability-quoting analyst

A linear SVM behaves like a bouncer enforcing a velvet rope: anyone safely inside the rope's buffer zone gets zero further attention, but anyone crossing the line or standing too close to it gets pushed back — the bouncer's whole effort concentrates on the borderline cases (the support vectors), and people standing comfortably in the middle of the room could double their distance from the rope and nothing about the bouncer's decision would change. Logistic regression is more like an analyst who assigns everyone a probability of belonging to one side or the other: someone deep in the middle of the room still gets an assigned probability (close to 100%), and if that person moved slightly, the analyst's overall fitted probabilities would shift too, at least a little — every point continues to matter, not just the borderline ones.

The loss functions, side by side

Let $z = y (w\cdot x + b)$ be the "signed margin" for a point with true label $y\in\{-1,+1\}$. Logistic regression minimizes the log loss:

$$L_{\text{log}}(z) = \log(1 + e^{-z})$$

The linear SVM minimizes the hinge loss:

$$L_{\text{hinge}}(z) = \max(0, 1-z)$$

Plain English: log loss is smooth and never truly reaches zero — even a point correctly classified with huge margin ($z=10$) still contributes a tiny, nonzero penalty and a tiny, nonzero gradient, so it keeps nudging the boundary, however slightly. Hinge loss hits exactly zero once $z \ge 1$ — a point correctly classified with margin at least 1 contributes nothing at all, zero gradient, and has no further influence on the fitted boundary. That's the mathematical source of "SVM only cares about the borderline points."

Worked example 1: comparing losses for three points

Three points with margins $z = -0.5$ (misclassified), $z = 0.5$ (correct but close), $z = 3$ (correct and far). Hinge loss: $\max(0, 1.5) = 1.5$; $\max(0, 0.5)=0.5$; $\max(0, -2) = 0$ — the far point contributes exactly zero. Log loss: $\log(1+e^{0.5})\approx0.974$; $\log(1+e^{-0.5})\approx0.474$; $\log(1+e^{-3})\approx0.049$ — nonzero for every point, however small for the far one. Only the SVM's loss actually zeroes out; logistic regression's asymptotically approaches zero but never gets there, so it keeps a faint pull from every point in the dataset.

Worked example 2: how an outlier moves each boundary

Suppose fitting on 20 well-separated points gives both models nearly the same boundary. Add one extreme outlier far on the correct side, at $z=20$: the SVM's hinge loss is $\max(0,1-20)=0$ exactly, so this point contributes literally nothing to the optimization and the boundary is unchanged. Logistic regression's log loss is $\log(1+e^{-20})\approx 2\times10^{-9}$ — vanishingly small but not zero, contributing a negligible but technically nonzero gradient that in principle still nudges the fit, in a way the SVM's identical-looking point would not.

Add points far from the boundary on the correct side and watch whether the boundary itself moves — under a hinge-loss (SVM-style) model those points would leave the boundary completely untouched, while a smooth log-loss model keeps a faint, ever-shrinking sensitivity to them.

What this means in practice

Choose the linear SVM when you want a decision boundary that's robust to well-separated, non-borderline points and don't need calibrated probabilities — just a clean side/no-side call, useful for a hard go/no-go filter. Choose logistic regression when you need actual probability estimates (position sizing proportional to confidence, or feeding a probability into a downstream expected-value calculation) since its output is a genuine (if sometimes miscalibrated) probability, whereas an SVM's output is just a raw, unnormalized distance from the boundary that requires extra calibration (e.g. Platt scaling) to turn into anything probability-like.