Linear Discriminant Analysis

Most classifiers you meet first — logistic regression, k-NN — try to directly draw a line between classes. Linear discriminant analysis (LDA) takes a different route: it models what a typical example of each class looks like (its average feature values and how spread out it is), and then classifies a new point by asking which class's typical profile it resembles more closely. Under one specific, common assumption — that both classes share the same spread, just different centers — that comparison collapses into drawing a single straight line.

The analogy: comparing a new fingerprint to two known suspects

Imagine two suspects with known, well-characterized fingerprint patterns, and a new print needs to be matched to one of them. Rather than trying to draw an abstract boundary in "fingerprint space," you'd compare the new print's characteristics to each suspect's typical pattern and see which one it's statistically closer to, accounting for how much natural variation each suspect's own prints normally show. LDA does exactly this for classes of data: model each class's typical shape, then compare a new point against both shapes.

Worked example: two classes, one shared spread

Two classes of trades — "profitable" and "unprofitable" — are described by one feature, signal strength. Profitable trades average $\mu_1 = 8$, unprofitable trades average $\mu_0 = 3$, and (LDA's key assumption) both classes share the same variance, $\sigma^2 = 4$. LDA's discriminant score for a new point $x$ reduces to a linear function:

$$\delta(x) = x \cdot \frac{\mu_1 - \mu_0}{\sigma^2} - \frac{\mu_1^2 - \mu_0^2}{2\sigma^2}$$

Plugging in numbers: $\delta(x) = x \cdot \frac{5}{4} - \frac{64-9}{8} = 1.25x - 6.875$. Classify as "profitable" when $\delta(x) > 0$, i.e. $x > 5.5$ — a single threshold, sitting almost exactly halfway between the two class means (5.5 vs the midpoint 5.5 here, since the shared-variance assumption makes the boundary land at the midpoint whenever classes are equally likely). A new trade with signal strength $x = 6$ crosses that threshold and gets classified as profitable.

Picture the curve above as two overlapping bell shapes (one per class) instead of a single parabola — LDA effectively slides a threshold between two such bumps, and because both bumps are assumed to have the same width, that threshold is always a single straight cut rather than a curve, which is the key geometric fact that makes the boundary linear.

What this means in practice

LDA works well and needs little data when its shared-variance assumption roughly holds, and it also naturally extends to more than two classes and can be used for dimensionality reduction (finding the direction that best separates classes), not just classification. When classes clearly have different spreads — one tight cluster, one wide sprawling one — the shared-variance assumption is wrong, and quadratic discriminant analysis, which lets each class have its own spread and draws a curved boundary instead, usually fits better.