Quadratic and Regularized Discriminant Analysis

You want to classify a stock as "likely to be acquired" or "not," using several financial characteristics at once. Linear discriminant analysis (LDA) draws a single straight boundary between the two groups, implicitly assuming both groups' characteristics scatter in the same shape around their centers — same spread, same correlations, just shifted. That assumption is often false: acquisition targets might cluster tightly on some characteristics while non-targets scatter widely, or the correlations between characteristics might differ entirely. Quadratic discriminant analysis (QDA) lets each group have its own covariance shape, drawing a curved boundary that fits this better — but that flexibility must be estimated from data, and with limited data per group, QDA can overfit badly, exactly the problem regularized discriminant analysis manages.

An analogy: fitting a fence around two flocks of birds

Imagine two flocks of birds in a field, and you want to fence them apart. If both flocks scatter in the same oval shape, just centered differently, a straight fence works well — that's LDA. But if one flock scatters in a wide oval and the other in a narrow, tilted oval, a straight fence cuts through both; a curved fence shaped to each flock's actual scatter separates them far better — that's QDA. The catch: drawing that curved fence accurately needs enough observed birds in each flock to see its true shape. With only a handful per flock, the "shape" you estimate is mostly noise, and a curved fence fit to noise looks great on birds you've already seen but fails on new ones.

The mechanics, one symbol at a time

Let $\mu_k$ and $\Sigma_k$ be the mean vector and covariance matrix of group $k$'s characteristics. LDA assumes all groups share one common covariance matrix ($\Sigma_k = \Sigma$ for every $k$), which makes the decision boundary between any two groups linear. QDA drops that assumption, estimating a separate $\Sigma_k$ for each group, which makes the boundary quadratic (curved) — it can bend to fit each group's actual shape. The classification rule for both compares each group's discriminant score; for QDA this is

$$\delta_k(x) = -\tfrac{1}{2}\log\det(\Sigma_k) - \tfrac{1}{2}(x-\mu_k)^\top \Sigma_k^{-1}(x-\mu_k) + \log \pi_k,$$

where $x$ is the observation's characteristic vector, $\pi_k$ is the prior probability of group $k$, and an observation is assigned to whichever group gives the highest $\delta_k(x)$. In plain English: each group "claims" an observation based on how close it is to its own center, measured in its own units of spread and correlation, adjusted for how common the group is and how spread out it tends to be. Because each $\Sigma_k$ has its own full parameter set, QDA needs enough data per group to estimate a separate covariance matrix reliably — with $p$ characteristics that's $p(p+1)/2$ numbers per group, growing fast.

Regularized discriminant analysis (RDA) blends the two extremes with a tuning parameter $\alpha$:

$$\widehat{\Sigma}_k(\alpha) = \alpha \widehat{\Sigma}_k + (1-\alpha)\widehat{\Sigma}_{\text{pooled}},$$

shrinking each group's own noisy covariance estimate toward the shared pooled estimate. In plain English: $\alpha=1$ recovers pure QDA (fully separate shapes), $\alpha=0$ recovers pure LDA (one shared shape), and anything in between lets each group deviate from the shared shape only as much as the data can reliably support, with a second parameter often added to further shrink toward a diagonal or identity matrix when even the pooled estimate is noisy.

Worked example 1: QDA overfitting with too little data

Suppose you have only 15 observations for the "acquired" group and 4 characteristics. Estimating a full $4\times4$ covariance matrix for that group requires estimating 10 distinct numbers ($4\times5/2$) from just 15 data points — barely more observations than parameters. The resulting $\widehat{\Sigma}_{\text{acquired}}$ will fit the 15 training points' quirks almost perfectly (near-zero apparent within-group scatter along some direction, purely by chance), giving QDA a training classification accuracy that looks excellent but a held-out test accuracy that's much worse, because the "shape" it learned was largely sampling noise, not the true group structure.

Worked example 2: RDA recovering stability

Same data, but now using RDA with $\alpha = 0.3$: each group's covariance estimate is a blend of 30% its own noisy 15-observation estimate and 70% the pooled covariance (estimated from all observations across both groups combined, hundreds of points, and therefore far more stable). The resulting decision boundary is only mildly curved rather than wildly bent to fit 15 points' noise, and cross-validation on held-out data shows accuracy much closer to the training accuracy — the regularization traded a small amount of training fit for a large gain in how well the classifier generalizes.

What this means in practice

QDA is worth reaching for over LDA whenever there's genuine reason to expect groups to differ in their covariance structure, not just their means — but only if there's enough data per group to estimate that structure reliably, a bar that's easy to miss with small or imbalanced classes, common in finance (few acquisition targets, few defaults, few regime-shift days). Regularized discriminant analysis, with $\alpha$ chosen by cross-validation, is the practical default whenever group sizes are small or uneven, letting the data itself decide how much separate covariance structure is trustworthy.