Regularized Discriminant Analysis

Linear discriminant analysis (LDA) assumes every class shares the same covariance structure, which keeps the model simple and stable but can be too rigid when classes genuinely spread out differently. Quadratic discriminant analysis (QDA) relaxes that assumption, letting each class have its own covariance matrix, which is more flexible but requires estimating far more parameters — a problem when there isn't much data per class, since a shaky covariance estimate can make QDA perform worse than the simpler LDA despite being more flexible in principle.

Regularized discriminant analysis (RDA) sits between the two, using a tuning parameter that blends each class's own covariance matrix with the shared, pooled covariance matrix from LDA. At one extreme the model behaves exactly like LDA; at the other, exactly like QDA; in between, it shrinks each class's individual covariance estimate toward the more stable pooled estimate, an approach that often outperforms either pure method when sample sizes are small relative to the number of features.

Worked example. A model classifying bonds into three credit-quality buckets has only 25 historical examples of the riskiest bucket, too few to reliably estimate a full covariance matrix for that class alone under QDA. Fitting RDA with a moderate blending parameter shrinks that class's noisy covariance estimate toward the more data-rich pooled covariance, producing more stable classification boundaries and, when checked against a held-out validation set, noticeably fewer misclassifications than pure QDA on that small class.