Complete Separation and Firth's Correction

Logistic regression normally estimates coefficients by finding the values that make the observed outcomes most likely under the model. But if one predictor happens to perfectly separate the two outcome classes — every default in a small credit dataset has a debt ratio above some threshold, and every non-default falls below it — the maximum likelihood estimate for that predictor's coefficient doesn't converge to a finite value. The optimizer keeps pushing the coefficient toward infinity, since an ever-more-extreme coefficient always fits the perfect split slightly "better," and standard software either fails to converge or reports absurdly large coefficients and standard errors.

This is called complete separation, and it's especially common in small samples or with rare-event data, where there simply isn't enough variation to prevent one predictor from lining up too neatly with the outcome. Firth's correction fixes this by adding a small penalty term to the likelihood function that discourages extreme coefficient estimates, guaranteeing a finite, well-behaved solution even in a perfectly or near-perfectly separated dataset, at the cost of a small amount of bias.

Worked example. A model predicting bond default from 40 historical issuers finds that every issuer with a debt-to-EBITDA ratio above 8x defaulted, and every issuer below it didn't. A standard logistic regression fit to this data would report a coefficient on debt-to-EBITDA in the hundreds with a similarly enormous standard error — a red flag that separation has occurred. Refitting with Firth's correction instead produces a moderate, finite coefficient with a sensible standard error, giving a usable model rather than a diverging one.