Poisson and Negative Binomial Regression

A lot of quant data comes as counts: the number of trades a stock does in a minute, the number of corporate defaults in a portfolio this quarter, the number of news articles mentioning a ticker today. Counts are never negative, are always whole numbers, and tend to cluster near zero with a long right tail — none of which ordinary linear regression respects, since it happily predicts −2.3 trades or 4.7 defaults. Poisson regression is built specifically for counts, and negative binomial regression is the fix you reach for when Poisson's central assumption doesn't hold.

An analogy: raindrops in a bucket

Think of counting raindrops that land in a bucket during a storm. On an "average" storm, you might expect 20 drops a minute, and the actual minute-to-minute count will bounce around that average in a predictable way — busier minutes aren't wildly busier, they cluster fairly tightly around the mean because each drop lands more or less independently. Poisson regression assumes count data behaves like this: spread is tied directly to the mean, with no extra clumping. But real trade counts or defaults often clump — a news event triggers a burst of trades all at once, or a sector-wide shock triggers correlated defaults — producing far more spread than raindrops would. Negative binomial regression is built for that clumpier, burstier kind of counting.

The models, one piece at a time

Poisson regression models the expected count $\lambda_i$ as a function of predictors through a log link (which keeps $\lambda_i$ positive):

$$\log(\lambda_i) = \beta_0 + \beta_1 x_{1i} + \cdots, \qquad P(y_i = k) = \frac{\lambda_i^k e^{-\lambda_i}}{k!} .$$

The log link means each $\beta$ is interpreted multiplicatively: a one-unit increase in $x_1$ multiplies the expected count by $e^{\beta_1}$. The defining assumption of the Poisson distribution is that its variance equals its mean, $\text{Var}(y_i) = \lambda_i$ — in plain English, the model expects counts to spread out around their average by an amount fully determined by that average, with no independent source of extra noise.

Negative binomial regression keeps the same log-linear mean structure but adds a dispersion parameter $\alpha$ that lets variance exceed the mean:

$$\text{Var}(y_i) = \lambda_i + \alpha \lambda_i^2 .$$

When $\alpha=0$ this collapses back to Poisson; as $\alpha$ grows, the model allows for counts that are far more spread out than their mean would suggest under Poisson — the "overdispersion" that clumped, bursty count data typically shows. Both models are fit by maximum likelihood.

Worked example 1: detecting overdispersion in default counts

A credit desk models quarterly default counts across 40 industry segments. The fitted Poisson model predicts a mean of $\lambda = 3$ defaults per segment per quarter, which implies a variance of 3 (standard deviation $\approx 1.73$). But the actual sample variance across segments is 11.4 — nearly four times what Poisson allows. This is the standard sign that defaults are clustering (a sector-wide credit shock hits several names in the same segment together, rather than each name defaulting independently), and the fix is a negative binomial fit, which estimates $\alpha \approx 0.93$ here — solving $3 + 0.93 \times 3^2 = 3 + 8.37 = 11.37$, matching the observed variance almost exactly.

Worked example 2: interpreting the coefficient

Suppose the negative binomial fit gives $\beta_1 = 0.4$ on a segment's average credit spread (in percentage points) as a predictor of default count. The multiplicative effect is $e^{0.4} \approx 1.49$: a one-percentage-point rise in average spread is associated with defaults being about 49% more frequent, holding other predictors fixed. If a segment's baseline expected count is $\lambda = 2$ defaults per quarter, a two-point spread widening predicts a new expected count of $2 \times e^{0.4 \times 2} = 2 \times e^{0.8} \approx 2 \times 2.23 = 4.45$ defaults — more than doubling, even though the spread only widened by two points, because the log link makes effects compound multiplicatively rather than add up.

Drag $\lambda$ up and compare the spread of the Poisson distribution to what an overdispersed real count series looks like — Poisson's bars cluster tightly around $\lambda$, tighter than bursty trade or default counts typically do.

What this means in practice

Poisson and negative binomial regression are the standard tools for modeling trade counts, order arrivals, corporate actions, defaults, or news-event frequency as a function of covariates. Checking whether sample variance exceeds the sample mean (a quick overdispersion test) should happen before trusting a Poisson fit — if it's ignored, Poisson's standard errors come out too small, making predictors look far more statistically significant than they really are, which matters a great deal when the count model feeds into a risk or sizing decision.