Exponential Family Distributions

The normal distribution, the Poisson distribution, the exponential distribution, and the Bernoulli distribution look like they have nothing in common — one is continuous and symmetric, one counts rare events, one models waiting times, one is a coin flip. Yet they all share one hidden algebraic skeleton, and that shared skeleton is precisely why so many "nice" statistical results — clean maximum likelihood formulas, simple sufficient statistics, tractable Bayesian updates — keep showing up for all of them at once. That skeleton is the exponential family.

An analogy: different outfits, same body

Imagine a family of people who all wear wildly different outfits — one in a lab coat, one in a tuxedo, one in hiking gear — but who all, underneath the clothes, share the exact same skeletal structure: same joint layout, same proportions. A doctor who understands that skeleton can predict how any of them will move, regardless of the outfit. The exponential family is the shared "skeleton" beneath a whole range of differently-dressed probability distributions — once you know a distribution belongs to the family, a whole toolkit of results applies automatically, without re-deriving them from scratch for each specific distribution.

The template, one symbol at a time

A distribution belongs to the exponential family if its probability function (density or mass) can be written in the form

$$f(x \mid \theta) = h(x)\, \exp\big(\eta(\theta)\, T(x) - A(\theta)\big)$$

Read each piece: $\theta$ is the parameter, $x$ is a data point. $h(x)$ ("base measure") is a piece depending only on the data, not the parameter. $T(x)$ ("sufficient statistic") is the single number — or short list of numbers — computed from the data that turns out to contain everything the data has to say about $\theta$; nothing else about $x$ matters once you know $T(x)$. $\eta(\theta)$ ("natural parameter") is a reparameterization of $\theta$ that makes the formula clean. $A(\theta)$ ("log-partition function") is just the piece that makes the whole thing integrate to 1, a bookkeeping constant. In plain English: the data and the parameter interact in the formula only through one simple product, $\eta(\theta)\,T(x)$, sitting inside an exponential — everything else is either pure data or pure parameter.

Why this template matters

Because the parameter only ever multiplies the sufficient statistic $T(x)$, and never appears tangled up with the data in any more complicated way, three things fall out almost for free: (1) the log-likelihood is always a simple, well-behaved function of $\theta$, making the maximum easy to find by setting the score to zero; (2) $T(x)$, or its sum across observations, is sufficient — you can throw away the individual data points and keep only $T$, with no loss of information about $\theta$; (3) a matching "conjugate prior" always exists for Bayesian updating, so combining prior belief with new data stays inside the same mathematical family instead of turning into an intractable mess.

The Beta-Binomial updater above is a live example: the binomial likelihood is exponential-family, and its conjugate Beta prior updates into another Beta posterior with simple arithmetic — a direct payoff of the structure described here, worked out concretely.

The Poisson curve above is another member of the same family — drag its rate and notice the shape changes smoothly and predictably, a consequence of the same exponential template underneath it, shared with the Beta-Binomial pair above despite looking like an unrelated distribution.

Worked example 1: showing the Poisson distribution fits the template

The Poisson probability mass function, used for modelling counts (say, number of large trades per minute) with rate $\lambda$, is $f(x\mid\lambda) = \dfrac{\lambda^x e^{-\lambda}}{x!}$ for $x = 0,1,2,\dots$

Rewrite it as an exponential: $f(x\mid\lambda) = \dfrac{1}{x!}\, e^{x\log\lambda - \lambda} = \dfrac{1}{x!}\exp\big(x\log\lambda - \lambda\big)$.

Match to the template: $h(x) = \dfrac{1}{x!}$, sufficient statistic $T(x) = x$, natural parameter $\eta(\lambda) = \log\lambda$, and $A(\lambda) = \lambda$. It fits exactly. The concrete payoff: because $T(x)=x$ is sufficient, if you have $n$ observed counts, you don't need to remember every individual count — only their sum, $\sum x_i$, carries all the information needed to estimate $\lambda$. For 5 observed counts $2, 5, 3, 4, 6$, the sufficient statistic is just $20$; the MLE is $\hat\lambda = 20/5 = 4$, and any other set of 5 counts summing to 20 would give the identical estimate.

Worked example 2: the normal distribution's sufficient statistics

The normal density with unknown mean $\mu$ and known variance $\sigma^2=1$ is $f(x\mid\mu) = \dfrac{1}{\sqrt{2\pi}} \exp\left(-\dfrac{(x-\mu)^2}{2}\right)$. Expand the square: $-\frac{1}{2}(x^2 - 2\mu x + \mu^2) = \mu x - \frac{1}{2}x^2 - \frac{1}{2}\mu^2$.

So $f(x\mid\mu) = \dfrac{1}{\sqrt{2\pi}}e^{-x^2/2}\exp\left(\mu x - \dfrac{\mu^2}{2}\right)$, matching the template with $h(x) = \frac{1}{\sqrt{2\pi}}e^{-x^2/2}$, $T(x)=x$, $\eta(\mu)=\mu$, $A(\mu) = \mu^2/2$. Given 4 returns $1.2, 0.8, 1.5, 0.9$, the sufficient statistic for $\mu$ is their sum, $4.4$ — this is exactly why, in the score-function derivation of the normal MLE, only $\sum x_i$ ever appeared in the answer $\hat\mu = 4.4/4 = 1.1$: the exponential-family structure guaranteed in advance that nothing besides the sum would matter.

What this means in practice

• Fitting GLMs (generalized linear models) — logistic regression, Poisson regression — relies entirely on the response distribution being exponential-family; that's literally what makes them "generalized linear," and it's why the fitting algorithm (iteratively reweighted least squares) is the same underlying procedure across all of them.
• Choosing a conjugate prior for a Bayesian model is only guaranteed straightforward when the likelihood is exponential-family — this is why Beta-Binomial, Gamma-Poisson, and Normal-Normal are the textbook conjugate pairs, and why more exotic likelihoods (like a Student-t) don't have a clean conjugate partner.
• Recognizing a model belongs to this family is a shortcut, not busywork: it tells you in advance that a clean sufficient statistic, a well-behaved likelihood surface, and a tractable Bayesian update all exist, without deriving each one separately.