The Exponential Family

Open any statistics textbook and the distributions arrive one at a time, each with its own personality: the binomial has its counting argument, the Poisson its limiting argument, the normal its bell. Each seems to need its own estimation theory, its own Bayesian machinery, its own regression model. Learning them individually is a lot of memorising, and worse, it hides the fact that almost all of them are the same object wearing different clothes. The exponential family is that realisation, written down.

The analogy: the shop only wants your total

Think about a coffee shop loyalty scheme. You buy dozens of things over a year — different drinks, different days, different prices. But when the shop decides what reward you have earned, it does not reread every receipt. It looks at two numbers: how much you spent in total and how many times you visited. Everything else about your year is thrown away, and nothing is lost, because the reward rule only ever depended on those two totals.

A distribution belongs to the exponential family when the same thing is true of its data. However many observations you gather, the distribution only "sees" them through a small fixed set of totals. Those totals are called sufficient statistics, and the whole payoff of the family follows from their existence: if a handful of running sums is all the information there is, then estimation, updating and prediction all become arithmetic on those sums.

The template

A distribution for an observation $x$ with parameter $\theta$ belongs to the exponential family if its probability (or density) can be written as

$$p(x \mid \theta) = h(x)\,\exp\big(\eta(\theta)\,T(x) - A(\eta)\big).$$

Four pieces, each with a job:

• $h(x)$, the base measure, is whatever depends on the data alone and not on the parameter. It carries no information about $\theta$.
• $T(x)$, the sufficient statistic, is the only function of the data the parameter ever multiplies. This is the loyalty-card total.
• $\eta(\theta)$, the natural parameter, is the parameter re-expressed in whatever units make the formula linear. It is often not the parameter you started with.
• $A(\eta)$, the log-partition function, is the bookkeeping term that makes the probabilities sum to one.

In words, the template says: take a data-only factor, multiply the parameter by one summary of the data, exponentiate, and divide by whatever makes it a valid distribution.

The one derivation worth doing

Because $p$ has to integrate to one, the template forces

$$\int h(x)\, e^{\eta T(x)}\,dx = e^{A(\eta)} .$$

Differentiate both sides with respect to $\eta$. On the left, differentiating $e^{\eta T(x)}$ brings down a factor $T(x)$; on the right, the chain rule gives $A'(\eta)e^{A(\eta)}$:

$$\int T(x)\,h(x)\,e^{\eta T(x)}\,dx = A'(\eta)\, e^{A(\eta)} .$$

Now divide both sides by $e^{A(\eta)}$. The left-hand side becomes $\int T(x)\,p(x\mid\eta)\,dx$, which is just the expected value of $T$. So

$$A'(\eta) = E[T(X)], \qquad A''(\eta) = \operatorname{Var}(T(X)).$$

In words: differentiate the bookkeeping term once and you get the mean of the sufficient statistic; differentiate twice and you get its variance. You never have to compute a moment integral for any member of the family again — you differentiate one scalar function.

The same trick gives the estimator. With $n$ independent observations the log-likelihood is $\sum_i \ln h(x_i) + \eta \sum_i T(x_i) - n A(\eta)$. Setting its derivative in $\eta$ to zero leaves $\sum_i T(x_i) - n A'(\eta) = 0$, so the maximum-likelihood estimate solves

$$A'(\hat\eta) = \frac{1}{n}\sum_{i=1}^{n} T(x_i).$$

In words: choose the parameter whose predicted average of $T$ equals the average of $T$ you actually observed. Every maximum-likelihood problem in the family is that one sentence.

Worked example 1: the Bernoulli, and where logistic regression comes from

A single coin flip, $x \in \{0,1\}$, heads with probability $p$:

$$p(x \mid p) = p^{x}(1-p)^{1-x}.$$

Take logs and regroup. $\ln p(x) = x\ln p + (1-x)\ln(1-p) = x\big(\ln p - \ln(1-p)\big) + \ln(1-p)$. Exponentiating back:

$$p(x\mid p) = \exp\left(x \ln\frac{p}{1-p} + \ln(1-p)\right).$$

Match it to the template: $h(x) = 1$, $T(x) = x$, the natural parameter is $\eta = \ln\frac{p}{1-p}$ — the log-odds — and $A(\eta) = -\ln(1-p) = \ln(1+e^{\eta})$.

Check it with real numbers at $p = 0.3$:

• $\eta = \ln(0.3/0.7) = \ln 0.4286 = -0.847$.
• $A(\eta) = \ln(1 + e^{-0.847}) = \ln(1 + 0.4286) = \ln 1.4286 = 0.357$.
• $A'(\eta) = e^{\eta}/(1+e^{\eta}) = 0.4286/1.4286 = 0.300$. That is the mean of a Bernoulli, $p$. Correct.
• $A''(\eta) = p(1-p) = 0.3 \times 0.7 = 0.21$. That is the Bernoulli variance. Correct.

Look at $A'(\eta)$ again: it is the logistic function. Mapping the natural parameter back to a probability is the logistic curve, which is why Logistic Regression puts a straight line on the log-odds and then squashes it — it is fitting the natural parameter of a Bernoulli, nothing more exotic than that.

The explorer below plots that curve. Set the shape to "logistic", then drag the probe to $x = -0.85$ and read the output: with steepness 1 you land on 0.30, the probability we started from.

Worked example 2: the Poisson, and why its mean equals its variance

Counts of events — trades hitting your quote in a minute, say — with rate $\lambda$:

$$p(x\mid\lambda) = \frac{e^{-\lambda}\lambda^{x}}{x!} = \frac{1}{x!}\exp\big(x\ln\lambda - \lambda\big).$$

Match the template: $h(x) = 1/x!$, $T(x) = x$, $\eta = \ln\lambda$, and since $\lambda = e^{\eta}$, the log-partition is $A(\eta) = e^{\eta}$.

Now differentiate. $A'(\eta) = e^{\eta} = \lambda$ and $A''(\eta) = e^{\eta} = \lambda$ as well, because $e^\eta$ is its own derivative. The mean and the variance are the same function, so for a Poisson they are the same number — a fact usually presented as a curiosity, here forced by one line of calculus.

Fit it to data. Five one-minute trade counts: $3, 5, 2, 6, 4$. Their sum is 20 and $n = 5$, so the average of $T$ is $20/5 = 4$. The estimating equation says $A'(\hat\eta) = 4$, that is $e^{\hat\eta} = 4$, so $\hat\eta = \ln 4 = 1.386$ and $\hat\lambda = 4$.

Notice what you needed: the total, 20, and the count, 5. The individual numbers 3, 5, 2, 6, 4 never entered. That is sufficiency doing real work — a monitoring system can keep two running numbers and refit exactly.

The explorer below shows the fitted distribution. Drag $\lambda$ to 4 and note that the spread grows in lockstep with the centre, exactly as $A'' = A'$ requires.

What this means in practice

• Generalised linear models. Logistic regression, Poisson regression and ordinary least squares are one idea: put a linear model on the natural parameter of an exponential-family response. Choosing "the right model for count data" is choosing $T$ and $\eta$.
• Bayesian updating that costs nothing. Conjugate priors exist precisely for exponential families, so a posterior update is addition on sufficient statistics. See Conjugate Priors.
• Streaming and aggregation. Sufficiency means a system can store totals rather than history. Ten years of tick counts collapse to two numbers with no loss.
• Curvature is information. $A''(\eta)$ is both the variance and the Fisher information, which is why a flat log-partition means a poorly identified parameter.
• Where it fails. Mixtures, and heavy-tailed models such as Student's t, are not in the family. Their tails cannot be captured by a fixed finite set of totals, which is exactly why fat-tailed estimation is harder.