Sufficient Statistics

Imagine you flipped a coin 1,000 times to estimate how biased it is. Do you need the full sequence, heads, heads, tails, heads, tails, tails... to figure out the bias? No. All you need is the count of heads. The exact order the flips came in tells you nothing extra about the coin. That count is a sufficient statistic: a compression of the raw data that keeps every last bit of information about the thing you're estimating, and discards only the noise.

This is a surprisingly deep idea. It says that for many models, a mountain of data collapses to one or two numbers with zero loss, as far as the unknown parameter is concerned. Knowing which summary is sufficient tells you exactly what to store, what to compute, and why certain estimators are the best possible.

The definition

A statistic $T(x)$, some function of the data $x_1, \dots, x_n$, is sufficient for a parameter $\theta$ if the conditional distribution of the data given $T$ no longer depends on $\theta$:

$$P(x_1, \dots, x_n \mid T = t, \theta) \;=\; P(x_1, \dots, x_n \mid T = t).$$

Read it as: once you're told the value of the summary $T$, the leftover detail in the raw data is pure randomness, it carries no further clue about $\theta$. Everything the sample could ever tell you about the parameter has already been squeezed into $T$.

The easy way to find one: the factorization theorem

Checking that definition directly is painful. In practice you use the factorization theorem: $T(x)$ is sufficient for $\theta$ if the likelihood splits into a part that touches $\theta$ only through $T$, times a part that doesn't involve $\theta$ at all:

$$f(x; \theta) \;=\; g\big(T(x), \theta\big)\, \cdot\, h(x).$$

The first factor $g$ is where $\theta$ and the data meet, and they only meet through $T$. The second factor $h$ depends on the raw data but not on $\theta$, so it's irrelevant to inference. If you can write the likelihood this way, $T$ is sufficient. That's the whole trick.

Worked example: the Poisson total

You count events over $n$ periods, $x_1, \dots, x_n$, each Poisson with unknown rate $\lambda$. The joint likelihood is

$$f(x;\lambda) = \prod_{i=1}^n \frac{\lambda^{x_i} e^{-\lambda}}{x_i!} = \underbrace{\lambda^{\sum x_i} e^{-n\lambda}}_{g(T,\,\lambda)} \cdot \underbrace{\frac{1}{\prod_i x_i!}}_{h(x)}.$$

The parameter $\lambda$ only ever touches the data through $\sum x_i$, the total count. The leftover factor $1/\prod x_i!$ depends on the individual numbers but not on $\lambda$. So the total $T = \sum x_i$ is sufficient. Concretely: if you observe events over $30$ days and someone tells you only that $90$ events happened in total, you can estimate $\lambda$ just as precisely as if you had the full day-by-day breakdown. The estimate $\hat\lambda = \sum x_i / n = 90/30 = 3$ per day uses the data through the sufficient statistic and nothing more.

Notice the pattern: the sufficient statistic $\sum x_i$ is exactly what the MLE and the method of moments both depend on. That's not a coincidence, good estimators are always functions of the sufficient statistic.

Why practitioners care

• Compression without loss. A sufficient statistic is the theoretically justified way to summarise data. For a Gaussian sample, the sufficient statistic is just the pair (sample mean, sample sum of squares), two numbers replace the whole dataset for inference about $\mu$ and $\sigma^2$.
• It's where the best estimators live. The Rao-Blackwell theorem says: take any estimator, condition it on a sufficient statistic, and you get one at least as good (lower variance). Combined with the The Cramér-Rao Bound, this is how you prove an estimator is optimal, efficient estimators are functions of a complete sufficient statistic.
• It flags the natural parameters. For the whole exponential family (normal, Poisson, exponential, binomial, gamma), the sufficient statistics fall out mechanically from the likelihood, telling you exactly which sample quantities matter.

Where the idea has limits

• It's model-specific. A statistic sufficient under one model can throw away crucial information under another. The sample total is sufficient for a Poisson rate but useless for detecting whether the data are actually Poisson, order and spacing (which sufficiency discards) are exactly what you'd test for overdispersion or clustering.
• Fat tails and misspecification. Sufficiency assumes you trust the model. In markets, the "discarded" detail, extreme observations, time ordering, is often the most important thing. Sufficiency tells you what's redundant if the model is right, which in finance is a big if.
• The whole sample is always sufficient. Trivially, keeping all the data loses nothing. The art is finding a low-dimensional sufficient statistic, a minimal one, that genuinely compresses.

The mental model to keep: sufficiency separates signal from noise inside a dataset. It answers "what do I actually need to remember?" and it's the backbone of why maximum likelihood, and the efficiency benchmark in Fisher Information and the The Cramér-Rao Bound, all point at the same handful of summary numbers.