The Rao-Blackwell Theorem

Suppose you have a perfectly valid, unbiased estimator of some quantity — it gets the right answer on average — but it's clumsy, throwing away useful structure in the data. Is there a mechanical way to make it better, without inventing a cleverer estimator from scratch? The Rao-Blackwell theorem says yes, and gives you the recipe: condition your estimator on a sufficient statistic, and the result is provably at least as good, usually strictly better, at no extra cost in bias.

An analogy: replacing one juror's guess with the jury's average

Imagine one juror is asked to guess a defendant's age just from a photo, while the rest of the courtroom has heard hours of relevant testimony. That one juror's guess, on its own, is noisy — an honest attempt, but wasteful, since it ignores everything the courtroom collectively knows. Now instead ask: given everything the full courtroom knows (the "sufficient statistic" — the complete relevant information), what is the average guess a juror in this juror's position would make? That averaged, better-informed guess is still centered on the same truth as the original guess, but it varies far less from juror to juror, because it has folded in strictly more information. Rao-Blackwell formalizes exactly this move: replace a noisy, information-wasting estimator with its average conditional on everything relevant that the data has to say.

The statement, one symbol at a time

Let $\hat\theta$ be any unbiased estimator of a parameter $\theta$, and let $T$ be a sufficient statistic for $\theta$ — meaning $T$ captures everything the data has to say about $\theta$, so nothing else in the data adds information once you know $T$. Define the new estimator

$$\hat\theta_{RB} = E[\hat\theta \mid T]$$

Read this as: "take the average value $\hat\theta$ would have, conditional on knowing $T$" — this is a genuine, computable estimator in its own right, since it's built purely from $T$ (using the tower property, $\hat\theta$'s own randomness averages out, leaving a function of $T$ alone). The theorem guarantees two things: $\hat\theta_{RB}$ is still unbiased for $\theta$, exactly like the original, and

$$\text{Var}(\hat\theta_{RB}) \le \text{Var}(\hat\theta)$$

In plain English: conditioning on a sufficient statistic never makes your estimator worse, and typically makes it strictly better — you get the same unbiasedness with equal or lower variance, purely as a mechanical consequence of using the sufficient statistic instead of ignoring it.

Two unbiased estimators can be pictured as two bell curves centered on the same true value but with different widths. Rao-Blackwell is a guarantee about which curve you end up with: starting from the wider one, conditioning on a sufficient statistic can only move you to the same width or narrower — never wider — while the center stays exactly put.

Averaging in more information generally tightens an estimate, as the running-average explorer above shows for growing $n$ — conditioning on a sufficient statistic is a different mechanism for the same kind of tightening: instead of adding more data, it wrings out the wasted variance already sitting inside the single estimator you started with.

Worked example 1: improving a wasteful estimator of a Bernoulli mean

You observe $n=3$ coin flips and want to estimate the heads-probability $p$. A deliberately wasteful (but unbiased) estimator: just use the first flip, $\hat p = X_1$ — it's unbiased since $E[X_1]=p$, but it throws away flips 2 and 3 entirely.

The sufficient statistic for $p$ from $n$ Bernoulli trials is $T = \sum X_i$, the total number of heads. Rao-Blackwell says compute $E[X_1 \mid T=t]$. By symmetry among the three flips — given that exactly $t$ of the 3 flips were heads, each individual flip is equally likely to be one of the heads — $E[X_1 \mid T=t] = t/3$.

So $\hat\theta_{RB} = T/3 = \bar X$, the sample mean. Concretely: if the observed flips are $1, 0, 1$ (2 heads, $T=2$), the original wasteful estimator gives $\hat p = X_1 = 1$. The Rao-Blackwellized estimator gives $\hat p_{RB} = 2/3 = 0.667$ — using all three flips instead of one, and, across repeated experiments, provably lower variance than the original, single-flip estimator, exactly as the theorem promises.

Worked example 2: comparing the variances directly

Continue the same setup with true $p=0.5$. The original estimator $\hat p = X_1$ has variance $\text{Var}(X_1) = p(1-p) = 0.5\times0.5 = 0.25$.

The Rao-Blackwellized estimator $\hat p_{RB} = \bar X$ (the sample mean of 3 flips) has variance $\dfrac{p(1-p)}{n} = \dfrac{0.25}{3} = 0.0833$.

$0.0833$ versus $0.25$: the improved estimator's variance is exactly one-third of the original's — because averaging 3 roughly-independent pieces of information cuts variance by a factor of 3, precisely the gain the theorem guarantees you cannot fail to get (and here, using all the data instead of one-third of it, you get the maximum possible gain).

What this means in practice

• Rao-Blackwell is rarely applied by name in day-to-day quant work, but its logic explains why "use all your data, not a subsample" is not just intuition — it's a proven variance-reduction result whenever a genuine sufficient statistic exists.
• It pairs naturally with the Cramér-Rao bound: repeatedly Rao-Blackwellizing (or starting from an already-sufficient-based estimator) is one route toward reaching the theoretical minimum-variance estimator the Cramér-Rao bound describes, and for exponential family models the two results interlock cleanly.
• In Monte Carlo simulation, "Rao-Blackwellization" is a standard variance-reduction technique — replacing part of a simulated estimate with its known conditional expectation wherever that expectation can be computed in closed form, cutting simulation noise for free.