The Cramér-Rao Bound

The Cramér-Rao bound is one of the most useful "you can't do better than this" results in statistics. It says that once you fix the amount of information in your data, there is a floor on how precise any unbiased estimate can be. You can invent clever new estimators all day, but none of them can beat this limit. That makes it a benchmark: if your estimator already sits on the floor, stop tinkering, you've found the best one there is.

It builds directly on Fisher Information, the number that measures how much a dataset tells you about an unknown parameter. The bound simply says: the more information, the lower the achievable error, and it makes that trade-off exact.

The bound

For any unbiased estimator $\hat\theta$ of a parameter $\theta$ built from $n$ data points, its variance obeys

$$\operatorname{Var}(\hat\theta) \;\ge\; \frac{1}{I_n(\theta)},$$

where $I_n(\theta) = n\,I_1(\theta)$ is the total Fisher information in the sample. In plain terms: the smallest error bar you could ever hope for is one over the information. Double the information and the variance floor halves. An estimator that actually reaches this floor is called efficient, it is squeezing every last drop of precision out of the data.

The efficiency scorecard

Because the bound gives a target, you can grade any estimator against it. The efficiency of $\hat\theta$ is the ratio of the floor to its actual variance:

$$\text{efficiency} = \frac{1/I_n(\theta)}{\operatorname{Var}(\hat\theta)} \in [0, 1].$$

An efficiency of $1$ (or $100\%$) means the estimator is optimal. An efficiency of $0.8$ means it needs $25\%$ more data to match what the best estimator gets from the original sample. This is exactly why the maximum-likelihood estimator is the default engine of parametric statistics: under regularity conditions it becomes efficient in large samples, its variance converges to the Cramér-Rao floor $1/I_n(\theta)$.

Worked example: estimating a Poisson rate

Suppose customer orders arrive at some unknown average rate $\lambda$ per minute, and you record counts $x_1, \dots, x_n$ over $n$ separate minutes, each Poisson with mean $\lambda$. The natural estimator is the sample mean $\hat\lambda = \bar x$.

The Fisher information for one Poisson observation is $I_1(\lambda) = 1/\lambda$ (the curvature of the Poisson log-likelihood), so the sample carries $I_n(\lambda) = n/\lambda$. The Cramér-Rao floor is therefore

$$\operatorname{Var}(\hat\lambda) \;\ge\; \frac{1}{n/\lambda} = \frac{\lambda}{n}.$$

Now check the estimator we actually chose. A Poisson variable has variance $\lambda$, so the sample mean has variance $\operatorname{Var}(\bar x) = \lambda/n$, exactly the floor. The sample mean is efficient: no unbiased method could estimate the arrival rate more precisely from this data. Concretely, if the true rate is $\lambda = 3$ orders per minute and you watch for $n = 100$ minutes, the best possible variance is $3/100 = 0.03$, a standard error of about $0.17$ orders per minute, and $\bar x$ already achieves it.

Where it applies and where it doesn't

• Unbiased only. The classic bound is for unbiased estimators. Allow a little bias and you can often get a lower total error (mean-squared error) than the Cramér-Rao floor, this is exactly the trade shrinkage estimators make, accepting bias to slash variance. The bound isn't the last word on accuracy, only on unbiased precision.
• Regularity conditions. The result assumes a smooth likelihood whose support doesn't depend on $\theta$. For odd cases (like estimating the upper edge of a uniform distribution, where the boundary moves with the parameter) the bound doesn't apply and estimators can be super-efficient.
• It's asymptotic for MLE. Maximum likelihood reaches the floor in large samples. In small, heavy-tailed financial samples the MLE can sit well above the floor, so don't quote the information-based standard error as gospel, bootstrap it.

The interview angle

The clean statement to have ready: variance $\ge 1/I_n(\theta)$, an unbiased estimator hitting it is efficient, and the MLE achieves it asymptotically. The favourite follow-up is "can you ever beat it?", and the sharp answer is yes, by giving up unbiasedness, which reframes the whole point of shrinkage. It pairs naturally with Sufficient Statistics (efficient estimators are functions of sufficient statistics) and with Maximum Likelihood Estimation (MLE) as the method that reaches the floor.