Bias, Variance, and the Quality of Estimators

An estimator is a recipe, a function of the sample $\hat\theta = \hat\theta(X_1, \dots, X_n)$, that produces a guess of an unknown parameter $\theta$. Because the sample is random, the estimator is a random variable with its own distribution, the sampling distribution. Almost every argument about whether a volatility estimate, a beta, or a Sharpe ratio is "good" is really an argument about the shape of that sampling distribution, and two numbers summarize most of it: how far off it is on average (bias) and how much it wobbles (variance).

Bias, variance, and the MSE decomposition

Define the bias as the average error and the variance as the spread:

$\operatorname{Bias}(\hat\theta) = \mathbb{E}[\hat\theta] - \theta, \qquad \operatorname{Var}(\hat\theta) = \mathbb{E}\big[(\hat\theta - \mathbb{E}[\hat\theta])^2\big].$

The natural scalar measure of quality is mean squared error, $\operatorname{MSE}(\hat\theta) = \mathbb{E}[(\hat\theta - \theta)^2]$. Add and subtract $\mathbb{E}[\hat\theta]$ inside the square:

$\operatorname{MSE}(\hat\theta) = \mathbb{E}\big[(\hat\theta - \mathbb{E}[\hat\theta]) + (\mathbb{E}[\hat\theta] - \theta)\big]^2.$

Expanding, the cross term $2\,\mathbb{E}[\hat\theta - \mathbb{E}[\hat\theta]]\,(\mathbb{E}[\hat\theta]-\theta)$ vanishes because $\mathbb{E}[\hat\theta - \mathbb{E}[\hat\theta]] = 0$, leaving the bias–variance decomposition:

$\boxed{\operatorname{MSE}(\hat\theta) = \operatorname{Var}(\hat\theta) + \operatorname{Bias}(\hat\theta)^2.}$

This is the single most important identity in estimation. It says an unbiased estimator is not automatically the best one: you can often trade a little bias for a large drop in variance and win on MSE. That trade is the entire justification for shrinkage, ridge/LASSO, and Bayesian priors. See Ridge and LASSO Regularization and Shrinkage.

Consistency

Unbiasedness is a finite-sample property; consistency is the asymptotic promise that more data eventually pins down the truth. Formally $\hat\theta_n$ is consistent if it converges in probability, $\hat\theta_n \xrightarrow{p} \theta$, i.e. for every $\varepsilon > 0$, $\mathbb{P}(|\hat\theta_n - \theta| > \varepsilon) \to 0$. A clean sufficient condition follows straight from the decomposition: if both $\operatorname{Bias}(\hat\theta_n) \to 0$ and $\operatorname{Var}(\hat\theta_n) \to 0$, then $\operatorname{MSE} \to 0$, which forces convergence in probability (via Chebyshev). An estimator can be biased in every finite sample yet consistent (the MLE of a variance is the standard example), and, more dangerously, it can be unbiased yet inconsistent if its variance never shrinks.

Efficiency and the Cramér–Rao bound

Among unbiased estimators we prefer the one with the smallest variance, the efficient one. There is a hard floor on how small that variance can be. For a model with density $f(x;\theta)$ and log-likelihood $\ell$, define the Fisher information

$I(\theta) = \mathbb{E}\!\left[\left(\frac{\partial \ell}{\partial \theta}\right)^2\right] = -\,\mathbb{E}\!\left[\frac{\partial^2 \ell}{\partial \theta^2}\right].$

The Cramér–Rao inequality states that any unbiased estimator satisfies

$\operatorname{Var}(\hat\theta) \ \ge\ \frac{1}{n\,I_1(\theta)},$

where $I_1$ is the information in one observation. An unbiased estimator that attains this bound is called efficient. The bound is what makes maximum likelihood special: the MLE is asymptotically efficient, reaching the Cramér–Rao floor as $n \to \infty$ (see Maximum Likelihood Estimation (MLE)). The bound also explains why fat tails hurt, heavier-tailed models carry less Fisher information per observation about location, so the same precision needs far more data.

Worked example: the two variance estimators

Take $x_1, \dots, x_n$ i.i.d. with mean $\mu$ and variance $\sigma^2$. Two competing estimators of $\sigma^2$ are

$S^2 = \frac{1}{n-1}\sum_i (x_i - \bar x)^2, \qquad \hat\sigma^2_{\text{MLE}} = \frac{1}{n}\sum_i (x_i - \bar x)^2.$

The $n-1$ version is unbiased, $\mathbb{E}[S^2] = \sigma^2$; the divisor is $n-1$ because estimating $\bar x$ uses up one degree of freedom. The MLE version has bias $\mathbb{E}[\hat\sigma^2_{\text{MLE}}] - \sigma^2 = -\sigma^2/n < 0$, it systematically underestimates. Yet under normality the MLE has smaller variance, and if you compute MSE the minimizer over divisors $n + c$ is actually $n+1$, not $n-1$: the unbiased estimator is not the MSE-optimal one. This is the bias–variance tradeoff in its most concrete, examinable form, being right on average is not the same as being close on average.

Failure modes in financial data

• The mean is nearly unestimable. For returns the standard error of the sample mean is $\sigma/\sqrt{n}$. With equity vol around 20% annually, pinning the mean to within 1% needs $\sigma^2/(0.01)^2 = 400$ years of data. Drift estimates are almost all variance, this is why The Kelly Criterion sizing on a point estimate of $\mu$ is so fragile.
• Autocorrelation breaks the variance formulas. The $\sigma^2/n$ variance of the sample mean assumes independence. With positively autocorrelated returns (or overlapping windows) the true variance is inflated by a factor like $1 + 2\sum_k \rho_k$, so naive standard errors are too small and estimators look more precise than they are. See Autocorrelation and Serial Correlation.
• Fat tails inflate the variance of variance. The sampling variance of $S^2$ depends on the fourth moment (kurtosis). Financial returns are leptokurtic, so volatility and correlation estimates are noisier than Gaussian intuition suggests, and a single crash day can dominate the estimate.
• Non-stationarity kills consistency. Consistency assumes the parameter is fixed. If $\theta$ (a beta, a vol, an edge) drifts over the sample, adding older data adds bias, not information, more data can make you worse. See Stationarity and Regime Detection.

In interviews

Be able to write $\operatorname{MSE} = \operatorname{Var} + \operatorname{Bias}^2$ and derive it by adding and subtracting $\mathbb{E}[\hat\theta]$ (the cross term vanishing is the whole trick). Expect "why $n-1$?", answer with the lost degree of freedom, and score bonus points by noting the unbiased divisor is not the MSE-minimizing one. Know the definitions of consistency versus unbiasedness (finite-sample vs asymptotic) and be ready to state the Cramér–Rao bound and why it makes MLE the benchmark. The sharpest follow-up is "give me a biased estimator you'd prefer to an unbiased one", the honest answer is a shrinkage or ridge estimator, justified directly by the decomposition above.