Minimax Estimation

Most estimators are judged by average performance: given a true parameter $\theta$, how small is the expected squared error over repeated sampling? But that average depends on knowing (or assuming) $\theta$, and an estimator that's excellent for one plausible $\theta$ can be terrible for another. Minimax estimation sidesteps this by asking a different question: for each candidate estimator, what is its worst-case risk over every possible true $\theta$ — and which estimator makes that worst case as small as possible?

Formally, for an estimator $\hat\theta$ with risk (expected loss) $R(\theta, \hat\theta)$ as a function of the unknown true $\theta$, the minimax estimator solves:

$$\hat\theta^* = \arg\min_{\hat\theta} \; \sup_{\theta} \; R(\theta, \hat\theta).$$

In plain English: look at every candidate estimator, find the worst possible true $\theta$ for each one, and pick the estimator whose worst case is the least bad. It's a pessimist's rule — it protects against the worst-case scenario rather than optimizing for an assumed typical one, which is valuable exactly when you have no reliable prior belief about $\theta$ to lean on.

Worked example. Estimating the mean of a bounded random variable, the sample mean $\bar X$ has risk (mean squared error) that grows the same way regardless of the true mean's actual value within the bound — so its worst-case risk over the whole allowed range is no worse than its risk anywhere else, which is exactly why the sample mean is minimax for this problem: no other estimator can guarantee a better worst-case bound.

This connects directly to why regularized estimators (ridge regression, shrinkage estimators like James-Stein) get used in practice: they trade a small amount of best-case performance for a much better worst-case guarantee across the range of plausible true parameters, which is the minimax logic even when the estimator wasn't derived by explicitly solving the minimax problem.