The Gauss–Markov Theorem

The Gauss–Markov theorem is the optimality result that justifies why OLS is the default. It says that under a specific set of assumptions, among all linear unbiased estimators of the regression coefficients, ordinary least squares has the smallest variance, it is BLUE, the Best Linear Unbiased Estimator. Knowing the theorem is one thing; being able to state its assumptions exactly, prove it, and articulate the four important qualifications it carries is what separates someone who has memorized "OLS is BLUE" from someone who understands it.

The assumptions

Take the linear model $y = X\beta + \varepsilon$ with $y \in \mathbb{R}^n$, $X \in \mathbb{R}^{n\times k}$. Gauss–Markov requires:

1. Linearity in parameters: $y = X\beta + \varepsilon$.
2. Strict exogeneity: $\mathbb{E}[\varepsilon \mid X] = 0$, errors have mean zero given the regressors. This delivers unbiasedness.
3. No perfect multicollinearity: $X$ has full column rank $k$, so $X^\top X$ is invertible.
4. Spherical errors: $\operatorname{Var}(\varepsilon \mid X) = \sigma^2 I_n$, homoskedastic (constant variance) and uncorrelated errors.

Notice what is absent: no normality assumption. Gauss–Markov is distribution-free; normality is needed only for exact finite-sample $t$/$F$ inference, not for BLUE. Assumptions 1–3 make OLS unbiased; assumption 4 is what makes it best.

The theorem

Under assumptions 1–4, the OLS estimator $\hat\beta = (X^\top X)^{-1}X^\top y$ is the Best Linear Unbiased Estimator: for any other estimator $\tilde\beta$ that is linear in $y$ and unbiased, $\operatorname{Var}(\tilde\beta \mid X) - \operatorname{Var}(\hat\beta \mid X)$ is positive semidefinite. Equivalently, for every linear combination $c^\top\beta$, OLS gives the minimum-variance linear unbiased estimate.

The proof

Write OLS as a linear estimator $\hat\beta = A y$ with $A = (X^\top X)^{-1}X^\top$. Consider any competing linear estimator $\tilde\beta = C y$. Decompose $C = A + D$, where $D = C - A$ is the deviation from OLS.

Step 1, unbiasedness forces a constraint on $D$. For $\tilde\beta$ to be unbiased for all $\beta$,

$\mathbb{E}[\tilde\beta \mid X] = C\,\mathbb{E}[y\mid X] = C X\beta = (A + D)X\beta \stackrel{!}{=} \beta.$

Since $AX = (X^\top X)^{-1}X^\top X = I_k$, this requires $(I + DX)\beta = \beta$ for all $\beta$, i.e.

$DX = 0.$

Step 2, compute the variance. With spherical errors $\operatorname{Var}(y\mid X) = \sigma^2 I$,

$\operatorname{Var}(\tilde\beta \mid X) = C(\sigma^2 I)C^\top = \sigma^2 (A + D)(A + D)^\top = \sigma^2\big(AA^\top + AD^\top + DA^\top + DD^\top\big).$

Step 3, the cross terms vanish. Note $A D^\top = (X^\top X)^{-1}X^\top D^\top = (X^\top X)^{-1}(DX)^\top = 0$ by Step 1, and likewise $DA^\top = 0$. Also $AA^\top = (X^\top X)^{-1}X^\top X (X^\top X)^{-1} = (X^\top X)^{-1}$, so $\sigma^2 AA^\top = \operatorname{Var}(\hat\beta\mid X)$. Therefore

$\operatorname{Var}(\tilde\beta\mid X) = \underbrace{\sigma^2 (X^\top X)^{-1}}_{\operatorname{Var}(\hat\beta)} + \ \sigma^2 D D^\top.$

Step 4, conclude. $D D^\top$ is positive semidefinite (it is a Gram matrix: $x^\top D D^\top x = \lVert D^\top x\rVert^2 \ge 0$). Hence

$\operatorname{Var}(\tilde\beta\mid X) - \operatorname{Var}(\hat\beta\mid X) = \sigma^2 D D^\top \succeq 0,$

with equality only when $D = 0$, i.e. $\tilde\beta = \hat\beta$. OLS has the smallest variance among linear unbiased estimators. $\blacksquare$

The proof's engine is Step 1: unbiasedness pins $DX = 0$, which is exactly the orthogonality that kills the cross terms, the same projection geometry that defines OLS in the first place.

What the theorem does not say

Four qualifications are where the real understanding lives:

• "Linear" is load-bearing. OLS is best only among linear estimators. Nonlinear estimators can beat it; under fat tails a trimmed or robust estimator often has lower variance.
• "Unbiased" is load-bearing. Drop unbiasedness and you can do better in MSE. Ridge is biased yet, for suitable penalty, has lower mean-squared error, the bias–variance trade of Bias, Variance, and the Quality of Estimators and Ridge and LASSO Regularization.
• Assumption 4 is fragile. If errors are heteroskedastic or autocorrelated, OLS is still linear and unbiased but no longer BLUE, generalized least squares (GLS) is. OLS coefficients remain consistent, but the efficient estimator reweights. See Heteroskedasticity and Autocorrelation and Serial Correlation.
• It says nothing about the standard errors. BLUE is about the point estimator's variance; if assumption 4 fails, the reported standard errors are also wrong and need a robust (White/Newey–West) fix even when you keep OLS.

Worked example: why GLS beats OLS under heteroskedasticity

Suppose $\operatorname{Var}(\varepsilon_i) = \sigma_i^2$ varies across observations (high-vol regimes, larger firms). OLS weights every observation equally, so noisy observations drag the fit. The BLUE is now weighted least squares, minimizing $\sum_i (y_i - x_i^\top\beta)^2/\sigma_i^2$, down-weighting the high-variance points by $1/\sigma_i^2$. Equivalently, transform the model by $\Omega^{-1/2}$ where $\operatorname{Var}(\varepsilon) = \sigma^2\Omega$, restoring spherical errors, then run OLS. OLS on the untransformed data is unbiased but inefficient, it leaves variance on the table that GLS recovers.

Failure modes in financial data

• Volatility clustering violates homoskedasticity constantly; OLS betas are unbiased but inefficient, and their naive standard errors are wrong.
• Overlapping and autocorrelated returns violate the "uncorrelated errors" half of assumption 4; again OLS stays unbiased but loses BLUE status.
• Endogeneity breaks strict exogeneity (assumption 2), a deeper failure that makes OLS biased and inconsistent, so Gauss–Markov does not even get off the ground. This is qualitatively worse than a spherical-errors violation. See Endogeneity and Instrumental Variables.

In interviews

State the theorem precisely, OLS is BLUE, Best Linear Unbiased Estimator, and stress that "linear and unbiased" are both essential qualifiers (candidates who say "OLS is the best estimator, full stop" are wrong). Be ready to sketch the proof: write $\tilde\beta = (A+D)y$, show unbiasedness forces $DX = 0$, and show the variance is $\operatorname{Var}(\hat\beta) + \sigma^2 DD^\top \succeq \operatorname{Var}(\hat\beta)$. Know that normality is not required for Gauss–Markov (only for exact inference), and that when spherical errors fail, GLS is BLUE while OLS is merely unbiased. The sharpest follow-up, "when is a biased estimator better than OLS?", should trigger ridge and the MSE trade-off.