The Sandwich Variance Estimator for M-Estimators

Once you've fit a robust regression — say, with Tukey's biweight — you still need a standard error on the coefficient, so you can say how confident you are in it. The tempting shortcut is to reuse the ordinary least squares (OLS) standard error formula. That formula is wrong here, because it assumes the estimator solves a plain sum-of-squares problem; a robust estimator instead solves a reweighted problem whose curvature and whose sensitivity to individual data points don't match OLS's. The sandwich variance estimator is the formula built specifically to get this right, no matter what estimating equation you actually used.

An analogy: sensitivity depends on the recipe, not just the ingredients

Imagine two chefs each estimating "the best amount of salt" from ten taste-testers' feedback. One chef averages every tester's opinion equally; the other quietly discounts the opinions of testers who gave wildly extreme answers. If you only look at how spread out the testers' raw opinions are, you'd assign both chefs the same uncertainty about their conclusion — but that's wrong, because the second chef's answer is less sensitive to any one extreme tester, precisely because of how they weighted the feedback. To get the uncertainty right, you need to know not just how noisy the inputs are, but exactly how sensitive each chef's method is to that noise. That sensitivity is different for each recipe (each estimating equation), and the sandwich formula is what plugs the right sensitivity in.

The formula, one symbol at a time

Say the estimator $\hat\beta$ solves an estimating equation $\sum_i \psi(u_i(\beta)) x_i = 0$ (zero total weighted pull, as in IRLS). The sandwich estimator for the covariance of $\hat\beta$ is

$$\widehat{\mathrm{Var}}(\hat\beta) = A^{-1} B A^{-1},$$

where $A$ (the "bread," used twice) measures the local curvature of the estimating equation — how much a small change in $\beta$ moves the fit — and $B$ (the "meat," used once, in the middle) measures the variance of the actual pulls $\psi(u_i)x_i$ each observation exerts. In plain English: the bread converts pulls into coefficient units on the way in and back out, and the meat captures how noisy those pulls really are; sandwiching the noise between two copies of the sensitivity gives the correct spread for $\hat\beta$, whether or not the data are perfectly well-behaved or the errors have some heavier tails than a textbook Gaussian.

Worked example 1: OLS is a special case

For OLS, $\psi(u) = u$ (no reweighting) and the sandwich formula reduces, under the assumption of constant error variance $\sigma^2$, to the familiar $\sigma^2 (X^\top X)^{-1}$ — the textbook OLS variance. Suppose $A = X^\top X = 40$ (a scalar, for a single-regressor toy case) and, under homoskedasticity, $B = \sigma^2 A = 4 \times 40 = 160$. Then $\widehat{\mathrm{Var}}(\hat\beta) = A^{-1} B A^{-1} = 40^{-1} \times 160 \times 40^{-1} = 0.1$, so the standard error is $\sqrt{0.1} \approx 0.316$ — exactly what the ordinary OLS formula would give, confirming the sandwich reduces correctly to the familiar case.

Worked example 2: a robust fit with heteroskedastic-looking pulls

Now fit the same data with a biweight estimator. Suppose the "bread" from the reweighted curvature comes out $A = 28$ (smaller than 40, because outliers were downweighted, reducing effective sample leverage), and the "meat," computed directly as the empirical variance of the individual pulls $\psi(u_i)x_i$ across the 20 observations, comes out $B = 95$. The sandwich variance is $A^{-1}BA^{-1} = 28^{-1} \times 95 \times 28^{-1} = 95/784 \approx 0.121$, giving a standard error of about $0.348$ — larger than the naive OLS-formula standard error of $0.316$ would suggest if you'd mistakenly reused it here, because the robust fit's pulls are less regular than the constant-variance assumption baked into the plain formula.

What this means in practice

Any time you fit a robust estimator — biweight regression, an M-estimator, a GEE model, or even OLS under suspected heteroskedasticity — the plain textbook variance formula is invalid unless the estimator is exactly OLS under exactly the OLS assumptions. Software packages that report "robust standard errors" for a robust fit are computing something like the sandwich formula under the hood. Getting this wrong means confidence intervals and t-statistics from a robust regression can be badly miscalibrated even though the point estimate itself is trustworthy.