S-Estimators for High-Breakdown Regression

Ordinary least squares picks the line that minimizes the sum of squared residuals — but that quantity, the sum of squares, is itself not a robust measure of "how tightly the data cluster around a line," because a single huge residual dominates the sum. S-estimators fix this at the source: instead of minimizing a sum, they minimize a genuinely robust measure of the residuals' scale (spread), one built using the same redescending psi-function machinery as Tukey's biweight. The "S" stands for scale — the whole method is defined around robustly estimating that one number.

An analogy: judging consistency, not total error

Imagine grading a dart player not by summing the squared distance of every throw from the bullseye (where one wildly wayward throw dominates the total), but by asking: "what's the smallest circle, centered somewhere, such that a robust majority of throws land inside it?" That circle's radius is a robust scale measure — it isn't dragged around by one bad throw the way a sum-of-squared-distances would be. An S-estimator finds the regression line (or the dartboard's center) that makes this robust "typical scatter" measure as small as possible, rather than the line that minimizes total squared error.

The estimator, one symbol at a time

For a candidate coefficient vector $\beta$, compute the residuals $u_i(\beta) = y_i - x_i^\top\beta$. Define their robust scale $s(\beta)$ implicitly as the solution to

$$\frac{1}{n}\sum_{i=1}^n \rho\left(\frac{u_i(\beta)}{s(\beta)}\right) = b ,$$

where $\rho$ is a bounded loss function (related to the biweight's $\psi$ by $\psi = \rho'$) and $b$ is a fixed constant chosen to calibrate the scale. In plain English: $s(\beta)$ is "the spread that makes the average bounded-loss score of the standardized residuals equal a target constant" — a self-consistent, robust stand-in for the standard deviation of the residuals, one that isn't inflated without limit by a few extreme points because $\rho$ is bounded. The S-estimator of $\beta$ is then

$$\hat\beta_S = \arg\min_\beta \; s(\beta),$$

i.e., search over candidate lines for the one whose residuals have the smallest robust scale. In plain English: find the line around which the well-behaved majority of points cluster most tightly, as measured by a robust yardstick that a handful of extreme points cannot stretch out.

Worked example 1: comparing scale measures directly

Ten residuals from a candidate fit: eight are within $\pm 1$ of zero, and two are far out at $+15$ and $-18$ (bad data points). The ordinary standard deviation of all ten residuals is inflated hugely by the two extreme values — roughly 7 or 8 in this case. A robust scale computed via the S-estimator's bounded-loss criterion instead comes out close to 1, because the bounded $\rho$ function caps how much the two extreme residuals can contribute to the average loss, regardless of how far out they sit. This is exactly the property an S-estimator exploits when searching for $\hat\beta$: candidate lines that make the ordinary standard deviation small (by accommodating the two bad points) are not rewarded, because the S-estimator only ever looks at the robust scale.

Worked example 2: high contamination in a factor-model fit

A desk fits a two-factor exposure model to 100 days of returns, of which 30 days (30% contamination) contain corrupted prints from a data vendor outage. An OLS fit is visibly distorted, with an $R^2$ of only 0.31 and factor loadings that don't match the desk's prior expectations. Refitting the same 100 days with an S-estimator, using a bounded $\rho$ function tuned for a roughly 50% breakdown point, the loadings come back close to what a manual fit on the 70 clean days alone would give, and the corrupted days show up with large residuals under the robust fit rather than distorting the coefficients. This is the practical payoff: the analyst never had to manually identify and remove the 30 bad days.

What this means in practice

S-estimators are mainly a building block: they achieve a high breakdown point (up to 50%) directly from first principles, without the somewhat ad hoc trimming of LTS (see least trimmed squares), but on clean data they're statistically less efficient than an M-estimator. In practice they're rarely used alone — they're the robust "first stage" that supplies both a starting fit and a robust scale estimate to an MM-estimator (see MM-estimators for regression), which then refines the fit for better efficiency while preserving the S-estimator's breakdown point.