Iteratively Reweighted Least Squares

Robust regression methods like Tukey's biweight define, in principle, an objective that's hard to minimize directly — it isn't the simple sum-of-squares that ordinary least squares (OLS) can solve in one shot with a formula. Iteratively reweighted least squares (IRLS) sidesteps that difficulty with a trick: instead of solving the hard robust problem directly, solve an easy weighted OLS problem, update the weights based on the new residuals, and repeat. Each individual step is something a spreadsheet could do; the robustness emerges from doing that step many times.

An analogy: adjusting a group photo crop

Imagine cropping a group photo to center on "the group," but a few people are standing far off to one side, pulling your first attempt at a center point away from the natural crowd. You crop again based on that estimate, notice the far-off people are now clearly on the fringe, so you weight them less in your next attempt at centering. You repeat: center, notice who's still an outlier, downweight them a bit more, re-center. After a few rounds, the crop settles onto the main crowd and the stragglers have been weighted down to almost nothing — without you ever explicitly deciding in advance who counted as a straggler.

The algorithm, one symbol at a time

Start with initial weights $w_i^{(0)} = 1$ for every observation $i$. At iteration $k$:

$$\hat\beta^{(k)} = \arg\min_\beta \sum_i w_i^{(k-1)} \left(y_i - x_i^\top \beta\right)^2 ,$$

a plain weighted least-squares fit — solvable in closed form given the weights. In plain English: refit the line, but let each point's contribution to the sum-of-squares be scaled by its current weight. Then compute the new residuals $u_i^{(k)} = y_i - x_i^\top\hat\beta^{(k)}$, scale them by a robust estimate of spread, and update weights using the chosen psi function: $w_i^{(k)} = \psi(r_i^{(k)})/r_i^{(k)}$ (see Tukey's biweight). In plain English: points whose residuals are still large after this fit get less say in the next fit. Repeat until the weights (and hence $\hat\beta$) stop changing meaningfully — this is the fixed point of the robust estimating equation.

Worked example 1: three points, two iterations

Fit a robust mean to $10, $11, $50 using biweight weights with $c=4.685$.

• Iteration 0: all weights 1, so $\hat\mu^{(0)} = (10+11+50)/3 = 23.67$.
• Residuals $-13.67, -12.67, 26.33$, scaled by a robust spread estimate ($s\approx1.4$ from the median absolute deviation) to roughly $-9.8, -9.1, 18.8$: the $50 point gets weight 0, while $10 and $11 keep small nonzero weights.
• Iteration 1: refitting with only $10 and $11 effectively counted gives $\hat\mu^{(1)} \approx 10.5$ — the answer a human would give by eye, ignoring the $50 outlier.

Worked example 2: convergence tracking

A desk robustly fits a factor-model beta using 60 days of returns, with two days containing fat-finger prints. Tracking the fitted beta: iteration 0 (plain OLS) gives $\beta=1.42$; iteration 1 gives $1.18$; iteration 2 gives $1.11$; iteration 3 gives $1.10$; iteration 4 matches to three decimals, so it has converged. The two fat-finger days end up with weights near zero, and the final $\beta=1.10$ matches what you'd get by manually deleting those days and rerunning OLS — except IRLS found them automatically.

What this means in practice

IRLS is the workhorse behind virtually every practical robust regression: M-estimators, biweight fits, and MM-estimators (see MM-estimators for regression) are all fit this way, because it turns an intractable non-convex problem into a sequence of easy weighted-OLS solves. It's the reason robust regression is computationally cheap despite its statistical sophistication — a few reweighted refits, not a hard global optimization.