Iteratively Reweighted Least Squares

Robust regression methods like Tukey's biweight define, in principle, an objective that's hard to minimize directly, it isn't the simple sum-of-squares that ordinary least squares (OLS) can solve in one shot with a formula. Iteratively reweighted least squares (IRLS) sidesteps that difficulty with a trick: instead of solving the hard robust problem directly, solve an easy weighted OLS problem, update the weights based on the new residuals, and repeat. Each individual step is something a spreadsheet could do; the robustness emerges from doing that step many times.

An analogy: adjusting a group photo crop

Imagine cropping a group photo to center on "the group," but a few people are standing far off to one side, pulling your first attempt at a center point away from the natural crowd. You crop again based on that estimate, notice the far-off people are now clearly on the fringe, so you weight them less in your next attempt at centering. You repeat: center, notice who's still an outlier, downweight them a bit more, re-center. After a few rounds, the crop settles onto the main crowd and the stragglers have been weighted down to almost nothing, without you ever explicitly deciding in advance who counted as a straggler.

The algorithm, one symbol at a time

Start with initial weights $w_i^{(0)} = 1$ for every observation $i$. At iteration $k$:

a plain weighted least-squares fit, solvable in closed form given the weights. In plain English: refit the line, but let each point's contribution to the sum-of-squares be scaled by its current weight. Then compute the new residuals $u_i^{(k)} = y_i - x_i^\top\hat\beta^{(k)}$, scale them by a robust estimate of spread, and update weights using the chosen psi function: $w_i^{(k)} = \psi(r_i^{(k)})/r_i^{(k)}$ (see Tukey's biweight). In plain English: points whose residuals are still large after this fit get less say in the next fit. Repeat until the weights (and hence $\hat\beta$) stop changing meaningfully, this is the fixed point of the robust estimating equation.

Worked example 1: three points, two iterations

Fit a robust mean to $10, $11, $50 using biweight weights with $c=4.685$.

• Iteration 0: all weights 1, so $\hat\mu^{(0)} = (10+11+50)/3 = 23.67$.
• Residuals $-13.67, -12.67, 26.33$, scaled by a robust spread estimate ($s\approx1.4$ from the median absolute deviation) to roughly $-9.8, -9.1, 18.8$: the $50 point gets weight 0, while $10 and $11 keep small nonzero weights.
• Iteration 1: refitting with only $10 and $11 effectively counted gives $\hat\mu^{(1)} \approx 10.5$, the answer a human would give by eye, ignoring the $50 outlier.

Worked example 2: convergence tracking

A desk robustly fits a factor-model beta using 60 days of returns, with two days containing fat-finger prints. Tracking the fitted beta: iteration 0 (plain OLS) gives $\beta=1.42$; iteration 1 gives $1.18$; iteration 2 gives $1.11$; iteration 3 gives $1.10$; iteration 4 matches to three decimals, so it has converged. The two fat-finger days end up with weights near zero, and the final $\beta=1.10$ matches what you'd get by manually deleting those days and rerunning OLS, except IRLS found them automatically.

What this means in practice

IRLS is the workhorse behind virtually every practical robust regression: M-estimators, biweight fits, and MM-estimators (see MM-estimators for regression) are all fit this way, because it turns an intractable non-convex problem into a sequence of easy weighted-OLS solves. It's the reason robust regression is computationally cheap despite its statistical sophistication, a few reweighted refits, not a hard global optimization.