Weighted Least Squares (WLS)

Ordinary least squares treats every data point as equally trustworthy. It squares each residual and adds them up, so a noisy, unreliable observation gets exactly the same vote as a clean, precise one. That's fine when all your data really are equally noisy, but often they aren't. Some observations are measured with more precision, cover more underlying samples, or come from calmer periods. Weighted least squares fixes the unfairness: it lets you hand each observation a weight that says how much it should count.

The rule of thumb is simple and intuitive: weight each point by the inverse of its noise. A point you trust ten times as much should pull on the fit ten times as hard. Do that, and you recover a fit that's both precise and, under the right weights, provably the best linear unbiased estimator again, which plain OLS forfeits the moment the noise stops being uniform.

The model

OLS minimises the plain sum of squared residuals. WLS minimises a weighted sum:

$$\hat\beta_{\text{WLS}} = \arg\min_{b}\; \sum_{i=1}^{n} w_i \big(y_i - x_i^\top b\big)^2,$$

where $w_i$ is the weight on observation $i$. A big $w_i$ means "fit this point closely, its residual is expensive"; a small $w_i$ means "don't sweat missing this one." The optimal choice, the one that gives the most precise estimate, is the inverse of each point's error variance:

$$w_i = \frac{1}{\sigma_i^2}.$$

Points with small variance (reliable) get large weights; points with large variance (noisy) get small ones. WLS is the special case of GLS where the errors are unequal in size but still uncorrelated, so the error covariance is a diagonal matrix and its inverse is just this list of weights.

Worked example: readings from unequal samples

You measure a signal at three settings, but each reading is an average over a different number of trials, so the more-averaged readings are more reliable (variance shrinks like $\sigma^2/n$). The readings:

• Setting A: averaged over $n = 100$ trials, reading $5.0$
• Setting B: averaged over $n = 25$ trials, reading $5.4$
• Setting C: averaged over $n = 4$ trials, reading $7.0$

Just averaging the three numbers gives $(5.0 + 5.4 + 7.0)/3 = 5.8$, but that lets the flimsy 4-trial reading count as much as the rock-solid 100-trial one. Since variance scales as $1/n$, the right weights are proportional to $n$. The WLS estimate is

$$\hat\mu = \frac{100 \cdot 5.0 + 25 \cdot 5.4 + 4 \cdot 7.0}{100 + 25 + 4} = \frac{500 + 135 + 28}{129} = \frac{663}{129} \approx 5.14.$$

The weighted answer $5.14$ sits right next to the two well-measured settings and shrugs off the noisy $7.0$, whereas the naive $5.8$ was dragged up by it. That's WLS doing its one job: letting evidence count in proportion to how much you can trust it.

Where it goes wrong

• Wrong weights hurt. WLS is only optimal if the weights really are $1/\sigma_i^2$. Guess the variance pattern badly and you can end up less precise than plain OLS, you've amplified the wrong points. When unsure of the pattern, keeping OLS estimates and using robust (White) standard errors is the safer play.
• You usually have to estimate the variances. The true $\sigma_i^2$ are rarely known, so you model them (e.g. variance rising with a predictor), then plug in, that's feasible WLS, and it inherits the risk of a mis-modelled variance.
• It doesn't cure bias. WLS fixes efficiency and standard errors under heteroskedasticity; it does nothing about a regressor being correlated with the error. Endogeneity biases WLS exactly as it biases OLS.
• Extreme weights = fragility. If one point gets a huge weight, the whole fit hinges on it. A single mismeasured "high-precision" observation can then dominate, so sanity-check the largest weights.

The clean way to remember the family: OLS is "everyone votes equally," WLS is "votes weighted by reliability," and GLS adds "and untangle voters who are echoing each other." WLS is the middle rung, the everyday tool for Heteroskedasticity when your observations differ in precision but not in correlation.