How Far Back Should You Look? Andrews and the Bandwidth Problem

Robust standard errors are one of the great success stories of applied econometrics. White showed how to survive unequal error variances. Newey and West showed how to survive errors that are correlated over time. Between them, they made honest inference possible in data that violates every classical assumption.

But there was a loose thread, and it was a big one. Newey-West standard errors require you to choose a number, and nobody knew how to choose it.

That number, called the bandwidth or the lag truncation parameter, controls how far back in time the correction looks when measuring how correlated your errors are. And it is not a technicality. Change it, and your t-statistic changes. Change it enough, and your conclusion changes. In practice, researchers picked a round number, or used a rule of thumb they half-remembered, and moved on.

Donald Andrews's 1991 paper is the definitive treatment of how that number should actually be chosen. It is a technical paper about an unglamorous problem, and it is one of the most consequential pieces of plumbing in modern econometrics.

The problem: a tuning knob with no instructions

Recall what the Newey-West correction does. Your regression errors are correlated over time, which means your data contains less independent information than the raw observation count suggests. To correct your standard error, you need to measure that correlation: look at the relationship between an error and the error one period later, two periods later, and so on. Add up the relationships, and inflate the standard error accordingly.

The question is: how many periods back do you look?

You are caught between two failure modes, and they pull in opposite directions.

Look too far back, and you add noise. The correlation at lag 40 is estimated from very few effective observations. It is essentially a random number. Including it, and lags 41 through 60 as well, injects a large amount of noise into your correction and makes the resulting standard error erratic and unreliable.

Look too short, and you miss real correlation. If the errors genuinely have memory extending twenty periods and you only look back five, you have failed to capture most of the redundancy in your data. You remain overconfident and your t-statistics remain inflated.

This is a bias-variance tradeoff, exactly the same shape as the one that runs through all of statistics, and it deserved a principled answer rather than a shrug.

The key idea via analogy: how much of the echo is worth measuring

Imagine you clap in a large hall and want to measure how much the sound echoes, so that you can correct for the echo in a recording.

You record the decaying reverberation. The first half-second is loud, clear, and clearly a real echo. By three seconds in, what you hear is mostly the hum of the air conditioning and the noise floor of your microphone. If you include those three seconds in your estimate of the echo, you have not measured more echo. You have measured mostly your own equipment noise, and added it to your correction.

But if you cut off after a tenth of a second, you have missed most of the genuine reverberation and your correction is badly incomplete.

The right cutoff depends on how fast the echo actually decays in this particular hall. A cathedral needs a longer window than a bedroom. And the key insight is that you can estimate how fast the echo decays from the recording itself, and then set your window accordingly.

That is Andrews's contribution in a sentence. He derived, formally, what the optimal bandwidth is as a function of how persistent the errors actually are, and then provided a data-dependent automatic procedure that estimates that persistence from your data and picks the bandwidth for you.

If your errors decay quickly, the procedure chooses a short window. If they are highly persistent, it chooses a long one. You stop guessing, and the data chooses.

The other half: which kernel

Andrews also addressed the second choice nobody knew how to make. Newey-West weights nearby lags heavily and distant ones lightly, using a specific triangular pattern of declining weights. That pattern is called a kernel, and Newey and West's choice, the Bartlett kernel, was made because it guaranteed a sensible non-negative answer, not because it was optimal.

Andrews studied the whole family of possible weighting schemes and worked out which ones are asymptotically optimal, in the sense of minimising the estimation error of the covariance matrix. The quadratic spectral kernel emerged as the optimal choice within an important class. He also showed which kernels preserve the crucial non-negativity property.

So the paper hands you both missing pieces: which weighting scheme to use, and how wide to make it.

Why it mattered

• It closed a hole in the most-used tool in applied econometrics. Newey-West standard errors are everywhere. Before Andrews, they contained an arbitrary user choice that could materially change conclusions. After, there was a defensible automatic answer.
• It is built into the software. When your statistical package offers "automatic bandwidth selection" alongside Newey-West standard errors, that is Andrews. Most people who use it have never read the paper, which is the highest compliment a piece of methodological plumbing can receive.
• It made results more reproducible. If two researchers analyse the same data with the same automatic procedure, they get the same standard errors. When bandwidth was a judgement call, they might not, and the difference could be the difference between a publishable result and nothing.
• It matters directly for anyone with overlapping data. Long-horizon return predictability regressions, momentum studies with overlapping formation periods, any strategy evaluated on rolling windows: all of these have serially correlated errors and all of them depend on getting the bandwidth roughly right. Using a bandwidth that is too short is a standard route to a t-statistic that lies.

The honest limitations

• Automatic does not mean correct. The data-dependent procedure estimates the persistence of the errors from a preliminary model. That preliminary model is itself a choice, and if it is wrong, the bandwidth it recommends is wrong. The arbitrariness has been pushed back a level, not eliminated. This is a recurring pattern in econometrics and it is worth naming honestly.
• The finite-sample behaviour is still poor when errors are very persistent. This is the important one. Andrews solved the asymptotic optimality problem. But when errors are highly autocorrelated and your sample is not enormous, HAC standard errors remain substantially too small no matter which bandwidth you pick. You are still overconfident. A whole later literature, on fixed-bandwidth or fixed-b asymptotics, exists precisely because the optimal-bandwidth framework does not deliver honest inference in the sample sizes people actually have.
• The optimal bandwidth is optimal for estimating the covariance matrix, not for testing. These are subtly different objectives. The bandwidth that gives you the most accurate variance estimate is not necessarily the one that gives your hypothesis test the best size and power properties. Later work made this point sharply.
• It cannot fix a misspecified model. Serial correlation in your residuals is very often a symptom of something else: an omitted variable, a wrong functional form, a spurious regression on non-stationary data. Applying an optimally-chosen robust standard error to such a model gives you a beautifully calibrated confidence interval around an estimate that is pointing at the wrong thing.
• It does not create information. If your errors are so persistent that your five hundred observations contain the information of twenty, no correction gives you back the other four hundred and eighty. Andrews tells you the truth about how little you know. He cannot make you know more.

The one-line takeaway

Andrews took the arbitrary knob at the heart of every robust standard error, how far back to look when measuring the memory in your errors, derived what its optimal setting actually is, and built a procedure that lets the data choose, which is why "automatic bandwidth" is a checkbox in your software and not a decision you have to defend.