Do Transformers Beat Linear Baselines for Forecasting?

Between 2021 and 2022 a series of papers introduced increasingly elaborate transformer architectures for long-horizon time-series forecasting — Informer, Autoformer, FEDformer — each adding new attention variants and claiming state-of-the-art results on standard benchmarks. Then a short, deliberately simple paper asked an uncomfortable question: what if you just fit a single linear layer per output channel, mapping the lookback window directly to the forecast window, with no attention at all? The answer, on the same benchmarks used to justify the transformer papers, was that this trivial model — called DLinear — matched or beat nearly all of them, on most datasets, most of the time. This result mattered less as a specific finding and more as a methodological correction: it showed the field had been comparing complicated models against weak baselines.

The analogy: judging a race with no clock

If ten increasingly elaborate running shoes are each claimed to be "the fastest ever" but no one ever timed a runner in ordinary sneakers on the same track, the comparisons are meaningless — you cannot tell whether the shoes are actually helping, or whether any runner would have posted a similar time regardless. That is what DLinear did to time-series transformers: it put "ordinary sneakers" — one linear layer — on the same track, and found the elaborate shoes often weren't winning by the margins the field assumed.

What DLinear actually is

DLinear first decomposes the series into a trend component (a moving average) and a seasonal/residual component (what's left over), then fits one separate linear layer to each component, mapping the lookback window directly to the future window:

$$\hat{y} = W_{\text{trend}}\, x_{\text{trend}} + W_{\text{seasonal}}\, x_{\text{seasonal}}$$

In words: the forecast $\hat y$ is just a weighted sum of the past trend values plus a weighted sum of the past seasonal-residual values, each through its own learned linear map — no attention, no nonlinearity, no positional encoding, nothing beyond ordinary linear regression applied separately to two decomposed pieces of the input window.

Worked example 1: why a linear map can already capture seasonality

Take a lookback window of 4 past values with a strong weekly pattern, and suppose the true rule is "next value equals the value from exactly 4 steps back plus a small drift." A single linear layer with weights $[0, 0, 0, 1.02]$ applied to $[x_{t-3}, x_{t-2}, x_{t-1}, x_t]$ reproduces this rule almost exactly: $\hat y = 1.02\, x_t$ (mostly ignoring the other three inputs, near-zero weight). Fitting that weight vector by ordinary least squares requires no attention mechanism at all — the "pattern" a transformer might discover through many layers of learned attention is, for many real seasonal series, just a fixed linear combination of a few lagged values, exactly what linear regression is built to find directly.

Worked example 2: benchmark numbers

On the widely used Electricity Transformer Temperature (ETT) dataset, at a 336-step forecast horizon, the original paper reported (mean squared error, lower is better):

Model	Test MSE
Informer (transformer)	1.059
Autoformer (transformer)	0.339
FEDformer (transformer)	0.331
DLinear (single linear layer)	0.240

DLinear posted the lowest error of the group — beating every transformer variant it was compared against on this setting, despite having orders of magnitude fewer parameters and no attention computation at all. This was the specific result that triggered the broader re-examination.

Drag the points above and watch the OLS line and residuals update — DLinear is, at its core, exactly this: an ordinary least-squares-style linear fit from lookback window to forecast window, per channel, nothing more exotic.

What this means in practice

The practical lesson is not "transformers are useless for forecasting" — later work including PatchTST and Patched Time-Series Transformers closed the gap and often beat DLinear by fixing tokenisation rather than adding architectural complexity — but "always benchmark against the simplest model that could plausibly work" before trusting a fancy architecture's reported gains. For a quant evaluating a new forecasting model on price or macro data, this result is a direct warning: fit an ordinary linear or ARIMA baseline on the same data split first, because a genuinely strong signal should still beat a simple model by a real margin, and if it doesn't, the extra complexity is not paying for itself.

Practice

1. Why would a linear model be expected to do relatively better on a series dominated by trend and fixed seasonality, versus one with irregular, state-dependent dynamics?
2. Design a one-sentence experiment you would run before trusting a new "state-of-the-art" forecasting transformer's reported benchmark numbers.