Autoformer and Series Decomposition Attention

Standard transformers treat a time series like a sentence, letting every point attend to every other point — but a long financial or sensor series is dominated by trend and repeating seasonality, and forcing generic attention to rediscover that structure from scratch wastes capacity and often overfits noise. Autoformer builds the structure in directly: before attention runs, each block splits the series into a slow-moving trend component and a seasonal (periodic) residual, using a moving average, and processes each separately.

For the seasonal part, Autoformer swaps standard point-to-point attention for auto-correlation: instead of asking "how similar is this point to every other point," it asks "at what lag does this series most resemble a shifted copy of itself," which is a far more natural question for something with a repeating pattern (like weekly retail sales or a daily volatility cycle) and is cheaper to compute than full attention.

Worked example. Given two years of daily retail sales with a clear weekly cycle, Autoformer's decomposition step extracts a smooth upward trend line plus a residual that repeats roughly every 7 days. Its auto-correlation attention then finds the strongest self-similarity at a lag of 7 days (correlation 0.81), letting the model forecast next Monday mainly from the last several Mondays rather than searching across all 730 days for relevant points.