PatchTST and Patched Time-Series Transformers

Feed a transformer a time series the naive way — one token per individual timestep — and two problems appear immediately. First, a single timestep of a price series or a sensor reading carries almost no information on its own; a lot of the meaningful structure lives in short local patterns (a rising streak, a dip-and-recover shape), which get diluted when the model has to reconstruct them one point at a time. Second, a long series of, say, 1,000 individual timesteps means 1,000 tokens, and attention's cost grows with the square of that number (see The Quadratic Cost of Attention). PatchTST fixes both at once with the same idea Vision Transformers used for images: don't tokenise the smallest unit, tokenise a small contiguous chunk.

The analogy: reading a heart-rate trace in beats, not in samples

A cardiologist reading an ECG does not reason about it one voltage sample at a time — they read short segments, each a few dozen samples wide, and recognise shapes within them ("that's a normal QRS complex," "that's an irregular beat"). The individual sample matters far less than the local shape it belongs to. PatchTST tokenises time series the same way: instead of one token per timestep, group $P$ consecutive timesteps into one patch, flatten that little window into a vector, and treat the whole patch — the shape, not the individual point — as a single token for the transformer to reason over.

Patching, formally

Given a univariate series of length $L$, split it into patches of length $P$ with stride $S$ (how far the window moves each step; $S=P$ gives non-overlapping patches, $S<P$ gives overlap):

$$N = \left\lfloor \frac{L - P}{S} \right\rfloor + 1$$

In words: the number of patch-tokens $N$ depends on how many times a window of length $P$ can slide across a series of length $L$ in steps of $S$ — far fewer tokens than the original $L$ timesteps whenever $P>1$.

Each patch (a vector of $P$ raw values) is linearly embedded into the model's dimension $d$, exactly as in Vision Transformers and Patch Embeddings, and PatchTST additionally treats each variable in a multivariate series independently — a separate univariate patching-and-attention pass per channel, sharing weights across channels — rather than mixing all variables together at every timestep.

Worked example 1: token count reduction

A series of $L=336$ timesteps, patch length $P=16$, stride $S=8$ (50% overlap): $N = \lfloor(336-16)/8\rfloor + 1 = 41$ tokens. Compare to the naive approach's 336 tokens — an 8× reduction in sequence length fed to attention. Since attention cost scales as $n^2$, cutting the token count by 8× cuts attention compute by roughly $8^2=64\times$, while each of the 41 tokens now carries a whole 16-step shape's worth of information rather than one bare number.

Worked example 2: why shape beats point in a concrete series

Take a toy daily price series with a repeating weekly pattern: [100, 102, 101, 105, 108, 104, 100] repeated three times with small noise. A per-timestep tokenisation must let the model discover, purely through attention weights over 21 individual noisy numbers, that "positions 1,8,15" behave similarly. A patch of length 7 (one week) turns each repetition into a single token whose flattened value directly encodes the whole week's shape — three tokens instead of 21, and the model's job becomes "notice these three patch-tokens look similar" rather than "reconstruct the weekly shape from scattered individual points first." The inductive bias of patching does real work here, not just token-count reduction.

Cutting token count by a factor and watching cost fall by its square, as this curve's shape suggests, is exactly why patching's modest reduction in tokens produces an outsized drop in attention compute.

What this means in practice

PatchTST was notable for beating more architecturally elaborate transformer forecasters (Informer, Autoformer) largely through this one simplification plus a channel-independent design, which pushed the field to question whether prior added complexity — specialised sparse-attention variants for time series — was actually earning its keep. It is a standard baseline architecture now for long-horizon forecasting tasks, including financial time series, where local shape (a breakout pattern, a mean-reversion wiggle) genuinely carries more signal than isolated points.

Practice

1. With $L=500$, $P=25$, $S=25$ (no overlap), how many patch tokens result, and how does that compare to $S=12$ (roughly 50% overlap)?
2. Why does channel-independent patching (a separate pass per variable, shared weights) reduce the number of parameters needed compared to mixing all variables into every patch token?