Sequence-to-Sequence Encoder-Decoder Models

A classifier or a plain regression network expects one fixed-size input and produces one fixed-size output. Plenty of real problems don't fit that mold at all: an English sentence of 8 words needs to become a French sentence of 11 words; a 60-day window of OHLCV data needs to become a 5-day forecast path, where the forecast isn't one number but a sequence with its own internal order (day 2's forecast should be informed by day 1's). Neither the input length nor the output length is fixed, and the two lengths don't need to match. Sequence-to-sequence (seq2seq) models were built specifically for this shape of problem: map an input sequence of any length to an output sequence of a different, independently chosen length.

The analogy: reading a whole report, then dictating a summary from memory

Imagine an analyst who reads an entire 40-page report once, closes it, and then dictates a 5-bullet summary out loud, one bullet at a time — each new bullet informed both by what's left in their memory of the report and by what they've already said in the summary so far, so bullet 3 doesn't repeat bullet 1. They don't flip back to page 12 while dictating bullet 3; they compressed the whole report into one mental digest first, and everything after that comes from that digest plus their own unfolding summary. That's the classic encoder-decoder split: an encoder reads the entire input sequence and compresses it into a single summary vector; a decoder then generates the output sequence one element at a time, conditioning each new element on that summary and on everything it has generated so far.

The maths

The encoder is a recurrent network (often an LSTM) that processes the input sequence $x_1, \dots, x_T$ step by step, producing a final hidden state:

$$h_t = \text{RNN}(h_{t-1}, x_t), \qquad c = h_T$$

In words: run the input through the recurrent network exactly as usual, but throw away every intermediate hidden state and keep only the last one, $h_T$, relabeled $c$ for context. Everything the decoder will ever know about the input is compressed into this single fixed-size vector — a real bottleneck, and, as the worked examples below show, the source of the classic seq2seq weakness.

The decoder is a second, separately-weighted recurrent network that generates the output sequence $y_1, \dots, y_S$ one token at a time, seeded with the context vector:

$$s_0 = c, \qquad s_j = \text{RNN}(s_{j-1}, y_{j-1}), \qquad y_j = \text{softmax}(W_o s_j)$$

In words: the decoder's hidden state starts at the encoder's final summary $c$. At each step $j$, it updates its state using the previous output token $y_{j-1}$ (this is why it's called autoregressive — it feeds its own past outputs back in as inputs) and produces a probability distribution over the vocabulary of possible next tokens via softmax, from which the actual token $y_j$ is chosen (typically the highest-probability one, or sampled). Generation continues until the decoder produces a special end-of-sequence token, which is how the output length ends up being whatever it needs to be rather than fixed in advance.

Worked example 1: a tiny numeric forecast, step by step

Suppose an encoder compresses a 3-day return history into a single scalar context $c = 0.85$ (a stand-in for "recent momentum is moderately positive" — in a real model this would be a vector, but the mechanics generalize). The decoder generates a 2-day forecast autoregressively. Its update rule, simplified to scalars, is $s_j = 0.5\, s_{j-1} + 0.3\, y_{j-1}$, and it reads out $y_j = s_j$ directly.

Step 0: $s_0 = c = 0.85$.

Step 1 (generating $y_1$, using a seed output $y_0 = 0$ as is conventional): $s_1 = 0.5(0.85) + 0.3(0) = 0.425$, so $y_1 = 0.425$.

Step 2 (generating $y_2$, now feeding back $y_1 = 0.425$): $s_2 = 0.5(0.425) + 0.3(0.425) = 0.2125 + 0.1275 = 0.34$, so $y_2 = 0.34$.

The forecast path is $(0.425, 0.34)$ — a gently decaying sequence, and note critically that $y_2$ depended on $y_1$, not just on the original context $c$. This is the autoregressive property in miniature: change $y_1$ (say, by having the decoder generate a different value there) and $y_2$'s computation changes too, exactly as a real multi-step forecast should have each day's prediction informed by the previous day's, not generated independently.

Worked example 2: the bottleneck, made concrete

Suppose the true task requires the decoder to reproduce, at output step 3, a specific detail that appeared only at input step 2 of a 10-step input sequence — say, an unusual volume spike on day 2 of a 10-day window that should specifically shape day 3 of the forecast. The encoder must pack all of the input's information — days 1 through 10 — into one fixed-size vector $c$. If $c$ has, say, 64 dimensions and the input sequence has 10 days each with 5 features (50 numbers total), the encoder is being asked to compress 50 numbers into 64 — seemingly fine dimension-wise, but in practice the network allocates its limited capacity toward whatever reduces average training loss across all input patterns it sees, not toward preserving any one specific rare spike perfectly.

As the input sequence length grows — 10 days becomes 100 becomes 1,000 — the same fixed-size $c$ is asked to summarize proportionally more, and the specific volume-spike-on-day-2 detail becomes progressively more likely to be washed out by everything else competing for space in that one vector. Empirically, plain encoder-decoder seq2seq models degrade sharply as sequence length grows for exactly this reason — it isn't a training bug, it's the fixed-size bottleneck being asked to do more than it structurally can.

A decoder's autoregressive generation is, in miniature, a sequence of estimates that should stabilize as more context accumulates — but note the seq2seq bottleneck isn't about running longer, it's about a fixed-capacity summary losing detail as the input it must compress grows, which no amount of additional training fixes on its own.

What this means in practice

The encoder-decoder split shows up anywhere a variable-length input needs to produce a variable-length, internally-ordered output: multi-day price or volatility forecasting, translating between differently-structured data feeds, or generating a sequence of trade instructions from a sequence of market observations. For short sequences (a handful to a few dozen steps) a plain LSTM-based seq2seq model is a reasonable, cheap baseline. The moment sequences get long or the task needs to reference specific distant details precisely, the fixed context-vector bottleneck becomes the binding constraint, and that's exactly the gap that attention-augmented and transformer-based sequence models were built to close.