Bidirectional Recurrent Networks

A standard recurrent network reading a sequence left to right only ever knows what came before the current position — at the middle of a sentence or a time series, it has no idea what comes next. For many tasks where the whole sequence is already available at once, that's throwing away useful information.

The analogy: reading a sentence twice

To fully understand a word in the middle of a sentence, you might read it once left-to-right for context, and separately reread it right-to-left, then combine both readings before deciding what the word means. A bidirectional recurrent network does exactly this: one RNN processes the sequence forward, a second, independent RNN processes it backward, and their hidden states are combined at every position.

The mechanics

Two separate recurrences run over the same sequence: a forward one, $h_t^{\text{fwd}} = \text{RNN}_{\text{fwd}}(x_t, h_{t-1}^{\text{fwd}})$, sweeping left to right, and a backward one, $h_t^{\text{bwd}} = \text{RNN}_{\text{bwd}}(x_t, h_{t+1}^{\text{bwd}})$, sweeping right to left. The output at each position concatenates both:

$$h_t = \big[\, h_t^{\text{fwd}} \,;\, h_t^{\text{bwd}} \,\big]$$

In words: at every timestep, the model has access to a summary of everything before it and a summary of everything after it, stitched together — twice the context a unidirectional network could ever see at that same position.

Worked example 1: what each direction has "seen"

A length-4 sequence, positions 1–4. At position 2, the forward RNN's hidden state has only processed position 1 and 2 — it knows nothing about positions 3, 4. The backward RNN's hidden state at position 2, however, has already processed positions 4 and 3 (in reverse) before reaching 2 — it knows the future relative to position 2. Only by combining both does position 2's final representation have the full sequence's context.

Worked example 2: combining the two hidden states

At position 2, suppose the forward hidden state is a scalar $h_2^{\text{fwd}} = 0.5$ and the backward hidden state is $h_2^{\text{bwd}} = -0.3$. Concatenated: $h_2 = (0.5, -0.3)$. A simple output layer with weights $w = (1, 1)$ computes $w \cdot h_2 = 0.5 + (-0.3) = 0.2$ — a decision that used information from both directions, something neither RNN alone could produce.

What this means in practice

Bidirectional RNNs are only usable when the whole sequence is genuinely available at prediction time — labeling historical days after the fact, classifying a completed document, cleaning a finished dataset. They cannot be used for live, real-time forecasting, because the backward direction needs values from the future relative to the point being predicted, and those values simply don't exist yet at the moment a live forecast is made.