Truncated BPTT and Window Length

Backpropagation through time unrolls a recurrent network across every timestep of a sequence and backpropagates the full chain of gradients from the last step all the way to the first. For a sequence of a few hundred or a few thousand steps — a year of daily bars, a day of minute bars — that means storing and differentiating through the entire unrolled chain at once, which is both memory-expensive and prone to vanishing or exploding gradients over such a long path.

The analogy: grading only the recent pages

Being graded on every single page of a year-long diary at once, all in one sitting, is impractical. Truncated BPTT instead only unrolls and grades the last $k$ pages at a time: the hidden state still carries forward continuously from page to page, but the gradient — the "grading" signal — only ever looks back $k$ pages before it stops.

The mechanics

Full BPTT differentiates the loss at time $T$ with respect to every parameter through the entire chain back to $t=1$. Truncated BPTT (TBPTT) instead splits the sequence into chunks of length $k$: within a chunk, it runs the forward pass and a full backward pass just for those $k$ steps, then carries the hidden state — but not its gradient history — forward into the next chunk as a fixed starting point.

$$h_t = \tanh(W h_{t-1} + U x_t)$$

In a simplified linear recurrence, $h_t \approx \gamma\, h_{t-1}$ for some effective decay factor $\gamma$, the contribution of an input from $k$ steps back to today's gradient scales roughly like $\gamma^k$ — restating: the further back a timestep is from the current truncation window, the more its gradient contribution has already decayed, so the window length $k$ directly controls how far back the network can be taught to look.

Worked example 1: chunk boundaries

A sequence of length $T=20$ with window $k=5$ splits into 4 chunks: steps 1–5, 6–10, 11–15, 16–20. Gradients computed during the chunk covering steps 11–15 never flow back past step 11 — no matter how relevant step 3 might have been, no credit or blame reaches it during that chunk's backward pass.

Worked example 2: how much of the past still gets credited

Take a toy recurrence $h_t = 0.9\,h_{t-1} + x_t$, where $0.9$ is the effective decay per step. The gradient contribution from an input $k$ steps back scales like $0.9^k$. With $k=3$: $0.9^3 = 0.729$ — nearly three-quarters of the signal from 3 steps back still reaches today. With $k=10$: $0.9^{10} = 0.349$ — only about a third survives from 10 steps back. Choosing window length $k$ this small effectively caps how far back the network is trained to notice anything, regardless of the sequence's true length.

The gradient contribution from k steps back decays exponentially with the decay factor raised to the k-th power — exactly the shape below:

What this means in practice

Window length $k$ is chosen based on how far back dependence plausibly matters for the task — a few days for a fast mean-reverting signal, weeks for a slower trend — balanced against GPU memory and training speed, since a longer $k$ means a longer chain to store and differentiate per chunk.