Temporal Convolutional Networks

A recurrent network reads a time series one step at a time, carrying forward a running summary — which means step 500 cannot start until step 499 has finished, and it must compress everything relevant from the past into a single fixed-size memory. That's slow to train (no parallelism across time) and fragile over long histories (old information keeps getting overwritten). An alternative borrows the tool built for scanning images for local patterns — the convolution — and points it at a time axis instead of a pixel grid.

Picture a small window sliding along a strip of daily returns, at every position multiplying the values under it by a fixed set of weights and summing them up — the same operation a moving average performs, except the weights are learned rather than chosen to be equal. Stack several such windows on top of each other, each one scanning the output of the layer below, and the network builds up an ability to recognize longer and longer patterns, the way stacking small local filters in an image network builds up recognition of larger shapes from edges to textures to objects.

The mechanics, one symbol at a time

For a 1-D convolution with filter weights $w_1, \dots, w_k$ (a window of size $k$) applied to a sequence $x_1, \dots, x_T$, the output at position $t$ is:

$$y_t = \sum_{j=0}^{k-1} w_j \, x_{t-j}$$

In words: to compute the output at time $t$, take the $k$ most recent inputs ending at $t$, multiply each by its own learned weight, and add them up — a weighted local summary, like a moving average with weights the model gets to choose rather than $1/k$ each. Crucially the sum only reaches backward to $x_{t-j}$ for $j \ge 0$, never forward to $x_{t+1}$ — this is called a causal convolution, and it exists specifically so the model at time $t$ never gets to see the future when predicting it.

A single layer with window size $k$ only sees $k$ steps back. To see further without a huge window, TCNs use dilation: layer $\ell$ skips over inputs with a stride of $2^{\ell}$, so layer 1 looks at consecutive steps, layer 2 looks at every other step, layer 3 every fourth step, and so on. Stacking $L$ dilated layers gives a receptive field — the span of raw input a single output can "see" — that grows exponentially with depth, roughly $k \cdot 2^{L}$, instead of only linearly with a plain stack.

Worked example 1: one causal convolution by hand

A filter with $k=3$ weights $w = (0.5, 0.3, -0.2)$ (most recent input weighted heaviest) applied to daily returns $x = (\dots, 1.0, -0.4, 0.8, 1.5)$, where $x_t = 1.5$ is today.

$$y_t = 0.5(1.5) + 0.3(0.8) + (-0.2)(-0.4) = 0.75 + 0.24 + 0.08 = 1.07$$

This output uses only today and the two days before it — nothing from tomorrow — and combines them with weights the network learned rather than a fixed average. Slide the same filter one step and recompute using the next three values; it's applied identically at every position, which is what makes it a convolution rather than a one-off weighted sum.

Worked example 2: receptive field growth with dilation

Three stacked layers, filter size $k=2$, dilation doubling each layer: layer 1 dilation 1, layer 2 dilation 2, layer 3 dilation 4.

Layer 1's receptive field: it looks back $k-1 = 1$ step, so it spans 2 raw time points.

Layer 2 looks back $(k-1) \times \text{dilation} = 1 \times 2 = 2$ steps in layer 1's output, and each of those layer-1 outputs already spans 2 raw points, so layer 2's total reach is $2 + 2 = 4$ raw time points.

Layer 3 looks back $(k-1) \times 4 = 4$ steps in layer 2's output; adding that to what layer 2 already covers gives a receptive field of $4 + 4 = 8$ raw time points.

Just three layers, filter size 2, and the network already sees 8 days back — a plain (non-dilated) 3-layer stack with the same filter size would see only 4. Add a few more dilated layers and the receptive field reaches hundreds of days, all while every layer's computation for every time step can run in parallel, unlike an RNN.

The explorer above lets you see how a power-law curve accelerates; receptive field versus depth under dilation grows the same way — each additional layer roughly doubles the reach instead of adding a fixed amount.

What this means in practice

TCNs are used for return forecasting, volatility modeling, and order-book sequence tasks where long lookback windows matter and training speed matters — because every time step's convolution can run in parallel on a GPU, TCNs train substantially faster than an RNN over the same data. They're paired with residual connections almost by default (deep dilated stacks otherwise suffer the same gradient problems as any deep network) and with weight normalization for stability. The causal masking is non-negotiable in finance specifically: a convolution that peeks even one step forward is a leak of future information into a supposedly historical forecast, and that leak will make backtests look far better than live trading ever will.