WaveNet and Dilated Causal Convolutions

A plain causal convolution stack can see further into the past only by adding more layers — one extra layer typically buys only a few extra timesteps of context. Covering weeks of tick data this way would need an impractically deep stack. Is there a way to cover a huge stretch of history using only a handful of layers?

The analogy: radar rings that skip further each time

Picture a series of radar sweeps down a street, each one spaced further apart than the last — the first sweep checks every step, the next checks every second step, the next every fourth step, and so on. After only a few sweeps, you've covered a huge distance using barely more equipment, because each sweep's spacing compounds on the one before it. Dilated convolutions do exactly this to a time series: each layer's kernel samples inputs spaced further apart than the layer before it.

The mechanics

Layer $i$ uses dilation $2^i$: with kernel size $k$, layer $i$'s output at time $t$ depends on $x_t, x_{t-2^i}, x_{t-2\cdot2^i}, \dots$ — always causal, per causal padding, but sampling positions spaced $2^i$ apart. After $L$ stacked layers with dilations $1, 2, 4, \dots, 2^{L-1}$, the overall receptive field is:

$$\text{RF} = 1 + (k-1)\sum_{i=0}^{L-1} 2^i = 1 + (k-1)(2^L - 1)$$

In words: because each layer's dilation doubles the one before it, the total span of past timesteps a prediction can draw on grows exponentially with the number of layers, not linearly — a small increase in depth buys a disproportionately large increase in lookback.

Worked example 1: receptive field with 4 layers

Kernel size $k=2$, dilations $1,2,4,8$ across 4 layers: $\text{RF} = 1 + (2-1)(1+2+4+8) = 1 + 15 = 16$ timesteps of context, using only 4 layers.

Worked example 2: compare to non-dilated layers

Same kernel size $k=2$, but 4 plain (non-dilated, dilation $=1$ throughout) causal conv layers: $\text{RF} = 1 + (2-1)(4) = 5$ timesteps. Dilation delivered $16$ timesteps of context versus $5$ for the same 4 layers and same kernel size — roughly $3\times$ more lookback for identical depth, and the gap widens further as more layers are added, since the non-dilated case grows only linearly while the dilated case keeps doubling.

Receptive field size against depth grows exactly like this — watch the curve as the exponent increases:

What this means in practice

Dilated causal convolutions are attractive for long financial time series because they capture long lookback windows — weeks of tick data — cheaply, and because all timesteps within a layer compute in parallel, unlike a recurrent network which must process one step at a time; training is correspondingly much faster, while remaining strictly causal for legitimate live forecasting use.