WaveNet and Dilated Causal Convolutions

A plain causal convolution stack can see further into the past only by adding more layers, one extra layer typically buys only a few extra timesteps of context. Covering weeks of tick data this way would need an impractically deep stack. Is there a way to cover a huge stretch of history using only a handful of layers?

The analogy: radar rings that skip further each time

Picture a series of radar sweeps down a street, each one spaced further apart than the last, the first sweep checks every step, the next checks every second step, the next every fourth step, and so on. After only a few sweeps, you've covered a huge distance using barely more equipment, because each sweep's spacing compounds on the one before it. Dilated convolutions do exactly this to a time series: each layer's kernel samples inputs spaced further apart than the layer before it.

The mechanics

Layer $i$ uses dilation $2^i$: with kernel size $k$, layer $i$'s output at time $t$ depends on $x_t, x_{t-2^i}, x_{t-2\cdot2^i}, \dots$, always causal, per causal padding, but sampling positions spaced $2^i$ apart. After $L$ stacked layers with dilations $1, 2, 4, \dots, 2^{L-1}$, the overall receptive field is:

In words: because each layer's dilation doubles the one before it, the total span of past timesteps a prediction can draw on grows exponentially with the number of layers, not linearly, a small increase in depth buys a disproportionately large increase in lookback.

Worked example 1: receptive field with 4 layers

Kernel size $k=2$, dilations $1,2,4,8$ across 4 layers: $\text{RF} = 1 + (2-1)(1+2+4+8) = 1 + 15 = 16$ timesteps of context, using only 4 layers.

Worked example 2: compare to non-dilated layers

Same kernel size $k=2$, but 4 plain (non-dilated, dilation $=1$ throughout) causal conv layers: $\text{RF} = 1 + (2-1)(4) = 5$ timesteps. Dilation delivered $16$ timesteps of context versus $5$ for the same 4 layers and same kernel size, roughly $3\times$ more lookback for identical depth, and the gap widens further as more layers are added, since the non-dilated case grows only linearly while the dilated case keeps doubling.

Receptive field size against depth grows exactly like this, watch the curve as the exponent increases:

What this means in practice

Dilated causal convolutions are attractive for long financial time series because they capture long lookback windows, weeks of tick data, cheaply, and because all timesteps within a layer compute in parallel, unlike a recurrent network which must process one step at a time; training is correspondingly much faster, while remaining strictly causal for legitimate live forecasting use.