Autoregressive Generation: PixelCNN and WaveNet

The most direct way to define a generative model is to write down $p(x)$ as a product of one-step-at-a-time conditional probabilities and predict each in turn: $p(x_1, \dots, x_n) = \prod_i p(x_i \mid x_1, \dots, x_{i-1})$. No latent variable, no adversarial game — just predict the next element given everything before it, exactly, using the chain rule of probability, which holds for any distribution with no approximation at all. PixelCNN applies this to images, pixel by pixel; WaveNet applies it to raw audio, sample by sample. Both live or die on one architectural guarantee: the model predicting position $i$ must never be allowed to see positions $i, i+1, \dots$

The analogy: filling in a crossword, one square at a time, blind to the answer key

Autoregressive generation is like solving a crossword by filling in one square at a time, using only the squares already filled and the clues — never peeking at squares not yet reached, and never at the answer key. Each square's fill is a genuine prediction conditioned only on what's legitimately known so far. If you accidentally let yourself glance at a not-yet-filled square while solving the current one, you're not really solving the puzzle sequentially anymore — you're cheating, and the "prediction" you write down is not one the model could actually make at generation time, when future squares genuinely don't exist yet.

The masking trick that enforces this

A standard convolution centered on pixel $i$ sees neighbors on every side, including "future" ones in raster-scan order. PixelCNN's masked convolutions zero out the filter weights that would look at not-yet-generated pixels, so the receptive field only ever includes pixels above and to the left (in scan order). WaveNet does the equivalent in 1-D with causal convolutions, restricting each output at time $t$ to inputs at times $\le t$, then stacking dilated versions of these (as in QuantGAN) to reach far back in the sequence cheaply:

$$p(x) = \prod_{t=1}^{T} p\big(x_t \mid x_1, \dots, x_{t-1}\big)$$

In words: the whole sequence's probability is the product of each element's probability given only the elements strictly before it — no term in this product is allowed to depend on anything to its right.

Worked example 1: computing a sequence probability from per-step predictions

Suppose a trained autoregressive model outputs these categorical probabilities for a 3-symbol binary sequence: $p(x_1{=}1) = 0.6$, then given $x_1=1$, $p(x_2{=}0\mid x_1{=}1)=0.7$, then given $x_1=1,x_2=0$, $p(x_3{=}1\mid x_1{=}1,x_2{=}0)=0.4$. The probability of the specific sequence $(1,0,1)$ is the product: $0.6 \times 0.7 \times 0.4 = 0.168$. Every autoregressive log-likelihood computation is exactly this — a sum of log-probabilities, one per position, each conditioned only on genuinely-prior positions — which is also directly why training is easy: teacher-forcing feeds the real previous values at every position in parallel and computes all these conditional log-probabilities in one forward pass.

Worked example 2: why generation is still sequential even though training isn't

Training can compute all $T$ conditional probabilities in one parallel forward pass, because the real $x_1,\dots,x_{t-1}$ are already known for every $t$ simultaneously. Generation cannot: to sample $x_3$ you need the actual sampled $x_1, x_2$ (not the real training data, which doesn't exist yet), so generating a length-1,000 sequence requires 1,000 sequential forward passes, one per new sample, each depending on the output of the last. If one forward pass costs 2 milliseconds, generating one 1,000-length sample costs roughly $1000 \times 2\text{ms} = 2$ seconds — versus training, where a full batch of many such sequences costs about the same 2 milliseconds per sequence position, computed all at once. This training/generation asymmetry is the central practical cost of every purely autoregressive generator.

A Brownian path is itself autoregressive in the loosest sense — each step depends only on where the path currently is; PixelCNN and WaveNet are the same idea with the "next step" rule learned by a deep network instead of fixed by a formula.

What this means in practice

Autoregressive generation is the natural choice whenever exact likelihoods matter (for model comparison, anomaly scoring, or compression) and when data is naturally ordered — which includes essentially all financial time series. It's also the standard second stage after a VQ-VAE compresses data to discrete tokens, since a discrete token sequence is exactly the input shape autoregressive models were built for.