Conditional Variational Autoencoders

A plain VAE trained on, say, digits 0–9 learns one shared latent space for everything, and if you want a sample of a specific digit you have no lever to pull — you sample a random $z$ and hope. The fix sounds almost too simple: tell the model the label every time, during both encoding and decoding, so the latent code is freed up to describe only what's left over — style, thickness, slant — rather than wasting capacity re-encoding "which digit is this."

The analogy: a form letter with a name field

A plain VAE is like a form letter that has to reconstruct the recipient's name from a compressed summary of the whole letter — wasteful, since the name is already known. A conditional VAE is the same form letter, but with an explicit name field: you fill in the name yourself, and the compressed summary only needs to capture the parts that actually vary between letters — tone, length, specific requests. Telling the model "this one is for Company X" up front means the latent code never has to rediscover Company X from scratch.

The mechanics: condition everything on $y$

A conditional VAE (CVAE) takes the same ELBO as a standard VAE but conditions every distribution on an extra input $y$ (a class label, a market regime, a tenor bucket — anything you know at generation time):

$$\text{ELBO}(x, y) = \mathbb{E}_{q(z|x,y)}\big[\log p(x|z,y)\big] - \text{KL}\big(q(z|x,y)\,\|\,p(z|y)\big)$$

In words: the encoder now looks at both the data and its label to produce $z$; the decoder reconstructs from both $z$ and the label; and the prior can itself depend on the label instead of being one fixed $\mathcal{N}(0, I)$ for everything. At generation time you pick the $y$ you want, sample $z$ from its prior, and decode — no guessing required.

Worked example 1: sampling on demand

Train a CVAE on handwritten digits with $y \in \{0, \dots, 9\}$ one-hot encoded and appended to both encoder and decoder inputs. To generate a "7": set $y =$ one-hot(7), sample $z \sim \mathcal{N}(0, I)$ (say $z = [0.3, -0.8]$ in a 2-D latent space), and decode $p(x \mid z=[0.3,-0.8],\, y=7)$. Change only the label to $y=3$, keep the same $z$: the decoder now produces a "3" with the same stylistic wobble that $[0.3,-0.8]$ encoded for a 7 — because $z$ never had to store "which digit," only "which style." A plain VAE cannot do this cleanly: its single $z$ tangles digit identity and style together, so swapping identity while holding style fixed isn't a defined operation.

Worked example 2: conditioning on a market regime

Suppose $x$ is a day's 1-minute return path and $y \in \{\text{calm}, \text{stressed}\}$ is a realized-vol regime flag known historically. A CVAE trained this way learns $p(x \mid z, y=\text{stressed})$ separately from $p(x \mid z, y=\text{calm})$ within one shared model. To stress-test a strategy, fix $y = \text{stressed}$ and sample many $z$'s: each decode is a different stressed-regime path, all sharing the property "this came from the stressed prior," letting a risk team generate a batch of plausible stress scenarios on demand rather than being stuck with the handful of stressed days actually observed in history.

Drag the mean and spread above and picture it as one class-conditional prior $p(z \mid y)$: a CVAE can give each label its own such prior rather than forcing every class through one identical $\mathcal{N}(0, I)$.

What this means in practice

CVAEs are the natural tool whenever generation needs to be steerable — synthetic scenario generation conditioned on a regime label, synthetic order-book snippets conditioned on a venue or time-of-day bucket, or portfolio return paths conditioned on a macro state. The label doesn't have to be a clean one-hot class either — it can be a continuous or structured context vector, as long as it's known at generation time.