Generative vs Discriminative Models

Ask two different people to sort a pile of photographs into "cats" and "dogs." One studies thousands of labeled examples and learns a rule purely about the boundary — pointy ears plus whiskers plus small size tends to fall on the cat side, everything else on the dog side. Ask them to draw a brand-new cat from imagination and they can't; they've never learned what a cat actually looks like as a whole, only how to tell it apart from a dog. The other person studies the same photographs but tries to internalize what a typical cat looks like and, separately, what a typical dog looks like — the full appearance, not just the distinguishing features. Ask them to sort a new photo and they compare it against both mental templates; ask them to draw a new cat and, in principle, they can, because they've actually modeled what a cat is, not merely where the dividing line falls. These are, respectively, a discriminative and a generative approach to the same classification task.

The mechanism: two different quantities to learn

Every classification problem has an input $x$ (the photo, the set of features) and a label $y$ (cat or dog). A discriminative model learns $P(y \mid x)$ directly — given this specific $x$, what's the probability of each label? — without ever modeling how $x$ itself was generated. Logistic regression is the standard example: it learns a decision boundary through feature space and stops there.

A generative model instead learns $P(x \mid y)$ for each class — what does a typical cat's feature vector look like, what does a typical dog's look like — plus the base rates $P(y)$, how common each class is overall. Classification then happens indirectly, via Bayes' rule:

$$P(y \mid x) = \frac{P(x \mid y)\, P(y)}{P(x)}$$

Here $P(x\mid y)$ (called the likelihood) says how plausible this particular $x$ is under class $y$'s learned template, $P(y)$ (the prior) says how common class $y$ is before seeing any features at all, and $P(x)$ is a normalizing constant — the total probability of seeing this $x$ under any class, ensuring the result is a valid probability. In words: a generative model doesn't jump straight to "which class"; it separately asks "how well does each class's template explain this data" and "how common is each class to begin with," then combines the two.

The crucial extra thing a generative model buys, beyond classification, is the ability to sample — because it has modeled $P(x\mid y)$, the full shape of each class's data, it can draw new, synthetic examples from that shape. A discriminative model has no such capability; it has never modeled $x$'s distribution at all, only the boundary that separates labels within it.

The decision-boundary explorer below lets you see how a classifier's dividing line adapts to the underlying data; picture the discriminative approach as fitting that line directly, and the generative approach as first fitting a cloud shape to each class and only then reading the boundary off where the two clouds cross.

Worked example 1: computing $P(y\mid x)$ from a generative model

Suppose a simple 1-feature model of daily return $x$ for "trending day" ($y{=}1$) versus "choppy day" ($y{=}0$) has learned $P(x\mid y{=}1) = 0.30$ and $P(x\mid y{=}0) = 0.05$ for a specific observed value $x=1.5\%$, with base rates $P(y{=}1)=0.4$ and $P(y{=}0)=0.6$ (trending days are less common). Applying Bayes' rule:

$$P(y{=}1 \mid x) = \frac{0.30 \times 0.4}{0.30 \times 0.4 + 0.05 \times 0.6} = \frac{0.12}{0.12+0.03} = \frac{0.12}{0.15} = 0.80$$

Despite trending days being the rarer class overall (only 40% base rate), this specific return value is so much more likely under the trending template ($0.30$ vs $0.05$) that the posterior probability swings heavily toward "trending," landing at $80\%$. A discriminative model given only this same $x$ would produce a similar final probability, but it would have arrived there by fitting a boundary directly, never separately quantifying "how typical is this $x$ for each class" or "how common is each class."

Worked example 2: why the generative route can also produce new samples

Using the same fitted class templates, suppose $P(x\mid y{=}1)$ is a normal distribution centered at $\mu=1.8\%$ with standard deviation $\sigma=0.6\%$. A generative model can draw a brand-new synthetic "trending day" return by sampling from $\mathcal{N}(1.8\%, 0.6\%^2)$ — for instance drawing $z=0.5$ standard deviations above the mean gives a synthetic sample of $1.8\% + 0.5(0.6\%) = 2.1\%$, a plausible new trending-day return that was never in the training data. A discriminative logistic regression, having only ever learned the boundary between the two classes, has no analogous "draw a new example of class 1" operation available at all — it was never given a distribution to sample from.

What this means in practice

Discriminative models are usually the better choice when classification accuracy is the only goal and there's plenty of labeled data, because they spend their entire learning capacity on the boundary that actually matters for the task, rather than on the harder job of modeling the full shape of each class's data. Generative models earn their extra complexity when you also need to handle missing features gracefully (a well-specified $P(x\mid y)$ can marginalize over what's missing), need to detect out-of-distribution inputs (data that fits neither class's template well is flagged as unusual, something a discriminative boundary can't naturally express), or need to generate synthetic data, as in synthetic scenario generation.

Practice

1. A generative model gives $P(x\mid y{=}1)=0.10$, $P(x\mid y{=}0)=0.40$, with priors $P(y{=}1)=0.7$, $P(y{=}0)=0.3$. Compute $P(y{=}1\mid x)$ using Bayes' rule.
2. Why can a generative model handle a missing feature more gracefully than a purely discriminative one, in terms of what each has actually learned?
3. Give one concrete reason a quant might prefer a discriminative model for a pure "will this trade be profitable, yes or no" classifier, even knowing a generative model would also let them sample synthetic trades.