Generative vs Discriminative Modeling

Suppose the task is: given a company's financial ratios, predict whether it will default within a year. Two entirely different modelling philosophies can both answer this. A discriminative model learns, directly, the boundary separating "defaults" from "doesn't default" in the space of input features — it never bothers modelling what defaulting companies' financials look like overall, only where the dividing line sits. A generative model instead learns the full probability distribution of financials for each class separately — and only at prediction time compares which distribution a new company's numbers are more consistent with. Both can produce the same yes/no answer. They differ sharply in what else they can do, how much data they need, and how they fail.

The analogy: describing a border versus describing both countries

A discriminative model is like someone who has only ever studied the border between two countries — they can tell you instantly which side of the line any coordinate falls on, but know nothing about either country away from that line. A generative model has studied both countries in full — geography, climate, culture — and derives the border only as a side-effect. The border-only expert is often faster and needs less study; the full-country expert can also answer entirely different questions ("generate a plausible new town that could exist there") the border-only expert has no way to answer.

What each side actually models

A discriminative model learns $P(y \mid x)$ directly — the probability of the label $y$ given the input $x$ — often by directly parameterising a decision boundary (logistic regression, most neural network classifiers).

A generative model learns $P(x \mid y)$, the distribution of inputs within each class, together with the class prior $P(y)$, and then derives $P(y\mid x)$ only indirectly via Bayes' rule:

$$P(y \mid x) = \frac{P(x \mid y)\, P(y)}{P(x)}$$

In words: the probability of a label given the data is worked out by combining how likely that data would be if it came from each class ($P(x\mid y)$, the generative part), weighted by how common each class is overall ($P(y)$), and normalised by the overall probability of seeing that data at all ($P(x)$) — the classification boundary emerges as a byproduct of fully modelling both classes, rather than being learned directly.

Worked example 1: Gaussian classes by hand

Suppose defaulting companies' debt ratio is modelled as $\mathcal{N}(\mu=0.8, \sigma=0.15)$ and healthy companies' as $\mathcal{N}(\mu=0.4, \sigma=0.15)$, with equal priors. A new company has debt ratio $x=0.6$. Evaluate each class's density: for default, $z = (0.6-0.8)/0.15 = -1.33$; for healthy, $z=(0.6-0.4)/0.15=1.33$ — symmetric distance, so both classes assign roughly equal likelihood ($\phi(\pm1.33)\approx0.165$), and with equal priors the model outputs close to a 50/50 split. This company sits almost exactly on the boundary the generative model implies — and it can also report how implausible this ratio is under either story, something a bare discriminative boundary cannot express.

Worked example 2: what each model can do with a new, malformed input

A discriminative logistic regression, given an impossible debt ratio $x=15$ (likely a data error), will still confidently output a probability near 0 or 1 — the decision boundary is just a line, and every point is on one side or the other, with no notion of "this doesn't look like anything I've seen." The generative model can compute $P(x\mid y)$ for both classes at $x=15$ and find both vanishingly small — flagging the point as implausible under either class, a form of built-in anomaly detection the discriminative model has no mechanism to provide.

Drag the complexity slider above — a discriminative model is exactly this: it fits the dividing boundary directly and never represents anything about the shape of each class's own data cloud, only where they separate.

What this means in practice

Most workhorse classifiers in quant workflows — logistic regression, gradient-boosted trees, most neural classifiers — are discriminative, because when the only goal is an accurate label given complete, well-behaved features, discriminative models are simpler to train and often more accurate for a fixed amount of data. Generative approaches earn their extra complexity when the task needs more than a label: detecting out-of-distribution inputs, generating synthetic scenarios (see Denoising Diffusion Probabilistic Models and Generative Adversarial Networks), or handling missing or corrupted features gracefully, since a generative model can still reason about $P(x)$ even when parts of $x$ are unobserved.

Practice

1. A dataset has only 200 labelled examples but a well-understood generative structure (e.g. two roughly Gaussian classes). Explain in one sentence why a generative model might outperform a discriminative one here, even though both could in principle fit the same boundary.
2. Why can a purely discriminative classifier not, on its own, generate a new synthetic example that looks like it belongs to a given class?