Discriminative vs Generative Models

Imagine training two currency experts to spot counterfeit banknotes. The first is shown thousands of real and fake notes side by side and learns, directly, a rule for telling them apart — texture here, color there — without ever thinking about why a real note looks the way it does. The second studies the actual printing process of genuine currency — ink chemistry, paper fiber, press mechanics — building a model of how a real note comes to exist at all. Both can spot a fake. But only the second, having modelled what a genuine note actually is, could also sketch a new genuine note, or recognize something that's neither a convincing fake nor a real note — a photocopy, say — as belonging to neither category. That's the discriminative/generative divide.

What each one actually learns

A discriminative model learns $P(y \mid x)$ directly — the probability of the label $y$ given the input $x$ — or, more directly still, just a decision boundary separating classes, without ever modelling how $x$ itself arises. Logistic regression, support vector machines, and most neural network classifiers are discriminative: they are trained purely to draw the line, and everything about where the data came from is irrelevant to that goal.

A generative model learns $P(x \mid y)$ — how the input data itself is distributed within each class — together with $P(y)$, the overall frequency of each class, and combines them via Bayes' theorem to get the same $P(y \mid x)$ a discriminative model targets directly:

$$P(y \mid x) = \frac{P(x \mid y)\, P(y)}{P(x)}$$

In words: the probability of a class given the data equals how likely that class was to produce data looking like this ($P(x\mid y)$), times how common that class is overall ($P(y)$), divided by how likely this data is under any class at all ($P(x)$, which normalizes everything to sum to one). Naive Bayes, Gaussian mixture models, and hidden Markov models are generative: they explicitly model $P(x \mid y)$ for every class, and the classification boundary emerges as a side effect of comparing those class-conditional models, rather than being the direct training target.

A discriminative classifier is trained to draw exactly the boundary shown here, and nothing more — it has no opinion about what the point cloud on either side "looks like" beyond where the boundary should sit. A generative approach to the same data would instead fit a full distribution to each colored cluster separately, and the boundary in this same picture would fall out automatically wherever those two fitted distributions cross — same picture, opposite direction of construction.

Worked example 1: same classification, two different amounts of information used

Two classes of daily return regime, "calm" and "stressed," each with one feature, realized volatility $x$. Suppose "calm" days have $x\sim N(1.0\%,0.3\%)$ and "stressed" days have $x\sim N(3.5\%,1.2\%)$, with calm days occurring 80% of the time.

A generative approach fits these two normal distributions from labelled data, then classifies a new day with $x=2.0\%$ by computing $P(x=2.0\mid\text{calm})$ and $P(x=2.0\mid\text{stressed})$, weighting each by class frequency, and picking the larger. This also directly tells you how unusual $x=2.0\%$ is under each hypothesis — both densities are fairly low there, meaning $2.0\%$ sits in a gap between the two regimes, information a discriminative model discards.

A discriminative approach — logistic regression on the same data — fits a single boundary directly, say "predict stressed if $x>1.9\%$," and for $x=2.0\%$ just outputs "stressed" with some probability, with no representation of what a "typical" day's volatility looks like, only where the dividing line sits.

Worked example 2: what the generative model can do that the discriminative one can't

Suppose a genuinely new kind of day occurs — extremely low volatility, $x=0.05\%$, far below anything seen in either training class (a market holiday, say). The discriminative boundary at $1.9\%$ classifies it confidently as "calm," since $0.05\%<1.9\%$ — it has no way to express doubt, because it never modelled what calm days actually look like, only which side of the line this point falls on.

The generative model computes $P(x=0.05\mid\text{calm})$ using the fitted $N(1.0\%,0.3\%)$ density and gets a value near zero — more than three standard deviations below the calm mean, essentially unseen territory for that class too. It can flag this point as unlikely under either class, not just assign it to the nearer side of a line. Because it models the data-generating process for each class, it recognizes when new data doesn't resemble any class well, something a purely discriminative boundary structurally cannot do.

What this means in practice

Most production classifiers in quant finance are discriminative, because the actual task — predict a label or probability from features — is exactly what discriminative training optimizes directly, and it typically needs less data than fitting full class-conditional distributions. Generative approaches earn their place where the extra capability matters: simulating synthetic scenarios for stress testing, detecting anomalous days resembling no known regime, or handling missing input features, which a generative model can marginalize over in a principled way.