Naive Bayes Classifiers

You want to classify something — spam or not spam, fraud or not fraud — using several pieces of evidence at once. Bayes' rule tells you exactly how to combine evidence with a prior belief, but it needs the joint probability of seeing all that evidence together given each class, and with even a dozen features, estimating a full joint distribution from limited data is hopeless. Naive Bayes escapes this by making a bold simplification and living with the consequences.

The analogy: a doctor reading symptoms one at a time

A doctor suspects a patient has one of two conditions and observes several symptoms: fever, cough, fatigue. The fully correct approach would account for how these symptoms interact — fever and cough often occur together, and their combination carries different information than either alone. Naive Bayes is the doctor who instead asks, one symptom at a time, "given condition A, how likely is a fever? How likely is a cough? How likely is fatigue?" — and just multiplies those separate likelihoods together, pretending the symptoms never influence each other. It's a shortcut, not the truth, but it turns an intractable estimation problem into three easy ones.

The maths, one piece at a time

Bayes' rule says the probability of class $C$ given features $x_1, \dots, x_k$ is proportional to the prior probability of $C$ times the probability of seeing those features under $C$:

$$P(C \mid x_1, \dots, x_k) \propto P(C) \, P(x_1, \dots, x_k \mid C)$$

$P(C)$ is how common the class is before seeing any evidence — the base rate. $P(x_1,\dots,x_k \mid C)$ is the hard part: the joint likelihood of every feature together, given the class. The naive assumption replaces that joint likelihood with a product of separate, one-feature likelihoods:

$$P(x_1, \dots, x_k \mid C) \approx \prod_{j=1}^{k} P(x_j \mid C)$$

In words: instead of asking "how likely is this whole combination of symptoms," ask "how likely is each symptom on its own," and multiply the answers. That is the entire "naive" part of the name — treating features as independent given the class, which is rarely exactly true (a cough and a fever really do move together) but is often close enough to be useful.

Worked example 1: spam filtering by hand

Two classes: spam ($P(\text{spam}) = 0.4$) and not-spam ($P(\text{ham}) = 0.6$). An email contains the words "free" and "winner." From training data: $P(\text{free}\mid\text{spam}) = 0.6$, $P(\text{winner}\mid\text{spam}) = 0.5$; $P(\text{free}\mid\text{ham}) = 0.1$, $P(\text{winner}\mid\text{ham}) = 0.05$.

Spam score: $0.4 \times 0.6 \times 0.5 = 0.12$. Ham score: $0.6 \times 0.1 \times 0.05 = 0.003$. Normalise by dividing each by their sum, $0.123$: $P(\text{spam}\mid\text{email}) = 0.12/0.123 \approx 0.976$. Despite ignoring any interaction between "free" and "winner" appearing together, the classifier is overwhelmingly confident this is spam — and it usually would be right.

Worked example 2: a numeric feature and Gaussian Naive Bayes

For continuous features like trade size, replace the lookup table with a Gaussian per class. Suppose "legitimate" trades have mean size $500 and standard deviation $100, and "fraudulent" trades have mean $5,000 and standard deviation $2,000. A new trade of $4,000 arrives. Using the normal density formula for each class (roughly, how many standard deviations away and how tall the bell curve is there), $4,000 is 17.5 standard deviations from the legitimate mean — essentially zero density — but only 0.5 standard deviations from the fraud mean, where the density is high. Even before weighting by priors, the likelihood ratio overwhelmingly favours fraud, illustrating how Naive Bayes handles numeric features: fit a simple distribution per class per feature, then multiply.

Picture two of these bell curves overlaid, one centred near $500 and a wider one centred near $5,000 — a new point's classification comes from which curve it sits under with higher density, which is exactly the mechanism in worked example 2.

Naive Bayes with Gaussian features draws boundaries that are curved but constrained by the independence assumption — drag the explorer above to compare how a flexible boundary differs from the more rigid shapes Naive Bayes is capable of.

What this means in practice

Naive Bayes is fast, needs little data per parameter, and is a strong baseline for text classification, spam detection, and quick fraud triage where features are numerous and roughly informative on their own. It struggles when features are strongly correlated in a way that matters — two nearly-identical features get double-counted, since the model has no mechanism to notice they're saying the same thing twice.