The Bayes Optimal Classifier

Imagine a weather forecaster with genuinely perfect knowledge — not a good model, but the actual true probability that tomorrow is rainy given every relevant fact about today's atmosphere. Even this all-knowing forecaster will sometimes call "sunny" and get rain, because on days that look 60% likely sunny, it does rain 40% of the time — that 40% isn't a modelling failure, it's honest uncertainty in the situation itself. The Bayes optimal classifier is exactly this all-knowing forecaster: the best decision rule achievable with perfect knowledge of the true probabilities, and its error rate is the floor beneath which no classifier, however sophisticated, can ever go.

The rule itself

For a classification problem with classes $y \in \{1, \dots, K\}$ and input features $x$, the Bayes optimal classifier picks whichever class is most probable given $x$:

$$\hat{y}_{\text{Bayes}}(x) = \arg\max_{k} \; P(y = k \mid x)$$

In words: for a given input $x$, look at the true probability of each possible class given that input, and pick the class with the highest one. That's the entire rule — not a learning algorithm, just a decision made with perfect information about the underlying probabilities. $P(y=k \mid x)$ means "the probability that the true class is $k$, given that we've observed $x$" — this is the exact same conditional-probability object from Bayes' theorem, just applied class by class.

The Bayes error rate is how often even this perfect rule is still wrong:

$$\text{Err}_{\text{Bayes}} = \mathbb{E}_x\big[1 - \max_k P(y=k \mid x)\big]$$

In words: at every point $x$, the best you can do is get it right with probability equal to the largest class probability there; the rest of the time — $1$ minus that largest probability — even the perfect rule is wrong, purely because the classes genuinely overlap at that $x$. Average this "at least this much error is unavoidable" quantity across all $x$ and you get the Bayes error rate.

Drag the model's flexibility here and watch how it draws a boundary between two colored classes. Notice that wherever the two colors' points genuinely overlap in the plot, no boundary — however flexible — can separate them cleanly; that overlap region is a visual picture of Bayes error. A perfectly flexible boundary reduces training mistakes in that overlap to zero by memorizing individual points, but that's overfitting, not a lower true error rate — the Bayes error there is fixed by how much the classes actually overlap, not by how cleverly you draw the line.

Worked example 1: computing Bayes error directly

Suppose $x$ takes one of two values, "high volume day" or "low volume day," and the true probabilities of the next day being "up" are known exactly: on high-volume days, $P(\text{up}\mid\text{high}) = 0.75$; on low-volume days, $P(\text{up}\mid\text{low}) = 0.55$. Suppose high-volume days occur 30% of the time, low-volume 70%.

The Bayes optimal classifier predicts "up" on high-volume days (since $0.75 > 0.25$) and "up" on low-volume days too (since $0.55 > 0.45$ — barely, but still the majority class). Its error rate on high-volume days is $1 - 0.75 = 0.25$; on low-volume days it's $1 - 0.55 = 0.45$. Weighting by how often each occurs:

$$\text{Err}_{\text{Bayes}} = 0.30 \times 0.25 + 0.70 \times 0.45 = 0.075 + 0.315 = 0.39$$

Even with perfect knowledge of these true probabilities, the best possible classifier is wrong 39% of the time — not because it's a bad classifier, but because "low volume day" genuinely only tilts 55/45 toward "up," and no amount of modelling skill turns a 55/45 tilt into certainty.

Worked example 2: a model doing worse than it looks

A researcher builds a classifier and measures 58% accuracy predicting next-day direction using the same setup as above. In isolation, 58% might look barely better than a coin flip and get dismissed as noise. But the Bayes optimal rate here is $1-0.39=61\%$, so a model at 58% is capturing about $\frac{58-50}{61-50}\approx73\%$ of the available edge above chance — far more flattering than "58%" suggests alone. Conversely, against a Bayes ceiling of 90% (a much cleaner problem), the same 58% would capture only $\frac{58-50}{90-50}=20\%$ of the available edge — a genuinely weak model. Raw accuracy means nothing without knowing, or estimating, the ceiling it's measured against.

What this means in practice

In real quant problems, the Bayes error rate is unknowable exactly, but the concept still governs expectations: financial markets are close to efficient, meaning the true class probabilities for "will this stock go up tomorrow" sit close to 50/50 for most stocks most days, which caps achievable accuracy near 50-55% even for the best possible model. A model reporting 70% next-day-direction accuracy on liquid large-cap equities is a red flag precisely because it implies a Bayes error rate lower than almost anyone believes plausible for that market — the far more likely explanation is a leak or an overfit backtest, not a genuinely learnable 70/30 split in the true probabilities.