Cross-Entropy and Log Loss

Suppose a model outputs "there is a 95% chance this loan does not default." The loan defaults. How bad was that prediction? Simple accuracy has no answer beyond "wrong" — it cannot distinguish a model that said 51% from one that said 99%, even though the second one should be punished far more severely for being so confidently mistaken. Cross-entropy, also called log loss, is the standard way to score a probability forecast rather than just a label, and it is built around one simple idea: punish confidence that turns out to be misplaced, and punish it harder the more confident it was.

The analogy: a bookmaker who bets his own odds

Imagine a bookmaker who is required to actually stake money according to the odds he quotes. If he says a horse has a 90% chance of winning and it loses, he loses a lot — he was staking heavily on being right. If he says 55% and it loses, he loses only a little, because he was barely committing to that outcome at all. A bookmaker who is well calibrated — right roughly as often as his stated probabilities imply — loses only a small, steady amount over time. One who is overconfident and wrong gets wiped out fast, because his losses on bad calls scale up faster than his gains on good ones.

Cross-entropy scores a model exactly like that bookmaker's ledger. Every prediction is treated as a bet sized by the stated probability, and the penalty for being wrong grows the more sure the model claimed to be.

The formula, one symbol at a time

For a single binary outcome, let $y$ be the true label (1 if the event happened, 0 if it did not) and $p$ be the model's predicted probability that it happens. The loss for that one prediction is

$$L(y, p) = -\big[\, y \ln(p) + (1-y)\ln(1-p) \,\big]$$

Read it in two pieces, only one of which is ever active for a given example. If the event actually happened ($y=1$), the second term vanishes and the loss is just $-\ln(p)$ — the model is graded only on how much probability it put on the thing that actually occurred. If the event did not happen ($y=0$), the first term vanishes and the loss is $-\ln(1-p)$ — graded on how much probability it put on the correct outcome, "no." The symbol $\ln$ is the natural logarithm; the reason it appears at all is that $-\ln(p)$ is the standard measure of "surprise" for an event with probability $p$ — it is exactly zero when $p=1$ (an event you were certain of causes no surprise when it happens) and it grows without bound as $p \to 0$ (an event you thought was nearly impossible causes unbounded surprise if it happens anyway). Cross-entropy is simply the average surprise, across all your predictions, at the outcomes that actually occurred. For $K$ possible classes with one-hot true label vector $y_c$ and predicted probabilities $p_c$, the same idea generalizes to $L = -\sum_{c=1}^{K} y_c \ln(p_c)$ — in words, look up the probability the model assigned to whichever class actually happened, and take the negative log of just that one number; the other classes' probabilities don't enter the formula for that example at all.

Worked example 1: four loans, one bad miss

A credit model scores four loans. For each, the loan's true outcome and the model's predicted default probability:

Loan	Outcome	P(default) predicted	Loss
1	no default ($y=0$)	0.10	$-\ln(0.90) = 0.105$
2	default ($y=1$)	0.80	$-\ln(0.80) = 0.223$
3	default ($y=1$)	0.05	$-\ln(0.05) = 2.996$
4	no default ($y=0$)	0.02	$-\ln(0.98) = 0.020$

$$\text{average log loss} = \frac{0.105 + 0.223 + 2.996 + 0.020}{4} = \frac{3.344}{4} = 0.836$$

Loan 3 alone contributes more than three times the sum of the other three losses combined. The model was confidently wrong — it said only a 5% chance of default on a loan that then defaulted — and log loss punishes exactly that mistake severely, on purpose. An accuracy score would have simply recorded "wrong" for Loan 3, identical in magnitude to any other wrong call; log loss instead tells you this was the single worst call in the batch, by a wide margin.

Worked example 2: three-class regime prediction

A model predicts tomorrow's market regime as one of "up," "flat," "down" using a softmax output — three probabilities that sum to 1.

Day 1: true regime is "up" (one-hot $[1,0,0]$), model predicts $[0.7, 0.2, 0.1]$. Only the probability on the true class matters: $L = -\ln(0.7) = 0.357$.

Day 2: true regime is "down" (one-hot $[0,0,1]$), model predicts $[0.6, 0.3, 0.1]$ — it put most of its mass on "up," which did not happen, and only 10% on "down," which did.

$$L = -\ln(0.1) = 2.303$$ $$\text{average over the two days} = \frac{0.357 + 2.303}{2} = 1.330$$

Notice what the loss ignores on Day 2: it does not matter that the model also put 30% on "flat" rather than "down" — only the probability mass on the class that actually happened enters the formula. This is why cross-entropy rewards concentrating probability on the right answer, not merely avoiding a specific wrong one.

The explorer above plots a logarithmic curve; drag its parameters and notice how flat it is near $p=1$ and how steeply it drops as $p \to 0$ — that steepness is the mechanism that makes log loss punish overconfidence so much harder than it punishes underconfidence.

What this means in practice

Cross-entropy is the standard training objective for classifiers precisely because it is smooth (a real gradient exists everywhere, unlike accuracy) and because minimizing it is mathematically equivalent to maximum-likelihood estimation — finding the model parameters that make the observed data least surprising. In trading applications where a model's output feeds directly into position sizing (bet bigger when more confident), log loss is also the right metric to monitor even after training, because a model that is well calibrated by log loss will size its bets sensibly, while a model that is merely "accurate" by a fixed threshold can still be dangerously overconfident on the calls that matter most.

Practice

1. A model predicts 99% probability of "up" and the market goes down. Compute the log loss for that single prediction and compare it to a model that predicted 55% and was also wrong.
2. Why does log loss have no upper bound, while accuracy is always between 0% and 100%?
3. Two models have identical accuracy, but Model X's average log loss is 0.30 and Model Y's is 0.65. What does that difference tell you about how the two models are wrong when they are wrong?