Negative Log-Likelihood as a Loss Function

A weather forecaster who says "90% chance of rain" and it rains should be scored better than one who says "55%" and it rains — both got the direction right, but the first was more confidently right. And a forecaster who says "95% chance of sun" when it pours should score much worse than one who hedged at "55% sun" — confidently wrong deserves a harsher penalty than cautiously wrong. Negative log-likelihood is exactly this scoring rule, made precise: it rewards confident correctness, tolerates cautious mistakes, and punishes confident mistakes severely.

Building the formula from the idea

A model that outputs probabilities assigns some probability $p$ to the outcome that actually happened. The likelihood of the data, given the model, is just that probability — how likely the model said the true outcome was, before it was known. Training wants to make the true outcomes look as probable as possible under the model, so naturally you'd want to maximize likelihood. Optimization algorithms are built to minimize, so the negative sign flips the direction: minimizing $-\log(\text{likelihood})$ is the same problem as maximizing likelihood.

The logarithm is there for a specific, important reason. With $n$ independent data points, the joint likelihood is a product of $n$ probabilities, $\prod_i p_i$ — a number that shrinks toward zero fast as $n$ grows and becomes numerically unworkable. Taking the log turns that product into a sum, $\sum_i \log p_i$, which is numerically stable and lets each data point contribute independently and additively to the total.

$$\text{NLL} = -\sum_{i=1}^n \log P(y_i \mid x_i)$$

In words: for every training example, look up the probability the model assigned to the actual observed outcome $y_i$, take its logarithm, and sum those logarithms across all examples, then flip the sign. A model that assigns high probability to what actually happened, every time, gets a small NLL. A model that confidently assigns low probability to what actually happened gets a large NLL, because $\log$ of a small number is a large negative number, and the outer negative sign turns that into a large positive penalty.

This is the shape doing all the work: $-\log(p)$ stays small and gentle as $p$ approaches $1$ (barely any penalty for being confidently right), but rises steeply — without bound — as $p$ approaches $0$ (an enormous, unbounded penalty for confidently assigning near-zero probability to something that then happens). No other simple penalty shape has this specific asymmetry built in.

Worked example 1: three forecasts scored

A binary classifier predicts, for three days that all actually ended "up": forecast probabilities of "up" equal to $0.9$, $0.6$, and $0.05$ (that last one confidently wrong). Compute $-\log p$ for each: $-\log(0.9) \approx 0.105$, $-\log(0.6) \approx 0.511$, $-\log(0.05) \approx 2.996$. Average NLL across the three: $(0.105 + 0.511 + 2.996)/3 \approx 1.204$.

Notice how lopsided the contributions are: the confidently correct forecast ($0.9$) contributes almost nothing to the total, the middling forecast ($0.6$) contributes a moderate amount, and the single confidently wrong forecast ($0.05$) contributes nearly three times as much as the other two combined. One bad, overconfident call can dominate the entire loss — which is deliberate: NLL is specifically designed to punish miscalibrated confidence, not just wrong direction.

Worked example 2: comparing two models with the same accuracy

Model A and Model B both correctly classify 8 of 10 days (80% accuracy). Model A makes correct predictions with modest confidence ($0.65$), and wrong ones modestly too ($0.45$ to the right class). Model B makes correct predictions very confidently ($0.95$), but its wrong predictions are also very confident and very wrong ($0.02$ to the correct class).

Model A's NLL: 8 terms at $-\log(0.65)\approx0.431$, 2 at $-\log(0.45)\approx0.799$: average $\approx0.505$. Model B's NLL: 8 terms at $-\log(0.95)\approx0.051$, 2 at $-\log(0.02)\approx3.912$: average $\approx0.823$.

Both models are 80% accurate, but Model B's NLL is substantially worse, because its confident mistakes are punished heavily while its confident correct calls barely help. Accuracy alone would call these models equal; NLL correctly flags Model B as more dangerous to trade on, since its wrong calls carry false conviction.

What this means in practice

NLL (often called log loss or, for classification, cross-entropy — the same object under different names) is the loss function actually being minimized inside logistic regression, most neural network classifiers, and any model that outputs calibrated probabilities rather than just a hard label. Choosing to optimize NLL instead of plain accuracy is what forces a model to output well-calibrated probabilities in the first place — a model trained purely to maximize accuracy has no incentive to get its confidence level right, only its label, whereas NLL directly penalizes miscalibration.