Loss Functions for Classification

Training a classifier means adjusting its parameters to make some number smaller, and that number — the loss function — quietly decides what "good" means. A model doesn't know it's being graded on accuracy or on calibrated probabilities unless the loss says so; two classifiers trained on the same data with different losses can behave very differently, especially on the examples they're least sure about.

The analogy: grading a multiple-choice test on confidence, not just correctness

A teacher could grade a test purely on right or wrong answers. Or the teacher could ask students to state a confidence level with each answer and grade them more harshly for being confidently wrong than for being wrong while admitting uncertainty. Most classification losses are the second kind of grading — they don't just check if the predicted class matches the true one, they penalize how confidently the model got it wrong.

Worked example: cross-entropy loss on three predictions

Cross-entropy loss for a true label $y \in \{0, 1\}$ and predicted probability $\hat p$ that $y=1$ is:

$$L = -\big[y \log(\hat p) + (1-y)\log(1-\hat p)\big]$$

Three cases, all with true label $y = 1$:

• Confident and correct: $\hat p = 0.95 \Rightarrow L = -\log(0.95) = 0.051$
• Unsure: $\hat p = 0.55 \Rightarrow L = -\log(0.55) = 0.598$
• Confident and wrong: $\hat p = 0.05 \Rightarrow L = -\log(0.05) = 2.996$

Being unsure costs about 12 times more loss than being confidently right, but being confidently wrong costs nearly 60 times more than being confidently right — the loss grows without bound as $\hat p \to 0$ for a true positive, which is exactly the "confidently wrong is punished hardest" behavior that pushes a model toward well-calibrated, cautious probabilities rather than reckless overconfidence.

The logistic curve above is the same S-shape a classifier squeezes its raw score through to produce $\hat p$ — drag the parameters and notice how a steeper curve turns small score differences into extreme probabilities near 0 or 1, which is precisely where cross-entropy's penalty for being wrong becomes severe.

What this means in practice

Cross-entropy (log-loss) is the default for most classifiers because it's differentiable everywhere and directly rewards calibrated probabilities, which matters whenever a downstream decision (a trading signal's position size, a fraud threshold) depends on the actual probability, not just the predicted label. Alternatives like hinge loss (used by SVMs) instead only care about being on the correct side of a margin, and stop caring once a prediction is confidently correct enough — useful when calibrated probabilities aren't needed and a hard margin between classes is.