Loss Functions for Classification

Training a classifier means adjusting its parameters to make some number smaller, and that number, the loss function, quietly decides what "good" means. A model doesn't know it's being graded on accuracy or on calibrated probabilities unless the loss says so; two classifiers trained on the same data with different losses can behave very differently, especially on the examples they're least sure about.

The analogy: grading a multiple-choice test on confidence, not just correctness

A teacher could grade a test purely on right or wrong answers. Or the teacher could ask students to state a confidence level with each answer and grade them more harshly for being confidently wrong than for being wrong while admitting uncertainty. Most classification losses are the second kind of grading, they don't just check if the predicted class matches the true one, they penalize how confidently the model got it wrong.

Worked example: cross-entropy loss on three predictions

Cross-entropy loss for a true label $y \in \{0, 1\}$ and predicted probability $\hat p$ that $y=1$ is:

Three cases, all with true label $y = 1$:

• Confident and correct: $\hat p = 0.95 \Rightarrow L = -\log(0.95) = 0.051$
• Unsure: $\hat p = 0.55 \Rightarrow L = -\log(0.55) = 0.598$
• Confident and wrong: $\hat p = 0.05 \Rightarrow L = -\log(0.05) = 2.996$

Being unsure costs about 12 times more loss than being confidently right, but being confidently wrong costs nearly 60 times more than being confidently right, the loss grows without bound as $\hat p \to 0$ for a true positive, which is exactly the "confidently wrong is punished hardest" behavior that pushes a model toward well-calibrated, cautious probabilities rather than reckless overconfidence.

The logistic curve above is the same S-shape a classifier squeezes its raw score through to produce $\hat p$, drag the parameters and notice how a steeper curve turns small score differences into extreme probabilities near 0 or 1, which is precisely where cross-entropy's penalty for being wrong becomes severe.

What this means in practice

Cross-entropy (log-loss) is the default for most classifiers because it's differentiable everywhere and directly rewards calibrated probabilities, which matters whenever a downstream decision (a trading signal's position size, a fraud threshold) depends on the actual probability, not just the predicted label. Alternatives like hinge loss (used by SVMs) instead only care about being on the correct side of a margin, and stop caring once a prediction is confidently correct enough, useful when calibrated probabilities aren't needed and a hard margin between classes is.