Objective Functions vs Evaluation Metrics

Nobody trains a model by directly telling it "get more accurate" or "make more money." Both of those things are what you actually care about, and both are terrible things to hand to a training algorithm, because they are jumpy, flat, or undefined almost everywhere — small changes to the model usually move them not at all, and then suddenly by a lot. So training uses a stand-in number instead: a smooth, well-behaved quantity that a gradient-based optimizer can climb down, chosen because improving it tends to improve the number you actually care about. Those are two different quantities with two different names — the objective function is what gets minimized during training, the evaluation metric is what gets reported to decide if the model is good — and confusing them is one of the most common and costly mistakes in applied machine learning.

The analogy: training miles versus race-day time

A marathon runner cannot directly practice "running 26.2 miles in under three hours" every single day; that would break their body. Instead a coach prescribes a proxy: total weekly mileage, interval-pace times, resting heart rate. Hitting those proxy targets tends to produce a fast marathon, but the relationship is not perfect — a runner can hit every mileage target and still have a bad race day, or under-train on paper and still run well because the proxies didn't capture something that mattered. The mileage log is the objective function. The race clock is the evaluation metric. They are correlated, not identical, and a coach who only ever looks at the mileage log can be badly surprised on race day.

Why you can't just optimize the metric directly

Say the metric that actually matters is classification accuracy: the fraction of predictions that end up on the correct side of a threshold. Written as a function of a model's raw output, accuracy is a step function — it is completely flat everywhere except at the exact point where a prediction crosses the decision threshold, where it jumps. A step function's slope is zero almost everywhere, which means gradient-based training gets no signal: nudging a weight up or down changes nothing about accuracy until, at one exact instant, everything flips. Gradient descent needs a slope to follow, and a flat function with occasional cliffs gives it none.

The fix is to replace the metric with a smooth surrogate — most commonly cross-entropy — that moves continuously as predictions move, and that is minimized, roughly, by the same predictions that would maximize accuracy. Cross-entropy does not just ask "was this on the right side of the line," it asks "how confidently correct or wrong was this," which gives a real slope everywhere and lets an optimizer take a graded series of small, informative steps instead of stumbling around a plateau.

Worked example 1: same accuracy, very different objective

Four labeled examples, true classes 1, 1, 0, 0. Two models both classify all four correctly at a 0.5 threshold, so both score 100% accuracy — indistinguishable by that metric.

Model 1 outputs probabilities 0.51, 0.51, 0.49, 0.49 (barely on the right side of the line). Cross-entropy is $-\frac{1}{4}\sum \ln(p_{\text{correct class}})$:

$$-\frac{1}{4}\big[\ln(0.51) + \ln(0.51) + \ln(0.51) + \ln(0.51)\big] = -\ln(0.51) = 0.673$$

Model 2 outputs 0.90, 0.90, 0.10, 0.10 (confidently correct):

$$-\frac{1}{4}\big[\ln(0.90) + \ln(0.90) + \ln(0.90) + \ln(0.90)\big] = -\ln(0.90) = 0.105$$

The metric — accuracy — says these two models are identical. The objective — cross-entropy — says Model 2 is more than six times better. That difference is not academic: Model 1 is one bad-luck data point away from every prediction flipping to the wrong side, while Model 2 has real margin. Anything downstream that uses the probability rather than just the label — position sizing scaled by confidence, for instance — would perform very differently between these two models despite their tied accuracy score.

Worked example 2: optimizing the wrong objective for the actual use case

Five assets have true forward returns of 5%, 4%, −1%, −2%, −3%. Model A is trained to minimize mean-squared error and gets close numerically: 4.5%, 4.2%, −0.8%, −1.9%, −3.1%.

$$\text{MSE}_A = \frac{(0.5)^2 + (0.2)^2 + (0.2)^2 + (0.1)^2 + (0.1)^2}{5} = \frac{0.25+0.04+0.04+0.01+0.01}{5} = 0.07 \text{ (in percentage points squared)}$$

Model B is trained to get the ranking right and does not care about matching the scale: 10%, 8%, 1%, −5%, −9%. Same order as the truth, but the numbers are wildly overstated.

$$\text{MSE}_B = \frac{(5)^2 + (4)^2 + (2)^2 + (3)^2 + (6)^2}{5} = \frac{25+16+4+9+36}{5} = 18.0$$

By mean-squared error, Model B is over 250 times worse than Model A. But suppose the actual trading rule is "go long whichever asset the model ranks first." Both models rank the same asset first — the strategy's realized outcome is identical either way. If the deployed metric is really "did the model pick the right top asset," MSE was the wrong objective to optimize for and the wrong number to have used when comparing the two models; a rank-correlation metric would have shown both models tied at perfect, which is what actually mattered here.

Try dragging the steepness parameter in the explorer above. A very steep logistic curve behaves almost like the accuracy step function — confident, near-binary — while a shallow one behaves like a smooth surrogate that gives small, graded gradients everywhere. The shape you pick for a loss function directly controls how forgiving or punishing it is toward predictions that are only slightly wrong.

What this means in practice

In quant research this mismatch shows up constantly: a model trained to minimize squared prediction error on returns is not the same thing as a model that maximizes a strategy's Sharpe ratio, because squared error punishes a big miss on a day that barely matters to P&L exactly as much as a big miss on a day that matters enormously. A model trained on cross-entropy for a binary up/down label is not the same as a model optimized for a cost-sensitive trading rule where false positives (buying and being wrong) and false negatives (missing a real opportunity) have very different dollar costs. Before training anything, write down the actual business metric first, then ask which differentiable objective is the closest smooth stand-in for it — and periodically check the two haven't drifted apart, the way Model A and Model B did above.

Practice

1. Two fraud-detection models have identical accuracy but one has much lower log-loss. Which would you prefer for a system that flags transactions by confidence score, and why?
2. A model minimizes MSE on predicted returns, but the trading desk only ever acts on the top-decile prediction each day. Propose one evaluation metric that would better reflect what the desk actually cares about.
3. Why is a step function a bad choice of training objective even though it is exactly the metric you want to improve?