Predict-Then-Optimise vs Decision-Focused Learning

Almost every ML pipeline in finance follows the same two-step shape: predict something (expected returns, default probability, volatility), then feed that prediction into a separate optimisation or decision rule (portfolio weights, credit limits, hedge sizing). The natural way to train the prediction step is to make it as accurate as possible on its own terms — minimize squared error, maximize likelihood. Decision-focused learning asks a pointed question: is "most accurate prediction" actually the same thing as "the prediction that leads to the best decision"? Often, it isn't.

An analogy: grading a map by distance versus by whether it gets you there

Two maps of a city could have identical average error in where they place streets, yet one systematically gets the route to the hospital slightly wrong while the other gets every unimportant back alley slightly wrong instead. Judged purely on average positional accuracy, the maps tie. Judged on "does following this map get you to the hospital fastest," they're not close. Predict-then-optimise grades the map on average accuracy; decision-focused learning grades it on whether it gets you where the decision actually needs to go.

Why the two can disagree

A predictor trained with a standard loss (say, mean squared error) treats every prediction error the same regardless of what happens to it downstream. But an optimiser doesn't treat all inputs the same — some errors get amplified into large decision changes (an asset the optimiser wants to lever up heavily), while other errors of identical size barely move the final decision at all (an asset already near a constraint boundary). Decision-focused learning trains the predictor using a loss that measures decision regret directly:

$$\text{regret} = f(w^*(\hat y)) - f(w^*(y))$$

In plain English: take the decision $w^*(\hat y)$ the optimiser would make using predicted values, evaluate it using true values $y$, and compare that to the best possible decision $w^*(y)$ made with perfect foresight — the gap between the two is what the predictor gets trained to minimize, rather than prediction error for its own sake.

Worked example: a resource-allocation toy case

A model predicts default probability for 100 loan applicants, feeding a downstream decision rule that approves the 60 with the lowest predicted risk, subject to a budget constraint. A predict-then-optimise model trained on log-loss achieves excellent overall calibration across all 100 applicants — but its errors happen to cluster right around the approval cutoff, the exact region where a small prediction error flips an approve/deny decision. A decision-focused model, trained instead to minimize regret in the actual approval decision, sacrifices some calibration accuracy on applicants far from the cutoff (where it barely matters) in exchange for being much more accurate exactly at the boundary that determines who gets approved. Backtesting shows the decision-focused model produces a lower realized default rate among approved loans at the same budget, despite scoring worse on a standard log-loss metric — a direct illustration of prediction accuracy and decision quality diverging.

What this means in practice

Decision-focused learning is worth the extra engineering complexity when the downstream decision has hard constraints or thresholds, and when prediction accuracy far from those thresholds genuinely doesn't matter for outcomes — portfolio construction, credit approval, and resource allocation are classic cases. When the downstream use of a prediction is diffuse or unclear (the prediction feeds many different, unspecified future decisions), plain predict-then-optimise with a standard loss is usually the more robust, simpler choice, since decision-focused training ties the model tightly to one specific downstream decision rule.