End-to-End Learned Portfolio Optimisation

The standard pipeline for turning a model into a portfolio has two separate stages: train a model to predict returns as accurately as possible (minimising, say, mean squared error), then feed those predictions into a portfolio optimiser that picks weights to maximise expected return for a given risk budget. This split makes sense organisationally — the modelling team and the portfolio-construction team can work independently — but it optimises the wrong thing at the first stage. A model trained to minimise prediction error treats every prediction error equally, when in fact some errors matter far more to portfolio performance than others: a small error on a stock the optimiser was never going to weight heavily is nearly free, while the same-sized error on a stock near a position-limit boundary can flip the optimal weight entirely.

Predict-then-optimise versus decision-focused learning

In the standard predict-then-optimise pipeline, the loss function used to train the prediction model (usually squared error between predicted and realised returns) is disconnected from the loss that actually matters — realised portfolio P&L, or risk-adjusted return, after the optimiser has acted on the predictions. A model can have excellent average prediction accuracy and still produce a mediocre portfolio, because prediction accuracy and decision quality are correlated but not identical: what matters for the portfolio is being accurate specifically where the optimiser is sensitive to the prediction, not accurate on average across every name.

End-to-end or decision-focused learning instead trains the prediction model with a loss function that runs the entire pipeline — predict, then optimise, then measure portfolio outcome — and backpropagates the portfolio-level loss all the way back through the optimisation step into the prediction model's parameters. This requires the optimisation step itself to be differentiable (or approximated by something that is), which is the technical core of the approach: a portfolio optimiser is normally a discrete "solve this problem and return the answer" black box, and making it differentiable means expressing how a small change in the predicted returns would have changed the optimal weights, then chaining that sensitivity back to the prediction model's own parameters (see Differentiable Optimisation Layers).

A worked example

A team compares two approaches to a 50-stock long-short portfolio built subject to sector-neutrality and a gross exposure cap.

Predict-then-optimise: a gradient-boosted model is trained to minimise mean squared error against next-month returns across all 50 names. It achieves a solid, roughly uniform prediction accuracy. Feeding these predictions into a mean-variance optimiser subject to the constraints produces a portfolio with a Sharpe ratio the team records as a baseline.

End-to-end: the same model architecture is instead trained with a loss defined as the negative of the realised, constrained portfolio return the predictions would have produced that month, using a differentiable relaxation of the optimiser. Over training, this model's raw prediction accuracy — measured the ordinary way, as mean squared error — actually ends up slightly worse than the predict-then-optimise model's. But its predictions are disproportionately more accurate for the small subset of stocks that regularly sit near the optimiser's constraint boundaries (the sector-neutrality and gross-exposure limits), which is exactly where prediction accuracy translates most directly into portfolio weight changes. Backtested with the same constrained optimiser, the end-to-end model's realised Sharpe ratio comes out higher than the predict-then-optimise baseline, despite its "worse" prediction accuracy by the conventional metric.

This is the central, slightly counterintuitive result of decision-focused learning: a model can be a worse general-purpose predictor and still be a better portfolio-construction input, because it was trained against the objective that actually gets paid.

Two different loss surfaces can share the same global structure but have very different sensitivity where it counts. Picture the gradient-descent path here as descending the portfolio loss surface rather than the prediction-error surface — end-to-end training walks downhill on the objective that matters, which can look like a worse fit if you're only watching the prediction-error number from the sidelines.

Where it breaks in practice

Differentiating through a real-world optimiser with discrete constraints, transaction costs, and non-convexities is often only approximate — many implementations relax the true optimiser to a smoother, convex surrogate for training purposes, then deploy the real (non-differentiable) optimiser at inference time. Any mismatch between the training-time surrogate and the deployment-time optimiser reintroduces exactly the disconnect end-to-end learning was meant to close, just one level removed.