Why Tying the Optimizer's Hands Makes It Smarter: Jagannathan and Ma

Here is a puzzle that should not have a resolution.

You want to build the lowest-risk portfolio you can from a large set of stocks. You run an optimizer. It gives you an answer that involves large short positions in some names. Now suppose you forbid short selling and re-run it. By pure logic, the constrained portfolio must be worse: you have removed options from the optimizer, and it was already choosing the best one available. The constrained answer can only be equal or inferior.

Except that in the real world, the constrained portfolio consistently turns out to have lower actual risk. Not lower predicted risk. Lower realized, out-of-sample risk.

The theory says the constraint hurts. The data says it helps. Jagannathan and Ma explained why, and the explanation is one of the more satisfying results in portfolio construction.

The problem: constraints as a guilty pleasure

Practitioners had always imposed constraints. No shorting. Cap any single name at 5 percent. Keep sector weights near the benchmark. Ask why and you would get answers about client mandates, regulation, and prudence. Nobody claimed the constraints made the portfolio better. They were understood as a tax you paid for practicality.

Meanwhile the evidence quietly piled up that constrained portfolios were performing better than unconstrained ones out of sample. This was awkward. Either the theory was wrong, or something was missing from how people were thinking about it.

The key idea via analogy: the constraint is a disguised opinion

Jagannathan and Ma's insight is a piece of mathematical detective work.

They took the problem of minimizing portfolio variance subject to no-short-sale constraints, and they asked: is there some other, unconstrained problem that has exactly the same solution? In optimization theory, constraints show up in the solution as shadow prices, or Lagrange multipliers, which measure how much each constraint is binding.

What they found is that solving the constrained minimum-variance problem with your original covariance matrix gives you exactly the same portfolio as solving the unconstrained problem with a modified covariance matrix. And the modification is not arbitrary. It goes in a very specific direction:

The no-short-sale constraint is mathematically equivalent to shrinking the largest covariance estimates downward.

Think about which stocks the optimizer wants to short. It shorts the ones that appear to have very high covariance with the rest of the portfolio, because shorting them appears to cancel out risk beautifully. Those apparently-huge covariances are precisely the ones most likely to be overestimated, because a covariance estimate that is too high is exactly the kind of error that makes a stock look like a magic hedge.

So the optimizer's desire to short a stock is a signal that it has probably been fooled by an upward error in a covariance estimate. When you forbid shorting, you block it from acting on that error. And the mathematics says that blocking it is equivalent to having reduced the offending covariance estimate in the first place.

The "wrong" constraint is secretly doing statistics. It is performing a targeted shrinkage, pulling in exactly the estimates that are most likely to be inflated, without you having to know which ones those are.

The same logic applies to upper bounds on weights. The optimizer wants to pile into stocks whose variances look too low. Capping the weight blocks it, and is equivalent to shrinking those suspiciously low variance estimates upward.

Why the argument is so satisfying

Note what has happened to the paradox. The theory was not wrong. The constraint really does make the portfolio worse if your covariance matrix is correct. But your covariance matrix is not correct, and the constraint's cost (giving up some genuinely useful shorting) turns out to be smaller than its benefit (blocking the optimizer from acting on inflated estimates).

The constraint is a free-riding regularizer. It does not know anything about statistics. It just happens to bind hardest in exactly the places where your estimates are most likely to be wrong. That is not a coincidence, it is a consequence of the fact that the optimizer chases extreme estimates and the constraints bind on extreme positions.

Empirically, Jagannathan and Ma found that constrained minimum-variance portfolios built from large universes of stocks had lower out-of-sample risk than unconstrained ones. They also compared this against building portfolios with more sophisticated risk models, including factor models and shrinkage estimators, and found that once you have imposed sensible constraints, the additional gains from the fancier covariance estimation were modest. The constraint had already done most of the work.

Why it mattered

• It gave constraints a theoretical justification, not just a practical excuse. After this paper, imposing a long-only constraint is not a concession to reality, it is a defensible statistical choice. That is a substantial change in how the profession thinks.
• It unified two separate literatures. The "improve the covariance estimate" literature (Ledoit-Wolf shrinkage, factor models) and the "constrain the portfolio" literature turn out to be doing the same thing by different means. That is a genuinely illuminating connection.
• It explains a large body of empirical results. Minimum-variance strategies, which are almost always run long-only in practice, work better than their unconstrained versions. Now we know why.
• It is a general lesson about regularization. Constraining a model is a way of encoding humility about your estimates. That idea runs straight through modern machine learning, where constraints and penalties are the primary defense against overfitting.

The honest limitations

• The shrinkage is implicit and uncontrolled. You do not get to choose how much shrinkage you get or where it lands. It is whatever the constraint happens to imply. A deliberate shrinkage estimator lets you tune the intensity; a constraint just does what it does. Sometimes that is enough, sometimes it is not.
• You are still throwing away real information. If a stock genuinely is a great hedge and genuinely should be shorted, the constraint stops you. You are trading away real alpha to protect yourself from imagined alpha, and there is no way to tell which is which in advance. The paper shows the trade is usually worth it, not that it is free.
• The result is about minimum-variance portfolios. The analysis concerns risk minimization. Once expected returns enter the problem, constraints interact with return forecasts too, and the clean equivalence does not carry over unchanged.
• It is not a licence for arbitrary constraints. The magic works because the specific constraints considered bind in the places where estimation error is worst. A constraint chosen for unrelated reasons, say a political restriction on certain sectors, has no such property and is simply a cost.
• Combining constraints and shrinkage can double-count. If you both impose a no-short constraint and shrink your covariance matrix, you may be applying the same medicine twice. The paper suggests the incremental benefit of adding more sophisticated estimation on top of constraints is limited.

The one-line takeaway

Jagannathan and Ma proved that imposing a no-short-sale constraint on a minimum-variance optimizer is mathematically identical to shrinking your most inflated covariance estimates, which is why a constraint that theory says can only hurt reliably produces lower-risk portfolios in practice: the constraint is doing statistics you did not know you needed.