Forecasting Direction vs Magnitude
Predicting whether a return will be positive and predicting how large it will be call for different models, different loss functions, and different ways of judging success — and a strategy's construction should determine which one is actually needed.
Prerequisites: Choosing What to Forecast
"Will it go up or down" and "by how much" sound like two versions of the same question, but they push a researcher toward different models, different evaluation metrics, and sometimes different conclusions about whether an idea is any good at all.
Two different problems
Forecasting direction treats the target as a label — up or down, or up/flat/down — and is naturally handled by a classification model, evaluated with something like accuracy, precision/recall, or area under an ROC curve. It answers "which side should I be on" and deliberately throws away how big the move turns out to be.
Forecasting magnitude treats the target as a continuous number and is handled by a regression model, evaluated with something like mean squared error or an information coefficient against the realized return. It answers "how much conviction should this position carry," but a model that's excellent on average magnitude can still get the sign wrong on individual observations more often than a direction-only model would.
The two aren't always aligned. A model can have good directional accuracy — right about 55% of the time — while its magnitude predictions are nearly worthless, because getting the sign right slightly more than half the time doesn't require accurately sizing the move. Conversely, a magnitude model minimizing squared error can end up systematically underweighting rare, large moves in favour of nailing the common, small ones, which can hurt directional accuracy on exactly the observations that matter most for PnL.
Choose direction when the downstream strategy only sizes by a fixed unit per bet (equal-weighted long-short deciles); choose magnitude when the strategy sizes positions in proportion to forecast strength. Building a magnitude model and only ever using its sign, or building a direction model and pretending its confidence score is a magnitude, both throw away information the strategy needs.
A worked example
A researcher tests two models on the same universe. Model A, a classifier, achieves 54% directional accuracy — modestly, but genuinely, better than a coin flip. Model B, a regression, achieves an of 0.002 against realized return magnitude — tiny, and on its own looks unimpressive. Converted into an information coefficient, however, Model B's predictions correlate with realized returns at 0.03, in line with what a usable equity signal typically looks like — the low is expected, because return variance is dominated by noise the model was never going to explain. Judging Model B by alone would have discarded a usable signal; judging it by IC, the standard metric for this kind of target, reveals it's comparable in strength to Model A.
Never compare a classification metric (accuracy, AUC) directly against a regression metric (, IC) to decide which model is "better" — they aren't on the same scale, and a low-looking regression score can still represent real, tradeable skill.
Related concepts
Practice in interviews
Further reading
- Lopez de Prado, Advances in Financial Machine Learning (ch. on labeling)