Multi-Task Learning and Loss Weighting

Suppose one network shares its early layers across two prediction heads: one forecasting next-day return, another forecasting next-day volatility. Training it means combining the return loss and the volatility loss into a single number to backpropagate — but return might be measured in loss units of order 0.0001 and volatility in loss units of order 10, purely because of how each target happens to be scaled. Add them with equal weight and the optimizer effectively ignores the return task entirely, chasing whichever loss happens to have the larger numbers, regardless of which task actually matters more.

The analogy: splitting a manager's attention across two reports

A manager grading two employees on a combined score, one measured in dollars of revenue and one measured in percentage points of customer satisfaction, can't just add the raw numbers — a $50,000 swing dwarfs a 2-point swing in scale, even if the 2-point swing is the more important signal. The manager has to consciously decide how many "points" a percentage-point swing is worth relative to a dollar, or the combined score is meaningless. Combining loss functions of different scale and different intrinsic difficulty requires exactly the same deliberate weighting decision.

The formula

For $T$ tasks with individual losses $L_1, \dots, L_T$, the simplest combination is a weighted sum:

$$L_{\text{total}} = \sum_{t=1}^{T} w_t L_t$$

In words: multiply each task's loss by a weight $w_t$ before adding them together, so the gradient contribution of each task can be controlled independently of its raw numeric scale. A more principled approach, from Kendall, Gal & Cipolla, treats each $w_t$ as a learned parameter tied to that task's uncertainty $\sigma_t$ — a task the model is inherently less certain about (noisier target) automatically gets down-weighted:

$$L_{\text{total}} = \sum_{t=1}^{T} \frac{1}{2\sigma_t^2} L_t + \log \sigma_t$$

In words: divide each task's loss by twice its learned variance — a noisier task gets a smaller effective weight — and add a penalty term $\log \sigma_t$ that stops the model from trivially setting $\sigma_t$ to infinity to make every loss disappear.

Worked example 1: fixed weights fixing a scale mismatch

Return loss $L_1 = 0.0002$ (mean squared error on returns of order 1%), volatility loss $L_2 = 8.5$ (mean squared error on volatility measured in variance-like units). Unweighted sum: $0.0002 + 8.5 = 8.5002$ — the gradient from $L_1$ contributes essentially nothing; $\partial L_{\text{total}}/\partial \theta$ is dominated almost entirely by the volatility head. Setting $w_1 = 10000, w_2 = 1$ instead gives $10000 \times 0.0002 + 1 \times 8.5 = 2.0 + 8.5 = 10.5$ — now both tasks contribute comparably-sized gradients, and the shared layers actually receive useful signal from both.

Worked example 2: learned uncertainty weighting adjusting automatically over training

Early in training, suppose $\sigma_1 = 0.02$ for return and $\sigma_2 = 3.0$ for volatility. Weight on $L_1$: $1/(2 \times 0.02^2) = 1250$. Weight on $L_2$: $1/(2 \times 3^2) = 0.056$. Later, once the model has gotten much more confident about volatility ($\sigma_2$ shrinks to $0.5$) while return stays hard ($\sigma_1$ unchanged): weight on $L_2$ becomes $1/(2\times0.25) = 2.0$, a roughly 36-fold increase given to the now-easier, more-precisely-known task — the weighting shifts automatically as training progresses, without hand-retuning $w_t$.

What this means in practice

Multi-task setups are common wherever one shared feature extractor feeds several forecasting heads (return, volatility, regime classification from the same market data), because sharing representations across related tasks can act as a useful regularizer and is cheaper than training separate models. The learned-uncertainty scheme above is a reasonable default when tasks' relative importance isn't known in advance; when it is known (say, return prediction is the only thing that will ever be traded on, volatility is just an auxiliary signal to help training), fixed weights that explicitly encode that priority are simpler and more predictable.