Backpropagation

A trained network is nothing more than a big pile of numbers — the weights. Training means repeatedly asking one question about every single one of them: if I nudged this weight up a hair, would the error get better or worse, and by how much? Answer that for all the weights and you know which way to step.

The obvious way to answer it is to try. Change one weight by a tiny amount, run the whole network again, see how the error moved. That works perfectly and it is completely useless in practice, because a modest network has a hundred thousand weights, so one training step would cost a hundred thousand full runs. Backpropagation gets every one of those numbers in about the cost of one extra run. That efficiency is the only reason neural networks are trainable at all.

The analogy: attributing a P&L miss backwards

A fund is down 40 basis points against target and you want to know how much each individual trade contributed. Nobody re-runs the entire month once per trade. You work backwards down the reporting chain instead.

Start at the top: the fund missed by 40bp. The Europe sleeve is 30% of the book, so 30% of any move in Europe shows up at the fund level. Within Europe, the mean-reversion strategy is 20% of the sleeve. So a trade inside that strategy influences the fund's number by $0.30 \times 0.20 = 0.06$ — six cents on the dollar. Nobody at any level needed to understand the whole firm. Each level only knew its own share, and you got the answer by multiplying the shares along the path as you walked down.

Backpropagation is that, applied to a network. Each layer knows only how sensitive its own output is to its own input. Multiply those local sensitivities from the output end backwards and you get every weight's influence on the final loss.

The one rule

The mathematical name for "multiply the shares along the path" is the chain rule. If a loss $L$ depends on a value $a$, and $a$ depends on $z$, then

$$\frac{\partial L}{\partial z} = \frac{\partial L}{\partial a} \cdot \frac{\partial a}{\partial z}$$

In words: the influence of $z$ on the loss equals the influence of $a$ on the loss, times how strongly $z$ moves $a$. The symbol $\partial L / \partial z$ just means "how much $L$ changes per unit change in $z$, holding everything else fixed" — a slope, nothing more.

Backprop turns this into a mechanical procedure. Every node in the network receives a gradient from its right-hand neighbour, multiplies it by its own local slope, and hands the result to its left-hand neighbour. That is the entire algorithm.

Worked example 1: one neuron, done by hand

Input $x = 2$, weight $w = 0.5$, bias $b = 0.1$, sigmoid activation, target $y = 1$, squared-error loss.

Forward pass. Compute left to right and write down every intermediate value, because the backward pass will need them.

$$z = 0.5(2) + 0.1 = 1.10, \qquad a = \frac{1}{1 + e^{-1.10}} = 0.750, \qquad L = \tfrac{1}{2}(0.750 - 1)^2 = 0.0312$$

Backward pass. Start at the loss and walk left, multiplying local slopes.

1. Loss to output. For $L = \frac{1}{2}(a-y)^2$ the slope is $a - y = 0.750 - 1 = -0.250$. Negative, meaning: increasing the output would reduce the loss. Sensible, since the output is below target.
2. Through the sigmoid. The sigmoid's slope is $a(1-a) = 0.750 \times 0.250 = 0.1874$. Multiply: $\partial L/\partial z = -0.250 \times 0.1874 = -0.0468$.
3. Through the weight. Since $z = wx + b$, the slope of $z$ with respect to $w$ is just $x = 2$. So

$$\frac{\partial L}{\partial w} = -0.0468 \times 2 = -0.0936, \qquad \frac{\partial L}{\partial b} = -0.0468$$

Both gradients are negative, so both $w$ and $b$ should go up to reduce the loss.

You can check this without any calculus. Set $w = 0.501$ and redo the forward pass: $z = 1.102$, $a = 0.750634$, $L = 0.0310917$, against the original $L = 0.0311850$. The loss fell by $0.0000933$ for a change of $0.001$, giving a slope of $-0.0933$ — matching the $-0.0936$ backprop produced, to the accuracy of the finite difference. This is exactly the gradient check every implementation should run once.

Worked example 2: two layers, and the sign flip

Now a network with a hidden ReLU unit. Input $x = 1.0$; hidden weight $w_1 = 0.8$, bias $0$; output weight $w_2 = -1.5$, bias $b_2 = 0.4$; linear output; target $y = 0.5$.

Forward. $z_1 = 0.8$, so $h = \max(0, 0.8) = 0.8$. Then $z_2 = -1.5(0.8) + 0.4 = -0.80$, which is the prediction. The loss is $\frac{1}{2}(-0.80 - 0.5)^2 = 0.845$. Badly wrong.

Backward. Write $\delta_2$ for the gradient at the output: $\delta_2 = \hat{y} - y = -1.30$.

$$\frac{\partial L}{\partial w_2} = \delta_2 \cdot h = -1.30 \times 0.8 = -1.04, \qquad \frac{\partial L}{\partial b_2} = -1.30$$

In words: a weight's gradient is the gradient arriving at its unit, times the value that weight was multiplying. Now push past the output weight into the hidden unit. The gradient gets multiplied by $w_2$ (the path it travelled through) and by the ReLU's slope, which is $1$ here because $z_1$ was positive:

$$\delta_1 = \delta_2 \cdot w_2 \cdot \text{ReLU}'(z_1) = -1.30 \times (-1.5) \times 1 = +1.95$$

The sign flipped. Because $w_2$ is negative, this unit helps by getting smaller, and backprop discovers that automatically. Then $\partial L / \partial w_1 = \delta_1 \cdot x = 1.95$.

Take one step with learning rate $0.1$: $w_1 \to 0.605$, $w_2 \to -1.396$, $b_2 \to 0.53$. Redo the forward pass and the prediction moves to $-0.315$, dropping the loss from $0.845$ to $0.332$. The gradients were right.

What this means in practice

Three consequences follow from "gradients are products along a path".

You must cache the forward pass. Every local slope needs the values computed on the way through, which is why training uses far more memory than inference and why long sequences or big batches run out of GPU memory rather than time.

Depth multiplies, so depth is fragile. Twenty layers means twenty factors multiplied together. If each is around $0.5$ the gradient reaching the first layer is $0.5^{20} \approx 10^{-6}$ and that layer never learns; if each is around $1.5$ it blows up. This is the whole story behind Vanishing and Exploding Gradients, and the reason careful initialisation, normalisation layers and residual connections exist.

Dead paths pass nothing. ReLU has slope zero for negative inputs, so a unit sitting on the wrong side sends zero gradient backwards and everything behind it goes unchanged that step.