Reverse-Mode vs Forward-Mode Autodiff

Automatic differentiation can compute the exact derivative of any function built from elementary operations, by walking a computational graph and applying the chain rule mechanically. But there are two directions to walk that graph, and they are not interchangeable in cost — one of them is what makes training a network with millions of weights and a single loss number tractable at all, and the other one would make it impossible.

The analogy: tracing water through pipes, forward or backward

Picture a network of pipes where water flows from several source taps, through junctions that mix and split flows, down to several drains. If you want to know "how much does drain 3's flow change if I open tap 1 a little more," you can either trace forward from tap 1, following every downstream path it touches until you reach every drain (one trace answers "this tap's effect on every drain"), or trace backward from drain 3, following every upstream path back to every tap that feeds it (one trace answers "every tap's effect on this drain"). If there is one tap and many drains, forward tracing is efficient — one sweep gets everything. If there are many taps and one drain, backward tracing wins for the identical reason, reversed.

The two sweeps, precisely

Forward mode seeds one input's derivative to $1$ and every other input's to $0$, then propagates that "tangent" forward through the graph alongside the ordinary values, so every node carries both its value and its derivative with respect to that one seeded input. One forward sweep yields $\partial(\text{every output})/\partial(\text{that one input})$.

Reverse mode runs the forward pass once to get all values, then walks backward from a single output seeded with derivative $1$, and at each node multiplies the incoming "adjoint" by that node's local derivative to push the sensitivity to its inputs. One backward sweep yields $\partial(\text{that one output})/\partial(\text{every input})$.

$$\text{cost(forward)} \propto (\text{number of inputs}) \times (\text{cost of one sweep})$$ $$\text{cost(reverse)} \propto (\text{number of outputs}) \times (\text{cost of one sweep})$$

In plain English: forward mode's total cost scales with how many inputs you need derivatives for; reverse mode's scales with how many outputs you have. A neural network training loss has millions of inputs (the weights) and exactly one output (the scalar loss) — so reverse mode needs one sweep total, forward mode would need millions.

Worked example 1: one input, three outputs — forward wins

Let $y_1 = x^2$, $y_2 = \sin(x)$, $y_3 = e^x$, all functions of a single input $x=1$. Forward mode: seed $dx=1$, propagate once — $dy_1 = 2x\,dx = 2$, $dy_2=\cos(x)\,dx=0.540$, $dy_3=e^x\,dx=2.718$ — all three derivatives from one sweep. Reverse mode would need three separate backward passes, one seeded from each output, to get the same three numbers. With one input and three outputs, forward mode is three times cheaper.

Worked example 2: three inputs, one output — reverse wins

Let $L = x_1 x_2 + x_2 x_3$ at $x_1=2, x_2=3, x_3=4$. Reverse mode: one forward pass computes $L = 6+12=18$; one backward pass, seeding $L$'s adjoint at $1$, gives $\partial L/\partial x_1 = x_2 = 3$, $\partial L/\partial x_2 = x_1+x_3=6$, $\partial L/\partial x_3=x_2=3$ — all three gradients from a single backward sweep. Forward mode would need three separate forward sweeps, one seeded per input, to get the same three numbers. With three inputs and one output, reverse mode is three times cheaper — and for a network with a million weights, this ratio is a million to one.

Every step shown in the explorer above is only possible because reverse-mode autodiff already collapsed a million-weight gradient into a single cheap backward sweep — the step itself is a separate question from how that gradient got computed.

What this means in practice

Backpropagation is reverse-mode automatic differentiation applied to a graph whose final node is a scalar loss — nothing more exotic than that. Forward mode still has real uses: computing directional derivatives, Jacobian-vector products when the number of outputs genuinely exceeds the number of inputs, or certain second-order methods that combine both directions to get Hessian-vector products cheaply.