Computational Graphs and Automatic Differentiation

Every deep learning framework claims to compute the exact gradient of a loss with respect to millions of parameters, instantly, for any model you can write in code — not an approximation, not a numerical estimate, the exact derivative. That sounds like it requires either a superhuman calculus student or a shortcut. It is a shortcut, but a completely mechanical one: break any calculation, no matter how complicated, into a sequence of tiny operations simple enough that their individual derivatives are trivial — addition, multiplication, sine, exponent — and then apply the chain rule along that sequence automatically. No symbolic algebra is ever needed, and no human ever has to differentiate the whole function by hand.

The analogy: a spreadsheet of linked formulas

Think of a large spreadsheet where some cells hold raw numbers you typed in, and every other cell holds a formula referencing earlier cells — C3 = A1 * B2, D4 = SIN(C3), and so on, all the way down to one final cell holding the answer. You never wrote down the whole formula for that final cell as one giant expression; you built it as a chain of small, individually simple steps. If you wanted to know how the final answer changes when you nudge cell A1, you would not need to expand the entire chain into one messy equation and differentiate it by hand — you could just walk the chain of cell-to-cell dependencies, and at each link ask "how much does this cell move per unit move in the one it depends on," multiplying those local sensitivities together as you go.

That dependency graph of cells is a computational graph, and walking it to compute sensitivities is exactly what automatic differentiation does — mechanically, for graphs with millions of cells, every one of which has a known, simple local derivative.

The graph, and two ways to walk it

A computational graph is a directed graph where each node is one elementary operation, and edges show which nodes feed into which. Every node knows two things: how to compute its own value from its inputs (the forward pass), and its own local derivative with respect to each of its inputs (a simple, closed-form rule, since the node performs one primitive operation like +, ×, or sin).

There are two ways to combine these local derivatives into the derivative of the whole graph's output with respect to any input, and they differ in which direction they sweep:

Forward-mode starts at one input, seeds it with a derivative of $1$ (and every other input with $0$), and pushes that sensitivity forward through the graph alongside the values themselves, accumulating via the chain rule as it goes. One forward sweep gives the derivative of every output with respect to that one input.

Reverse-mode — what Backpropagation is, applied to any computational graph, not just neural networks — runs the forward pass once to get all the values, then walks backward from the final output with a seed derivative of $1$, and at every node multiplies the incoming "adjoint" (gradient with respect to that node's output) by the node's local derivative to get the adjoint for its inputs. One backward sweep gives the derivative of one output with respect to every input.

That asymmetry is the whole reason deep learning uses reverse-mode almost exclusively: a training loss is a single number (one output) depending on millions of weights (many inputs), and reverse-mode gets every one of those millions of derivatives in one sweep, while forward-mode would need one sweep per weight.

Worked example 1: reverse-mode by hand on a tiny graph

Let $f(x, y) = xy + \sin(x)$, evaluated at $x = 2$, $y = 3$. Break it into elementary steps:

$$v_1 = xy, \qquad v_2 = \sin(x), \qquad v_3 = v_1 + v_2 = f$$

Forward pass: $v_1 = 2 \times 3 = 6$, $v_2 = \sin(2) = 0.909$, $v_3 = 6 + 0.909 = 6.909$.

Backward pass, seeding the output's own adjoint at $1$ (the derivative of anything with respect to itself), and walking backward node by node:

• $v_3 = v_1 + v_2$: the local derivative of a sum with respect to either input is $1$, so the adjoint of $v_1$ is $1 \times 1 = 1$ and the adjoint of $v_2$ is $1 \times 1 = 1$.
• $v_1 = xy$: local derivatives are $\partial v_1/\partial x = y = 3$ and $\partial v_1/\partial y = x = 2$. Multiply by $v_1$'s adjoint: contributes $1 \times 3 = 3$ to $x$'s running total, and $1 \times 2 = 2$ to $y$'s.
• $v_2 = \sin(x)$: local derivative is $\cos(x) = \cos(2) = -0.416$. Multiply by $v_2$'s adjoint: contributes $1 \times (-0.416) = -0.416$ to $x$'s running total.

$$\frac{\partial f}{\partial x} = 3 + (-0.416) = 2.584, \qquad \frac{\partial f}{\partial y} = 2$$

Check against ordinary calculus: $\partial f/\partial x = y + \cos(x) = 3 + \cos(2) = 3 - 0.416 = 2.584$, and $\partial f/\partial y = x = 2$. Both match exactly — the graph walk produced the true derivative, using only elementary local rules, never symbolically expanding $f$ as a whole.

Worked example 2: why reverse-mode wins when inputs outnumber outputs

Forward-mode can compute the same derivatives, but it needs one full sweep per input. Seed $dx = 1, dy = 0$ (asking "how does everything move if only $x$ moves") and propagate a derivative alongside every value:

$$dv_1 = y\,dx + x\,dy = 3(1) + 2(0) = 3, \quad dv_2 = \cos(x)\,dx = \cos(2)(1) = -0.416, \quad dv_3 = dv_1 + dv_2 = 2.584$$

That one sweep recovers $\partial f/\partial x = 2.584$. A second sweep, seeded $dx=0, dy=1$, is required to get $\partial f/\partial y$:

$$dv_1 = 3(0) + 2(1) = 2, \quad dv_2 = \cos(2)(0) = 0, \quad dv_3 = 2 + 0 = 2$$

Two inputs required two separate forward sweeps. Reverse-mode got both derivatives from a single backward sweep in worked example 1. Scale this up: a neural network with a million weights and one scalar loss would need one million forward-mode sweeps to get every weight's gradient, or exactly one reverse-mode sweep. That is not a minor efficiency difference — it is the difference between "trainable" and "impossible," and it is the entire reason backpropagation (reverse-mode automatic differentiation, applied to a graph whose final node is a scalar loss) is how every modern network is trained.

The $\cos(x)$ term above is one of the "elementary operations with a known local derivative" that autodiff leans on — every node in the graph is something this simple, chained together. Drag $b$ below and watch the tangent line update: that live slope is exactly the local derivative autodiff looks up at each node, before the chain rule stitches all of them together.

What this means in practice

Every operation used inside a model — matrix multiplies, activation functions, even control flow like loops over a variable-length sequence — must have a known local derivative for automatic differentiation to work through it; frameworks like PyTorch and TensorFlow build the computational graph as the forward pass executes (or ahead of time), record every elementary operation, and then run reverse-mode automatically when .backward() or its equivalent is called. This is also precisely why a custom operation that is not differentiable — like a hard threshold, or code that reads a value out to plain Python and back — silently breaks gradient flow: the graph has a node with no known local derivative, or a gap where no node was recorded at all, and the chain rule cannot walk across it.

Practice

1. For $f(x,y) = x^2 y$ at $x=3, y=2$, break it into elementary steps, run a forward pass, then a reverse-mode backward pass to find $\partial f/\partial x$ and $\partial f/\partial y$, and check both against direct calculus.
2. A model has 3 inputs and 1 output. How many forward-mode sweeps would be needed to get the full gradient, and how many reverse-mode sweeps?
3. Explain in one sentence why a graph with 1 input and 1,000 outputs would actually prefer forward-mode over reverse-mode.