Automatic Differentiation

Training a risk model or calibrating a pricing model to market quotes both boil down to the same task: adjust a handful of parameters to minimize an error, which means computing the derivative of that error with respect to every parameter. When the error is the output of a program with thousands of lines and millions of parameters — not a clean textbook formula — you can't just do the calculus by hand, and approximating derivatives by nudging each parameter slightly and re-running the whole program (finite differences) is both slow and imprecise. Automatic differentiation solves this by getting the exact derivative for free, alongside the original computation.

An analogy: a factory with a receipt for every step

Imagine a factory assembly line where every single machine — cutting, bending, welding — logs exactly how its output changes if its input changes by a tiny amount, and pins that "local sensitivity" receipt to the part as it moves down the line. To find out how the finished product changes with the very first input, you don't need to re-derive anything: you just multiply the local sensitivities along the line, machine by machine, using the chain rule. Automatic differentiation (AD) does exactly this to a computer program: it breaks the program into elementary operations (add, multiply, sin, exp, ...), each with a known, trivial derivative, and chains those local derivatives together mechanically.

The idea, one piece at a time

Any program that computes a function $f$ is really a sequence of elementary operations, $v_1, v_2, \dots, v_n$, each depending only on earlier ones — this sequence is called a computational graph. AD comes in two flavors. Forward mode starts at the input, carries a derivative alongside each intermediate value as it's computed, and applies the chain rule forward: if $v_k = g(v_i, v_j)$, its derivative with respect to the input is built from the already-known derivatives of $v_i$ and $v_j$. Reverse mode (the mode behind neural-network backpropagation) instead runs the program forward once to get the final output, then walks backward through the graph, propagating "how much does the final output change per unit change in this intermediate value" — each such quantity is called an adjoint. Reverse mode computes the derivative of one output with respect to all inputs in a single backward pass, which is why it's the standard choice when there are many parameters and one loss value, as in model calibration.

Worked example 1: forward mode by hand

Let $f(x, y) = x \cdot y + \sin(x)$, evaluated at $x = 2, y = 3$, and suppose we want $\partial f / \partial x$. Forward mode carries a value and a derivative-with-respect-to-$x$ for every intermediate quantity. Step 1: $v_1 = x = 2$, derivative $1$ (it's the input we're differentiating against). Step 2: $v_2 = y = 3$, derivative $0$ (constant with respect to $x$). Step 3: $v_3 = v_1 \cdot v_2 = 6$, derivative $= v_1' \cdot v_2 + v_1 \cdot v_2' = 1 \times 3 + 2 \times 0 = 3$ (product rule, applied locally). Step 4: $v_4 = \sin(v_1) = \sin(2) \approx 0.909$, derivative $= \cos(v_1) \cdot v_1' = \cos(2) \times 1 \approx -0.416$. Step 5: $v_5 = v_3 + v_4 \approx 6.909$, derivative $= 3 + (-0.416) = 2.584$. That final derivative, $2.584$, matches the exact calculus answer $\partial f/\partial x = y + \cos(x) = 3 + \cos(2) \approx 2.584$ — computed without ever writing down the symbolic formula for $f$.

Worked example 2: reverse mode and a calibration payoff

Now suppose $f$ is a one-parameter pricing model, $f(\theta) = \theta^2 + 4\theta$, and we're minimizing squared error $L = (f(\theta) - 5)^2$ at $\theta = 1$. Forward pass: $f(1) = 1 + 4 = 5$, wait — try $\theta=1$: $f=5$, $L = (5-5)^2=0$; to make it instructive use $\theta = 2$: $f(2) = 4 + 8 = 12$, $L = (12-5)^2 = 49$. Reverse pass starts at the output: $\partial L/\partial L = 1$. Then $\partial L/\partial f = 2(f - 5) = 2(7) = 14$. Then $\partial f/\partial \theta = 2\theta + 4 = 8$, so by the chain rule $\partial L/\partial \theta = 14 \times 8 = 112$. A calibration routine uses this single number directly to update $\theta$ in the direction that reduces $L$ — no finite-difference re-runs, no symbolic differentiation of $L$ as a whole.

What this means in practice

Every modern machine-learning framework (and increasingly, pricing libraries that need fast Greeks) is built on reverse-mode AD: it is exact to machine precision, unlike finite differences, and its cost is roughly independent of the number of inputs, unlike computing one derivative per parameter by hand or by bumping. Quants use it to get thousands of risk sensitivities from one backward pass through a pricing function, and to train models where computing gradients any other way would be computationally hopeless.