Numerical Stability and Error Analysis

Two mathematically equivalent formulas for the same quantity — algebraically identical on paper — can give a computer answers that differ by many orders of magnitude. This isn't a bug in the sense of a typo; it's a structural fact about doing arithmetic with numbers that are only ever stored to about 15-16 significant digits. Every operation on a computer introduces a tiny rounding error, and numerical stability is the question of what happens to that error as it flows through the rest of the calculation: does it stay small and harmless, or does it get amplified into something that swamps the true answer?

An analogy: a whispered message down a line

Picture a game of telephone where each person, on top of relaying the message, also has a small independent chance of mishearing a word. Most versions of the game just accumulate a bit of noise — the message at the end is roughly right, slightly garbled. But some versions of the "rules" for relaying can amplify a mishearing — for instance, if each person is instructed to repeat the difference between what they heard and what they expected, a tiny mishearing early on can get blown up into nonsense by the time it reaches the last person. An unstable numerical algorithm is the second kind of relay: it doesn't just pass rounding error along, it magnifies it.

The math, one piece at a time

Every floating-point number is stored with a relative error bounded by machine epsilon, $\varepsilon \approx 2.2 \times 10^{-16}$ for standard double precision — meaning a stored value $\hat{x}$ satisfies $\hat{x} = x(1+\delta)$ for some tiny $|\delta| \le \varepsilon$. In words: each stored number is correct to about 16 significant digits, off by a relative error no bigger than roughly $10^{-16}$. The dangerous operation is subtracting two nearly-equal numbers, called catastrophic cancellation: if $a$ and $b$ are both accurate to relative error $\varepsilon$ but $a \approx b$, then $a - b$ is a small number computed from the difference of two large numbers' rounding errors, so its relative error can be enormous even though $a$ and $b$ individually were fine.

Formally, if $a$ and $b$ have absolute errors of size roughly $\varepsilon \cdot |a|$, the absolute error in $a - b$ stays roughly the same size — but the relative error in the result is that absolute error divided by $|a-b|$, which explodes if $|a-b|$ is much smaller than $|a|$ and $|b|$ themselves:

$$\text{relative error of } (a-b) \; \approx \; \frac{\varepsilon \cdot |a|}{|a - b|} .$$

In plain English: subtracting two close numbers doesn't create more absolute error, but it shrinks the answer's own size, so the same absolute error now represents a much larger fraction of a much smaller result — the accurate digits cancel out along with the values, leaving only noise.

Worked example 1: catastrophic cancellation by hand

Compute $x = \sqrt{1{,}000{,}001} - \sqrt{1{,}000{,}000}$ using 6 significant digits, as a limited calculator might. $\sqrt{1{,}000{,}001} \approx 1000.0005$, rounded to 6 digits: $1000.00$. $\sqrt{1{,}000{,}000} = 1000.00$ exactly. Subtracting gives $0$ — completely wrong; the true answer is about $0.0005$. Rewriting algebraically using $\sqrt{a}-\sqrt{b} = \frac{a-b}{\sqrt{a}+\sqrt{b}}$ avoids the subtraction of close numbers entirely: $\frac{1}{\sqrt{1{,}000{,}001}+\sqrt{1{,}000{,}000}} \approx \frac{1}{2000.00} = 0.0005$, matching the true value even with only 6-digit precision.

Worked example 2: a P&L example with real dollar figures

A book's mark-to-market P&L is computed as $\text{ending value} - \text{starting value}$, where a position worth roughly $50,000,000 moves by only $120 in a quiet session — a true relative change of about $2.4 \times 10^{-6}$. If ending and starting values are each stored with a rounding error of about $\varepsilon \times 50{,}000{,}000 \approx 50{,}000{,}000 \times 2.2\times10^{-16} \approx 0.000011$, that's a rounding noise floor of roughly one-hundredth of a cent — negligible next to $120, so this particular subtraction is actually fine. The lesson isn't "always avoid subtraction" but "check how large the noise floor is relative to the answer you're computing" — cancellation is only catastrophic when the result is close in size to the rounding error itself.

What this means in practice

Numerical stability governs which formula a pricing library actually implements, not just which formula a textbook writes down: computing implied volatility, Greeks via finite differences, variance as $\mathbb{E}[X^2]-(\mathbb{E}[X])^2$ (unstable — prefer accumulating squared deviations from a running mean), or covariance matrices from near-collinear factors. Anywhere a calculation subtracts two large, similar quantities to get a small answer, or divides by a number close to zero, stability deserves a second look before trusting the output.