Floating-Point Arithmetic and Precision

Run 0.1 + 0.2 in almost any programming language and you get 0.30000000000000004, not 0.3. Nobody made a coding mistake. This is how every computer on earth stores non-integer numbers, and once you've traded on a system for long enough you will meet the consequence: two P&L numbers that "should" match to the penny differ by a cent, a strategy that compares a price to a threshold and gets it backwards once in a million calls, a risk check that passes when it should have failed by a hair.

The analogy: a ruler with fixed-width tick marks

Imagine a ruler that can only be printed with exactly 7 significant digits of ink, no matter how you zoom. Near the number 1, those 7 digits buy you extremely fine resolution — ticks a fraction of a millionth apart. Near the number 10,000,000, the same 7 digits of ink have to stretch across a much bigger range, so the ticks are now a full unit apart. The ruler didn't get worse — it always shows 7 digits — but the gap between adjacent representable numbers grows as the numbers themselves grow. Floating-point numbers work exactly like this ruler: fixed number of significant digits, gaps that scale with magnitude.

near x = 1,000,000 same 15-16 digit budget, but gaps now about $10^{-10}$ — still tiny, but 6 orders of magnitude coarser

Writing it down

A floating-point number is stored as

$$x = \pm\, m \times 2^{e}$$

where $m$ (the mantissa, or significand) holds the significant digits and $e$ (the exponent) says how far to shift the point. In words: it's scientific notation in binary — like $6.02 \times 10^{23}$, but with base 2 and a fixed budget of bits for $m$. The standard "double precision" format (what Python's float, and most languages' default, use) gives the mantissa 52 bits, which works out to about 15–17 significant decimal digits.

The critical number to know is machine epsilon, $\varepsilon$, the smallest gap representable next to 1.0 — about $2.2 \times 10^{-16}$ for double precision. In words: any two numbers that differ by less than roughly $\varepsilon$ times their own size are, to the computer, the same number. This is why 0.1 + 0.2 == 0.3 returns False: neither 0.1 nor 0.2 has an exact binary representation (just as $1/3$ has no exact finite decimal), so both are stored as the closest representable value, and the tiny stored errors don't cancel when added.

$$0.1 \approx 0.1000000000000000055511151231257827021181583404541015625$$

In words: that's not a bug in the display — that ugly long number is the actual value sitting in memory when you write "0.1" in code, because 0.1 in binary is a repeating fraction (like $1/3$ in decimal) that gets truncated to fit the mantissa's bit budget.

Drag the exponential explorer above and watch how quickly the curve's scale changes even while its shape stays the same — that's the same intuition as floating-point gaps: the absolute spacing between representable numbers grows exponentially with the exponent, even though the relative precision (digits of accuracy) stays constant throughout.

Worked example 1: catastrophic cancellation in a variance calculation

You compute the variance of daily returns using the "naive" formula $\text{Var} = E[X^2] - (E[X])^2$, on a set of prices all clustered near $1,000,000 (say a well-funded book's daily NAV). Suppose $E[X^2] = 1{,}000{,}000{,}002{,}345{,}678.9$ and $(E[X])^2 = 1{,}000{,}000{,}002{,}345{,}677.2$ — the true variance is their difference, $1.7$.

Both operands carry about 16 significant decimal digits of precision. But look at where the "interesting" digits are: they're the last few digits of a 16-digit number, which is exactly the region where floating-point rounding error lives. If each operand is only accurate to about $10^{15} \times \varepsilon \approx 0.2$ in absolute terms (since $\varepsilon\approx 2\times10^{-16}$ scaled by the magnitude $10^{15}$), then the true answer, 1.7, is being computed as a difference of two numbers each individually uncertain by about $0.2$ — meaning the result could be off by a comparable amount. A quantity you needed to three significant figures might come back with barely one reliable digit. This is catastrophic cancellation: subtracting two nearly equal large numbers amplifies relative error enormously, because the answer is small but the error in each operand isn't.

The fix used in every real numerical library is to compute variance from differences from the mean directly, $\frac{1}{n}\sum(x_i - \bar x)^2$, which never subtracts two huge numbers — it subtracts numbers that are already close to the small answer, so there's nothing large to cancel.

Worked example 2: why if price == threshold fails

A risk script checks if position_value == 500000.0: send_alert(). The position value is computed as 1000 * 500.0 + 0.0 across several intermediate float operations (position size times price, plus accumulated fees that should cancel). On paper this equals exactly 500,000. In the running system, the alert never fires.

Trace it: suppose one intermediate step computes 500000.00000000006 because of an earlier division that didn't land on an exact binary fraction (this happens routinely — even simple decimal quantities like prices in cents often aren't exactly representable in binary, the same way $1/3$ isn't exact in decimal). The stored value differs from 500000.0 by $6\times10^{-11}$ — utterly immaterial to any human, but == in floating point asks for bit-for-bit identity, and $6\times10^{-11} \ne 0$. The comparison silently returns False.

The standard fix is a tolerance check: instead of a == b, test abs(a - b) < tol for some small tol chosen relative to the scale of the numbers involved (often tol = 1e-9 * max(abs(a), abs(b)), a relative tolerance, since an absolute tolerance like 1e-9 that's fine for prices near $100 is far too tight for a notional near $10 billion and far too loose for a rate near 0.0001).

What this means in practice

• Never compare floats with ==. Use a relative tolerance everywhere a risk check, a reconciliation, or a unit test compares two computed floating-point values.
• Order of operations matters. Summing a long list of numbers from smallest to largest accumulates less error than largest to smallest, because adding a tiny number to an already-huge running total can lose it entirely (it falls below the resolution of that huge number's ruler ticks).
• Avoid subtracting nearly-equal large numbers. Rewrite formulas (like the variance example) to work with differences directly rather than computing them as a subtraction of two large intermediate quantities.
• Accumulated error in simulation. A Monte Carlo path or an Euler-Maruyama integration that runs millions of small update steps can drift measurably from the "true" trajectory purely from rounding, independent of any modeling error — worth checking against a higher-precision reference run.