Numerical Tolerance in Tests

assert result == 2.1 looks harmless until result is the output of a chain of floating-point additions and comes back as 2.099999999999998. The math was right; the hardware just doesn't represent most decimals exactly. A test written with exact equality on numerical code will fail on correct code and, worse, sometimes pass on subtly wrong code purely by coincidence. Numerical tolerance is the discipline of comparing "close enough" instead of "identical," and choosing what "close enough" means on purpose rather than by trial and error.

The idea: absolute tolerance, relative tolerance, and why you need both

Two numbers a and b are compared as equal-within-tolerance when:

$$|a - b| \le \text{atol} + \text{rtol} \cdot |b|$$

In words: the allowed gap between the two numbers is a fixed floor (atol, the absolute tolerance) plus a fraction of the size of b (rtol, the relative tolerance). Both terms exist because neither alone works across every scale of number.

Relative tolerance alone breaks near zero. If the expected value is 0.0 and the actual is 1e-15 (rounding noise), the relative gap is technically infinite — 1e-15 is not "close" to 0 in relative terms, even though every reasonable person would call that a pass. atol is the floor that rescues this case.

Absolute tolerance alone breaks at large scale. If atol=1e-6 and you're comparing portfolio values in the millions, a genuinely wrong calculation that's off by $50 would still pass, because 1e-6 is a meaningless bar next to a seven-figure number. rtol scales the bar to the size of the number being checked.

Worked example 1: why == fails on correct code

a = 0.1 + 0.2
print(a == 0.3)              # False
print(a)                     # 0.30000000000000004

0.1 and 0.2 cannot be represented exactly in binary floating point, so their sum is off from 0.3 by about 5.5e-17 — a rounding error, not a bug. Using numpy.testing.assert_allclose(a, 0.3) passes because its default tolerances (rtol=1e-7, atol=0) comfortably cover an error of that size, while a == 0.3 will fail forever, on every correct implementation, on every machine.

Worked example 2: picking tolerance for a real pricer test

A Black-Scholes call pricer should return 10.4506 for a known set of inputs, computed by a trusted reference. The pricer itself sums several terms, each carrying its own small rounding error.

import numpy as np

def test_black_scholes_call_matches_reference():
    price = black_scholes_call(s=100, k=100, r=0.05, sigma=0.2, t=1.0)
    expected = 10.4506
    np.testing.assert_allclose(price, expected, rtol=1e-4, atol=1e-6)

Here rtol=1e-4 allows the price to be off by up to about 0.01% of its own value — generous enough to absorb differences between two correct implementations that sum terms in a different order, but tight enough (roughly $0.001 on a $10 price) to catch a real bug like a wrong sign on the dividend term or a swapped d1/d2. atol=1e-6 is there only to protect against the price legitimately rounding to something extremely close to zero for a deep out-of-the-money option, where rtol alone would demand unreasonable precision.

# in practice, always prefer the library helper over hand-rolled comparisons
np.testing.assert_allclose(actual, expected, rtol=1e-6, atol=1e-9)

What this means in practice

There's no fixed "right" tolerance — it depends on how many floating-point operations accumulate error along the way (more operations, more accumulated rounding, wider justified tolerance) and how sensitive the downstream use is (a risk limit check needs tighter tolerance than a display rounding to two decimals). A useful habit: set the tolerance an order of magnitude tighter than the smallest error that would actually matter for the test's purpose, not as tight as the hardware allows.

Where it shows up

In interviews, this surfaces as "why doesn't this test pass" when the interviewer hands you code with an == comparison on floats — spotting it and reaching for rtol/atol is the expected fix. In production, every pricing library, risk engine, and backtest reconciliation depends on numerical tests with correctly chosen tolerances; get them wrong in either direction and either CI is unusable (flaky) or a real regression ships silently (too loose) — see Golden Master and Regression Tests for the related problem of comparing whole outputs, not just single numbers.