Linear Solvers and the Condition Number

A portfolio optimizer solves $Ax = b$ for optimal weights, where $A$ is a covariance matrix. The inputs — estimated covariances — are never known exactly; there's always some noise from finite sample size. For a well-behaved matrix, small noise in the inputs produces small changes in the solution. For an ill-conditioned matrix, the same tiny input noise can blow up into wildly different, useless output weights — and this happens even with perfect floating-point arithmetic, purely because of the matrix's own geometry.

What the condition number measures

For a matrix $A$, the condition number is $\kappa(A) = \|A\| \cdot \|A^{-1}\|$ — informally, the ratio of the matrix's largest to smallest "stretch factor" (for a symmetric matrix, the ratio of its largest to smallest eigenvalue). A matrix that stretches some directions enormously and barely touches others has a huge condition number: information about the barely-touched direction is nearly invisible in $Ax$, so recovering it by inverting is extremely sensitive to any noise in $b$ or $A$ itself.

$$\frac{\|\delta x\|}{\|x\|} \lesssim \kappa(A) \times \frac{\|\delta b\|}{\|b\|}$$

In words: the relative error in your solution is bounded by the condition number times the relative error in your input. A condition number of 100 means input noise gets amplified up to 100x; a condition number of $10^{10}$ means the solution can be almost entirely noise.

Worked example

Take a nearly-singular 2x2 matrix $A = \begin{pmatrix} 1 & 1 \\ 1 & 1.0001 \end{pmatrix}$, with $b = \begin{pmatrix} 2 \\ 2.0001 \end{pmatrix}$. The exact solution is $x = (1, 1)$. Now perturb $b$ by a tiny amount, to $b' = \begin{pmatrix} 2 \\ 2.0002 \end{pmatrix}$ — a relative change of about $5 \times 10^{-5}$. Solving $Ax' = b'$ gives $x' \approx (0, 2)$ — the solution moved by 100%, not 0.005%. This matrix's condition number is roughly 40,000, and the amplification in the example (relative output change divided by relative input change) is consistent with that order of magnitude: a tiny nudge in the input became a completely different answer.

What this means in practice

Ill-conditioned systems show up whenever a quant inverts a covariance matrix built from highly correlated assets (two ETFs tracking nearly the same index), fits a regression with near-collinear features, or calibrates a model where two parameters trade off against each other almost perfectly. np.linalg.cond(A) reports the condition number directly; a common rule of thumb is to be cautious above $10^4$-$10^6$ and treat results as unreliable well before reaching the machine-precision limit (around $10^{16}$ for double precision, where the matrix is numerically singular). Standard fixes include ridge regularization (adding a small multiple of the identity before inverting), dropping near-duplicate features, or using a solver method (like SVD-based pseudo-inverse) that degrades more gracefully than direct inversion.