Nonlinear Least Squares

Ordinary linear regression has a closed-form answer: plug the data into a formula and the best-fit line pops out. But if you're fitting a decay curve like $y = a e^{-bt}$, the parameter $b$ sits inside an exponential — there's no formula that solves for $a$ and $b$ directly. Nonlinear least squares handles this by starting from a guess and repeatedly adjusting the parameters downhill, step by step, until the fit stops improving.

Concretely, the method picks parameters to minimize the sum of squared errors between the curve and the data, exactly as in linear regression — the difference is purely in how it finds the minimum. Since there's no direct formula, an algorithm like Gauss-Newton or Levenberg-Marquardt approximates the error surface locally, takes a step toward lower error, re-approximates, and repeats until the parameters stop changing meaningfully. Unlike linear regression, this process can get stuck in a local minimum or fail to converge if the starting guess is poor, which is why a sensible starting value — often from domain knowledge, like "a bond's price decay rate is roughly its yield" — matters more here than it does for a linear fit.

A worked example: fitting $y = a e^{-bt}$ to a handful of bond price observations over time, a solver might start from $a=100$, $b=0.05$ and, after several iterations of nudging both parameters to shrink the squared error, converge to $a=98.7$, $b=0.043$ — a result you couldn't get from a single algebraic formula.