Gradient Checking by Finite Differences

Backpropagation code can run without crashing and still compute the wrong gradient — a sign flipped, a term forgotten, a chain-rule step applied to the wrong layer — and a training loss that goes down slightly can mask a real bug rather than confirm correctness, since even a partially wrong gradient sometimes nudges the loss in the right direction by luck. Gradient checking gives a cheap, mechanical way to catch this before trusting a training run: instead of relying on the analytic formula your code implements, estimate the gradient a completely different way and see if the two answers agree.

The idea is the plain definition of a derivative: how much does the loss change if you nudge one parameter by a tiny amount? For a single weight $w$, the numerical gradient is

$$\frac{\partial L}{\partial w} \approx \frac{L(w+\epsilon) - L(w-\epsilon)}{2\epsilon} ,$$

which in words means: increase the weight a tiny bit, decrease it the same tiny bit, rerun the forward pass both times, and see how much the loss moved per unit of nudge — using both directions (rather than just $w+\epsilon$) cancels out more of the approximation error. A typical value for $\epsilon$ is around $10^{-4}$ to $10^{-7}$; too large and the finite-difference estimate is inaccurate, too small and floating-point rounding error swamps the calculation.

In practice you don't check every weight — that would mean one full forward pass per parameter, which is far too slow for a network with millions of weights. Instead you pick a handful of parameters at random, compute both the analytic gradient (from your backprop code) and the numerical estimate for just those, and compare them: if they match to several decimal places, your backprop implementation is very likely correct; if they diverge meaningfully, you have a bug to hunt down before spending compute training on top of it.