Gradient Checking by Finite Differences

Backpropagation code can run without crashing and still compute the wrong gradient, a sign flipped, a term forgotten, a chain-rule step applied to the wrong layer, and a training loss that goes down slightly can mask a real bug rather than confirm correctness, since even a partially wrong gradient sometimes nudges the loss in the right direction by luck. Gradient checking gives a cheap, mechanical way to catch this before trusting a training run: instead of relying on the analytic formula your code implements, estimate the gradient a completely different way and see if the two answers agree.

The idea is the plain definition of a derivative: how much does the loss change if you nudge one parameter by a tiny amount? For a single weight $w$, the numerical gradient is

which in words means: increase the weight a tiny bit, decrease it the same tiny bit, rerun the forward pass both times, and see how much the loss moved per unit of nudge, using both directions (rather than just $w+\epsilon$) cancels out more of the approximation error. A typical value for $\epsilon$ is around $10^{-4}$ to $10^{-7}$; too large and the finite-difference estimate is inaccurate, too small and floating-point rounding error swamps the calculation.

In practice you don't check every weight, that would mean one full forward pass per parameter, which is far too slow for a network with millions of weights. Instead you pick a handful of parameters at random, compute both the analytic gradient (from your backprop code) and the numerical estimate for just those, and compare them: if they match to several decimal places, your backprop implementation is very likely correct; if they diverge meaningfully, you have a bug to hunt down before spending compute training on top of it.