The Neural Tangent Kernel

A neural network's training dynamics are normally opaque: millions of weights update simultaneously, and there is no simple formula for what the network will predict after training. The neural tangent kernel (NTK) is a surprising theoretical shortcut discovered for the special case of very wide networks: as the width of the hidden layers grows toward infinity, the network's behavior during gradient descent converges to a fixed, describable kernel — meaning the messy, nonlinear learning process starts to look exactly like a classical, well-understood kernel regression.

Concretely, imagine slightly nudging a network's weights and asking how its output changes at every possible input — that sensitivity pattern is captured by the NTK, and the key finding is that in the infinite-width limit, this kernel stops changing during training. Weights still move, but the function they represent evolves as if governed by a fixed similarity measure between inputs, exactly analogous to how a kernel machine (like a support vector machine) compares new points to training points through a fixed kernel function rather than learning new "features" as it goes.

This matters because it gives researchers a tractable proxy for understanding why extremely wide networks train reliably and generalize as well as they do, even though the underlying optimization is nonconvex. It also explains a counterintuitive fact: very wide networks tend to behave more predictably and are easier to analyze than the moderately-sized networks used in practice, precisely because they sit closer to this fixed-kernel regime.