The Universal Approximation Theorem

A reasonable worry about neural networks is that their shape — layers of weighted sums passed through simple nonlinear functions — seems too rigid to capture arbitrarily complicated real-world relationships. The universal approximation theorem answers this in the affirmative: a feedforward network with just one hidden layer, using a broad class of nonlinear activation functions and given enough hidden neurons, can approximate any continuous function on a bounded input region as closely as you like. The nonlinearity of the activation function is essential here — stack any number of purely linear layers together and the whole network collapses back to a single linear function, no matter how many layers you add.

The intuition is that each hidden neuron, with a sigmoid or similar activation, can be shaped into something close to a smooth "step" or "bump" located at a particular part of the input space; combining enough of these bumps with different heights and positions lets you build up an approximation to almost any smooth curve or surface, the same way enough narrow rectangles can approximate the area under any curve.

Concrete illustration. A one-hidden-layer network approximating a wiggly nonlinear function of a single stock feature might use a few dozen hidden units, each contributing a shifted, scaled sigmoid "step" at a different point along the input range; summed together with the right weights, these steps combine into a smooth curve that tracks the target function closely — with more hidden units, the approximation can be made arbitrarily tight.