Activation Functions

Stack ten layers of pure linear transformations — each just a weighted sum of the layer before — and multiply them out, and the result is still just one linear transformation. Depth adds nothing if every layer only does addition and scaling. Activation functions are the ingredient that breaks this: a small nonlinear bend inserted after each layer's linear step, without which a thousand-layer network would be mathematically no more expressive than a single layer.

The analogy: a thermostat versus a dimmer switch with a kink

A dimmer switch that scales brightness perfectly linearly with the dial position can only ever represent "more dial, proportionally more light" — chain ten such dimmers together and you still just get one overall scaling factor. Now imagine a switch with a kink: below a certain dial position, nothing happens at all; above it, brightness ramps up. That single kink is enough that combining several such switches can approximate almost any on/off, staged, or curved lighting pattern you like — sharp transitions, plateaus, complex combinations no pure scaling could ever produce. The kink is the nonlinearity, and it's the entire reason chaining many of these switches together buys you anything at all.

The maths: three common shapes

The sigmoid squashes any real number into the range $(0,1)$:

$$\sigma(z) = \frac{1}{1+e^{-z}}$$

In words: very negative inputs go to nearly 0, very positive inputs go to nearly 1, and there's a smooth S-shaped transition in between — useful for representing a probability, but with a problem: for large $|z|$ the curve goes almost flat, meaning its slope (needed for backpropagation) is nearly zero there.

Tanh is a rescaled sigmoid ranging over $(-1,1)$: $\tanh(z) = 2\sigma(2z)-1$. Same S-shape, but centred at zero, which tends to make gradients flow more evenly through a network than plain sigmoid.

ReLU (rectified linear unit) is the simplest and most widely used today:

$$\text{ReLU}(z) = \max(0, z)$$

In words: negative inputs are crushed to exactly zero, positive inputs pass through completely unchanged. It has no flat region on the positive side, so its gradient there is always exactly 1 — a huge practical advantage for training deep networks, at the cost of a "dead" flat zero region on the negative side.

Worked example 1: sigmoid saturates, ReLU doesn't

Take input $z = 5$. Sigmoid: $\sigma(5) = 1/(1+e^{-5}) = 1/(1+0.0067) \approx 0.993$. Its derivative is $\sigma(z)(1-\sigma(z)) \approx 0.993 \times 0.007 \approx 0.0067$ — tiny. Now $z=10$: $\sigma(10)\approx0.99995$, derivative $\approx0.00005$, even tinier. As inputs grow, sigmoid's gradient shrinks toward zero — this is saturation, and it's why gradients vanish in deep sigmoid networks. ReLU at $z=5$ and $z=10$: output is $5$ and $10$ respectively, gradient is exactly $1$ at both — no shrinking at all, however large the input.

Worked example 2: a dead ReLU

A ReLU neuron has weight $w=-2$, bias $b=0.1$. For input $x=1$: $z = -2(1)+0.1=-1.9$, so $\text{ReLU}(-1.9)=0$, and the gradient flowing backward through this neuron is exactly $0$ (ReLU's slope is $0$ for any negative input). No matter what the loss says should happen, this neuron passes zero gradient back and its weights don't update on this example. If most training inputs push $z$ negative for this neuron, it can get permanently stuck outputting zero forever — a "dead ReLU." Variants like Leaky ReLU, $\max(0.01z, z)$, fix this by giving negative inputs a small nonzero slope: at $z=-1.9$, Leaky ReLU outputs $-0.019$ and its gradient is $0.01$, not $0$ — small, but alive.

Drag the parameters in the explorer above to see the sigmoid's characteristic S-shape flatten out at the extremes — that flattening is exactly the saturation from worked example 1, and it's the visual signature of vanishing gradients.

The explorer above shows how a flat region of a loss surface makes descent crawl to a near-halt — a saturated sigmoid or a dead ReLU produces exactly this kind of flat region locally within the network, even if the overall loss surface elsewhere is fine.

What this means in practice

ReLU and its variants (Leaky ReLU, GELU, ELU) dominate hidden layers in modern deep networks because they avoid saturation on the positive side and are cheap to compute. Sigmoid survives mainly at output layers producing probabilities, and tanh sometimes in recurrent architectures. The choice of activation interacts directly with weight initialization and learning rate — a network can train fine or fail to train at all depending on whether these three choices are consistent with each other.