Sigmoid Saturation and Logit Scaling

The sigmoid function squashes any input into a value between 0 and 1: $\sigma(z) = \frac{1}{1 + e^{-z}}$. It's shaped like a flattened S — steep and responsive near $z=0$, but nearly flat once $z$ gets much larger than about $+5$ or smaller than about $-5$. In that flat region the function is said to saturate: its output barely changes as the input changes further, meaning $\sigma(z)$ is close to 1 for large positive $z$ and close to 0 for large negative $z$, no matter how much larger or smaller $z$ gets.

The gradient of the sigmoid is $\sigma(z)(1-\sigma(z))$, which is largest (0.25) at $z=0$ and shrinks toward zero as $\sigma(z)$ approaches 0 or 1. So a saturated neuron — one whose input $z$ is far from zero — has a near-zero gradient, meaning backpropagation passes almost no learning signal through it: the weights feeding into that neuron barely update, even if the neuron's output is completely wrong for the task.

This is why practitioners scale the pre-activation logits (the raw $z$ values before the sigmoid) to stay in a moderate range, typically by careful weight initialization and input normalization, rather than letting them grow large during training. A logit of $z=8$ gives $\sigma(8) \approx 0.9997$, with gradient $\sigma(z)(1-\sigma(z)) \approx 0.0003$ — a value 800 times smaller than the gradient at $z=0$, effectively freezing that neuron's learning.