GELU and Swish Activations

ReLU makes an all-or-nothing decision at zero: pass the input through unchanged, or kill it completely, with a sharp corner exactly at the boundary. GELU and Swish replace that hard corner with a smooth curve that makes a soft, probabilistic-flavored decision instead — mostly passing through positive inputs, mostly killing negative ones, but blending smoothly through the region near zero rather than snapping between the two. That smoothness turns out to matter for how well very deep networks, especially transformers, train.

The analogy: a bouncer with judgment versus a strict age cutoff

ReLU is a bouncer with a rigid rule: exactly 21 years old and older gets in, everyone younger is turned away, full stop, no exceptions, no partial admittance. GELU is a bouncer using judgment: someone clearly 30 gets in without hesitation, someone clearly 12 is turned away without hesitation, but someone right around the boundary — 20, 21, 22 — gets a probability-weighted decision that blends smoothly from "mostly no" to "mostly yes" rather than flipping instantly at one exact cutoff. That smooth blending region is exactly where GELU and ReLU differ, and it is also exactly the region where ReLU's derivative discontinuity caused problems.

The maths

GELU multiplies the input by the probability that a standard normal random variable is less than that input:

$$\text{GELU}(z) = z \cdot \Phi(z), \qquad \Phi(z) = P(X \le z) \text{ for } X \sim \mathcal{N}(0,1)$$

In plain English: scale the input down by how "likely" it is to be a large positive value under a standard bell curve — large positive $z$ gets multiplied by something close to $1$ (pass through almost unchanged), large negative $z$ gets multiplied by something close to $0$ (mostly killed), and values near zero get a smooth, partial pass-through instead of a hard cutoff.

Swish (also called SiLU) uses a similar idea with the logistic sigmoid instead of the normal CDF:

$$\text{Swish}(z) = z \cdot \sigma(z), \qquad \sigma(z) = \frac{1}{1+e^{-z}}$$

In plain English: same "scale the input by a smooth 0-to-1 gate" idea, just using the sigmoid curve as the gate instead of the normal CDF — the two functions look almost identical in shape and both smoothly interpolate ReLU's hard corner.

Crucially, unlike ReLU, both have a nonzero derivative almost everywhere, including for negative inputs — so a unit that currently outputs something small and negative still receives a usable, nonzero gradient, sidestepping the dying-unit problem from ReLU, Dying Units and Leaky Variants without needing an explicit leak parameter.

Worked example 1: comparing outputs at a few points

At $z=2$ (clearly positive): ReLU gives $2$; GELU gives $2 \times \Phi(2) = 2 \times 0.977 = 1.954$; Swish gives $2 \times \sigma(2) = 2 \times 0.881 = 1.762$ — all three close to $2$, as expected for confidently positive inputs. At $z=-1$ (moderately negative): ReLU gives exactly $0$; GELU gives $-1 \times \Phi(-1) = -1 \times 0.159 = -0.159$; Swish gives $-1 \times \sigma(-1) = -1 \times 0.269 = -0.269$ — both let a small negative signal through where ReLU kills it outright. At $z=-4$ (strongly negative): GELU gives $-4 \times \Phi(-4) \approx -4 \times 0.0000317 \approx -0.00013$ — essentially zero, just like ReLU, showing the smoothness is concentrated near the boundary, not spread everywhere.

Worked example 2: the derivative near zero

At exactly $z=0$: ReLU's derivative jumps discontinuously from $0$ (just below zero) to $1$ (just above zero) — a genuine kink with no single well-defined value at the point itself. GELU's derivative at $z=0$ is $\Phi(0) = 0.5$, smoothly connecting the region below to the region above, with no discontinuity anywhere. This matters most for gradient-based optimizers operating near that boundary — a discontinuous derivative can make optimization landscapes locally jagged in a way a smooth derivative does not.

The sigmoid gate inside Swish is exactly this logistic curve — drag its steepness in the explorer above and watch the transition sharpen toward ReLU's hard corner or soften into a gentler blend, which is the entire tunable difference between the two families.

What this means in practice

GELU is the default activation in most modern transformer architectures (BERT, GPT-family models, and their descendants), largely because empirical results on very deep networks favored its smoothness. Swish and GELU perform similarly in most settings, and the choice between them is often more a matter of what a given codebase or paper standardized on than a large practical difference — unlike the ReLU-versus-Leaky-ReLU choice, where dying units are a concrete, diagnosable failure mode being fixed.