Feedforward Blocks and Gated Linear Units

Attention is the transformer's mechanism for mixing information between tokens — deciding what each token should borrow from every other token. But attention alone never processes a single token's content in isolation; every transformer layer follows its attention step with a feedforward block that runs independently, with the same weights, on every token's vector one at a time, with no mixing across positions at all. This is where most of a transformer's actual per-token "thinking" happens, and roughly two-thirds of a typical transformer's parameters live in these feedforward blocks, not in attention.

The analogy: a shared meeting followed by private desk work

Attention is like a meeting where everyone briefly shares information and each person walks away having incorporated relevant input from others. The feedforward block is what happens next: everyone returns to their own desk and processes what they learned privately, using the exact same standard procedure (the same weights) everyone else also uses, with no further cross-talk. A gated linear unit changes that private procedure slightly: each person also decides, based on what they learned, how much of a second processing pathway to let through — a built-in, content-dependent volume knob, rather than a rigid, one-size-fits-all rule.

The mechanics: plain feedforward versus a gated variant

A standard transformer feedforward block is two linear layers with a nonlinearity between them, applied identically to every token:

$$\text{FFN}(x) = W_2\, \sigma(W_1 x + b_1) + b_2$$

In words: project the token's vector up to a wider intermediate dimension with $W_1$, apply a nonlinearity $\sigma$ (originally ReLU, later often GELU) elementwise, then project back down to the original dimension with $W_2$ — a simple expand-then-compress computation, applied independently to every position in the sequence.

A gated linear unit (GLU) variant instead computes two separate linear projections of the same input, uses one of them (passed through a nonlinearity) as a gate that multiplies elementwise into the other:

$$\text{GLU}(x) = \big(\sigma(W_1 x)\big) \odot (W_3 x), \qquad \text{FFN}_{\text{gated}}(x) = W_2\, \text{GLU}(x)$$

In words: one projection ($W_1 x$, passed through $\sigma$) decides, dimension by dimension, how much of the other projection ($W_3 x$) to let through, via elementwise multiplication $\odot$ — some dimensions of the intermediate representation get mostly shut off (gate value near 0), others pass through almost fully (gate value near 1), and this gating pattern is different for every input token, learned rather than fixed. This costs a third weight matrix compared to plain feedforward, so gated variants are typically built with a somewhat narrower intermediate dimension to keep total parameter count comparable.

Worked example 1: gating in action on two token types

Suppose, for a punctuation-mark token, the gate projection $\sigma(W_1 x)$ outputs values close to $[0.05, 0.9, 0.1]$ across three intermediate dimensions, while the content projection $W_3 x$ outputs $[4.0, 3.0, 5.0]$. The gated output is elementwise: $[0.2, 2.7, 0.5]$ — dimension 2 passes through almost fully, the others are mostly suppressed. For a content-bearing noun token, the same gate projection might instead output $[0.85, 0.1, 0.9]$, flipping which dimensions dominate: $[3.4, 0.3, 4.5]$. The same fixed weights $W_1, W_3$ produce a completely different effective computation depending on the input, because the gate itself is a function of that input.

Worked example 2: parameter budget comparison

A plain feedforward block with model dimension $d=512$ and intermediate dimension $4d=2048$ has two weight matrices of size $512\times2048$ each, for $2 \times 512 \times 2048 \approx 2.1$M parameters. A gated variant typically uses a smaller intermediate dimension, say $\tfrac{8}{3}d \approx 1365$, with three matrices of size $512\times1365$: $3 \times 512 \times 1365 \approx 2.1$M — deliberately shrinking width so the extra gating matrix doesn't blow past the plain version's parameter count.

The gate's sigmoid-shaped nonlinearity squashes values toward 0 or 1 — near-off or near-on — exactly the smooth switching behavior that lets a gated unit act like a soft, learned, per-dimension volume control rather than a rigid on/off rule.

What this means in practice

Feedforward blocks account for the bulk of a transformer's parameters and computation, so their design choices matter for both quality and cost. Gated linear unit variants (SwiGLU, GeGLU — named for which nonlinearity gates which projection) are now the standard choice in most modern large language models, replacing the plain ReLU feedforward of the original transformer, because they reliably improve downstream task performance at roughly matched parameter count, at the cost of a slightly more involved implementation with three matrices instead of two.

Practice

1. Write out the plain feedforward equation and identify which matrix expands the dimension and which compresses it back.
2. In the punctuation-token example, explain what it means for the gate value at a dimension to be close to 0.
3. Why do gated feedforward variants typically use a narrower intermediate dimension than plain feedforward blocks at the same total parameter budget?