Mixture-of-Experts Layers

Make a network bigger and it usually gets better — but every extra parameter costs compute on every single input, whether that parameter turns out useful for this particular input or not. A dense layer with a trillion weights processes a one-word question with all trillion weights just as heavily as it processes a paragraph that actually needs that capacity. Most of that compute is wasted: a huge fraction of a giant network's knowledge is irrelevant to any given input, yet a dense architecture pays for all of it regardless. Mixture-of-experts (MoE) breaks that link between total capacity and per-input compute.

The analogy: a hospital's specialist roster

A large hospital keeps dozens of specialists on staff — cardiologists, neurologists, orthopedists, dermatologists — far more expertise than any single patient needs. A patient walking in doesn't see all of them; a triage nurse (the router) looks at the symptoms and sends the patient to the two or three specialists actually relevant to that case. The hospital's total capacity is enormous, but the cost of treating one patient only scales with the handful of specialists that patient actually sees. Add more specialists to the roster — more rare sub-specialties — and the hospital's total expertise grows, but an individual visit doesn't get slower, because the triage step keeps routing to a small, relevant subset. A mixture-of-experts layer is that hospital: many "expert" sub-networks, a router that reads each input and sends it to a few, and a total parameter count that can grow almost without bound while the compute per input stays roughly fixed.

The maths

An MoE layer holds $N$ expert networks $E_1, \dots, E_N$ (typically small feed-forward networks, each with its own weights) and a gating network $G$ that, given input $x$, produces a weight for each expert:

$$G(x) = \text{softmax}(x W_g)$$

In words: multiply the input by a learned gating weight matrix $W_g$, then run softmax to turn the raw scores into a probability-like distribution over the $N$ experts — a set of numbers between 0 and 1 that sum to 1, one per expert, saying how relevant each expert is to this specific input.

The layer's output, in the fully dense version, would be a weighted sum over every expert:

$$y = \sum_{i=1}^{N} G(x)_i \, E_i(x)$$

In words: run the input through every single expert, then blend their outputs using the gate's weights. This computes correctly but defeats the purpose — you still pay to run all $N$ experts on every input. The actual trick, sparse gating, zeroes out all but the top $k$ gate values (commonly $k=1$ or $k=2$) before the softmax normalization is applied to just those survivors:

$$G(x)_i = \begin{cases} \text{softmax}(x W_g)_i & \text{if } i \in \text{top-}k \\ 0 & \text{otherwise} \end{cases}$$

In words: find the $k$ experts the gate scored highest for this input, only compute those experts' outputs, and set every other expert's contribution to exactly zero — not small, zero, meaning they are never even evaluated. If $N = 128$ and $k = 2$, each input touches roughly 1.6% of the total expert parameters, while the model as a whole still has all 128 experts' worth of learned capacity available across the full range of inputs it might see.

Worked example 1: routing and blending with top-2 gating

Four experts, $N=4$, top-$k=2$. The gate's raw scores (before softmax) for an input are $x W_g = (2.0,\ 0.5,\ 3.0,\ -1.0)$. The top 2 by score are experts 1 and 3, with scores 2.0 and 3.0. Softmax over just those two:

$$G_1 = \frac{e^{2.0}}{e^{2.0}+e^{3.0}} = \frac{7.39}{7.39+20.09} \approx 0.269, \qquad G_3 = \frac{e^{3.0}}{7.39+20.09} \approx 0.731$$

Experts 2 and 4 get gate weight exactly 0 and are never evaluated. Suppose expert 1 outputs $E_1(x) = 1.2$ and expert 3 outputs $E_3(x) = 0.4$ (each a scalar here for simplicity). The layer's output:

$$y = 0.269 \times 1.2 + 0.731 \times 0.4 = 0.323 + 0.292 = 0.615$$

Notice the router's confidence directly sets the blend: expert 3 got more than double the weight of expert 1 because its raw score was higher, and the final output leans accordingly, without ever consulting experts 2 or 4.

Worked example 2: the compute saving, in FLOPs

Say each expert is a small feed-forward block costing 2 million FLOPs to evaluate on one input, and there are $N=64$ experts. A dense layer using all of them costs:

$$64 \times 2\text{M} = 128\text{M FLOPs per input}$$

A sparse MoE layer with top-$k=2$ costs the router (negligible, a single small matrix multiply, say 0.1M FLOPs) plus just the two selected experts:

$$0.1\text{M} + 2 \times 2\text{M} = 4.1\text{M FLOPs per input}$$

That's roughly a 31x reduction in per-input compute, while the model still has all 64 experts' combined parameters — over 100x the parameter count of a single dense 2-expert-equivalent layer — available to draw on across the full range of inputs it sees. This is precisely why MoE lets total model size scale far faster than compute budget: parameters and compute have been decoupled.

Sparsity's payoff scales with how large $N$ gets while $k$ stays fixed — compute grows linearly in $k$ regardless of $N$, while total capacity grows with $N$. Try steepening the exponent to see how quickly a fixed small budget ($k$) can leave a fast-growing quantity ($N$) behind, the same shape of gap that makes MoE attractive at scale.

What this means in practice

MoE layers are why the largest modern language and vision models can keep scaling parameter counts into the trillions without a proportional blowup in inference cost — a handful of the most capable production LLMs are MoE architectures for exactly this reason. For a quant, the more relevant lesson is architectural: any pipeline that would benefit from specialized sub-models — one expert tuned to trending regimes, another to mean-reverting ones, another to high-volatility crash days — can be built as an MoE instead of a hand-coded regime switch, letting a learned router figure out which specialist fits each market state rather than hard-coding the split.