Mixture of Experts,
explained.

Rung 1, so the comparison is against the dense network every other ladder assumes. In a dense model, every parameter participates in every token's computation. That means the two things you most want to separate are welded together: to give the model more capacity you must also make every forward pass more expensive, for training and for every request you ever serve.

The change is conditional computation. Replace one large feed-forward layer with many expert sub-networks plus a small gating network that reads each token and selects a few experts to handle it. Only the selected experts run. Total parameter count can then grow by a large factor while the arithmetic per token stays roughly flat — the model is enormous in storage and ordinary in cost, and capacity and compute are decoupled.

Two failure modes are inherent to that design and both are addressed here, because neither is optional. The first is collapse: the gate has every incentive to converge on a few experts, since the experts it picks get trained and improve, so the rest stay bad and are never picked. The fix is an auxiliary loss that penalises unbalanced routing, plus noise in the gate so alternatives get sampled. The second is the shrinking batch: if a batch of tokens is split across many experts, each expert sees only a sliver, and the matrix multiplications that hardware is fast at become small and inefficient. Both are consequences of sparsity rather than bugs in this implementation, and both reappear in every rung above.

In a normal dense network, every parameter is used for every input. That's a hard tax: to make the model twice as capable you double the parameters, and now every single token costs twice as much compute. Capacity and cost are chained together, so scaling up gets expensive fast.

But intuitively, that's wasteful. A token about French poetry and a token about Python syntax probably don't need the same billions of parameters. What if different parts of the model handled different kinds of input?

A Mixture of Experts replaces one big dense layer with many smaller sub-networks — the experts — plus a small gating network that acts as a receptionist. For each token, the gate picks just the top one or two experts to handle it. The rest sit idle for that token.

So the model can have dozens or hundreds of experts — an enormous total parameter count — while any single token only ever touches a couple of them. This is sparse activation: lots of capacity, little compute per token.

The gate is itself learned. For each token it scores every expert, keeps the highest few (top-k routing, often k=1 or 2), and sends the token only to those. Their outputs are blended, weighted by the gate's scores. Over training, the gate discovers which experts are good at what and routes accordingly.

Only the two lit experts run. The other N−2 — the bulk of the model's parameters — cost nothing for this token.

Left to its own devices, the gate cheats. It finds a few experts that work well early on and keeps sending everything to them, while most experts starve — never getting enough tokens to learn anything. A model with 128 experts that only ever uses 4 is mostly dead weight.

Mixture of Experts is one of the main levers behind the frontier. Open models like Mixtral and DeepSeek are MoE, and several leading proprietary models are widely believed to be as well. When you hear a model has "hundreds of billions of parameters" but runs surprisingly cheaply, MoE is usually why — only a slice of it wakes up for each token.

The deeper idea — conditional computation, spending compute only where it's needed — is one of the most powerful tools we have for scaling models without scaling their bills. This paper is where it went from theory to practice.

The 2017 paper by Shazeer and colleagues puts its mixture-of-experts layer inside recurrent models, not Transformers: in its language models the layer sits between two stacked LSTM layers, and in its translation models inside the encoder and decoder. One pass through the layer goes like this.

1. Batch every time step together. The LSTM below is unrolled first, and its outputs at all time steps of all sequences reach the MoE layer at once, as one large batch. Applying the layer convolutionally across time multiplies the number of examples each expert can see by the sequence length.
2. Score every expert. The gate computes one clean score per expert, x·W_g, and a second projection, x·W_noise, that sets how much Gaussian noise to add to each expert's score during training. A softplus keeps the noise scale positive.
3. Keep the top k. All but the k largest noisy scores are set to −∞, and a softmax over the survivors gives weights that sum to 1. Every other expert gets exactly zero weight and is never computed. The paper uses small values of k.
4. Dispatch across devices. The LSTM layers and the gate are replicated with data parallelism, each of d devices processing its own batch of b examples. The experts are not replicated: each lives on one device, and every device sends each example to the devices holding its chosen experts. An expert therefore receives about k·b·d/n examples from all devices combined, rather than the k·b/n it would get from one device's batch. This is the paper's answer to what it calls the shrinking batch problem.
5. Compute and combine. Each expert is a small feed-forward network with one ReLU hidden layer. The outputs are sent back, multiplied by their gate weights and summed: y = Σ G(x)_i·E_i(x).
6. Add two balancing losses. Over each batch, an expert's importance is the sum of its gate weights, and the importance loss is a weight times the squared coefficient of variation of those sums. Equal importance can still hide unequal example counts, so a second load loss uses the noise to compute a smooth probability that each example selects each expert, and penalizes the squared coefficient of variation of the summed probabilities.
7. Backpropagate through the gate. Gradients reach W_g and W_noise through the gate weights of the selected experts and through the balancing losses. Experts that an example did not select get no gradient from it.
8. Go hierarchical for thousands of experts. For the largest layers, a primary gate picks a few groups of experts and a secondary gate inside each group picks experts, so no single softmax has to span tens of thousands of options.

Making it efficient on 2017 hardware

The binding constraint was arithmetic per byte moved: an expert does little computation for each input and output it exchanges over the network, so the paper makes the experts' hidden layers large relative to their inputs and outputs, and batches examples across time steps and devices. The largest model, with 131,072 experts and about 137B parameters, trained on a corpus of about 100 billion words of Google News.

• Ei — expert i, one of n; G(x)i — its gate weight, zero for all but k experts, so the sum runs over only k terms in practice.
• KeepTopK — leaves the k largest scores and sets the rest to −∞, so the softmax gives them exactly 0.
• Softplus((x·Wnoise)i) — a positive noise scale learned per expert from the input. Noise lets close runners-up sometimes win, and it is what makes the load loss differentiable.
• CV — the coefficient of variation, standard deviation over mean, across experts for a batch X; wimportance — a hand-set weight.

At the extremes: k = n turns the layer back into a dense mixture with every expert computed. With the noise scale at zero the gate is deterministic and the smooth load estimate disappears. CV = 0 means every expert collected the same total gate weight, and one expert taking everything maximizes it. The load loss swaps G for P(x, i) = Φ(((x·W_g)_i − kth_excluding(H(x), k, i)) / Softplus((x·W_noise)_i)): the probability that expert i stays in the top k when only its own noise is redrawn.

The comparisons hold computation per time step roughly fixed and grow only the expert count, so any gain is bought with parameters, not compute.

The layers held up to 131,072 experts and up to 137B parameters while beating strong dense baselines at a fraction of the compute. Inspecting the gate showed experts specializing by syntax and meaning, with individual experts drawing tokens from a coherent topic. The same capacity-for-compute trade reads clearly in later terms: a Mixtral-8×7B-shaped model stores about 47B parameters but runs about 13B per token, and giving it 64 experts would raise storage to about 363B with the same compute per token. The Switch Transformer later simplified routing to one expert per token and passed a trillion parameters.

The balancing ablation

The paper trained the same model with each balancing loss switched on and off. With neither loss, test perplexity was markedly worse, about 39.8 against about 35.6 with them, and the busiest expert received roughly 17.8 times the average load. Either loss alone recovered most of the gap, and using both gave the evenest load. The negative result at 131,072 experts is the other limit: past some point, extra sparsity stopped buying quality.

The paper is direct about the costs sparsity introduces: routing tokens to experts that live on different devices is a communication problem, so the gains are contingent on the interconnect, and the auxiliary balancing loss is a term that is explicitly fighting the objective you actually care about — it exists to keep the mechanism healthy, not to make the model better at language.

One belief it establishes was later overturned. Routing to at least two experts was argued to be necessary for the gate to receive a useful learning signal, on the reasoning that you need to compare alternatives to know a choice was good. Rung 2 shows a single expert per token works, which removes roughly half the cost and most of the complexity.

How it held up: it is the architectural basis of the current frontier. The models at the top of this ladder and several on the open-weights ladder are mixtures of experts, and the reason is exactly the decoupling above — it is the only known way to buy capacity without buying proportional serving cost. What has been revised is every mechanism: top-1 replaced top-k, and rung 3 removes the auxiliary loss itself in favour of something that does not compete with the language objective.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Sparsity: route each token to a few experts and buy parameter count without paying for it at inference. 2017 idea, 2026 frontier default.

1. 1Mixture of Experts2017
2. 2Switch Transformer2022
3. 3DeepSeek-V32024
4. 4DeepSeek-V42026
5. 5Qwen4-Coder2026

All 110 papers, in 19 ladders →