Mixture of Experts,
explained.

Rung 1, so the comparison is against the dense network every other ladder assumes. In a dense model, every parameter participates in every token's computation. That means the two things you most want to separate are welded together: to give the model more capacity you must also make every forward pass more expensive, for training and for every request you ever serve.

The change is conditional computation. Replace one large feed-forward layer with many expert sub-networks plus a small gating network that reads each token and selects a few experts to handle it. Only the selected experts run. Total parameter count can then grow by a large factor while the arithmetic per token stays roughly flat — the model is enormous in storage and ordinary in cost, and capacity and compute are decoupled.

Two failure modes are inherent to that design and both are addressed here, because neither is optional. The first is collapse: the gate has every incentive to converge on a few experts, since the experts it picks get trained and improve, so the rest stay bad and are never picked. The fix is an auxiliary loss that penalises unbalanced routing, plus noise in the gate so alternatives get sampled. The second is the shrinking batch: if a batch of tokens is split across many experts, each expert sees only a sliver, and the matrix multiplications that hardware is fast at become small and inefficient. Both are consequences of sparsity rather than bugs in this implementation, and both reappear in every rung above.

In a normal dense network, every parameter is used for every input. That's a hard tax: to make the model twice as capable you double the parameters, and now every single token costs twice as much compute. Capacity and cost are chained together, so scaling up gets expensive fast.

But intuitively, that's wasteful. A token about French poetry and a token about Python syntax probably don't need the same billions of parameters. What if different parts of the model handled different kinds of input?

A Mixture of Experts replaces one big dense layer with many smaller sub-networks — the experts — plus a small gating network that acts as a receptionist. For each token, the gate picks just the top one or two experts to handle it. The rest sit idle for that token.

So the model can have dozens or hundreds of experts — an enormous total parameter count — while any single token only ever touches a couple of them. This is sparse activation: lots of capacity, little compute per token.

The gate is itself learned. For each token it scores every expert, keeps the highest few (top-k routing, often k=1 or 2), and sends the token only to those. Their outputs are blended, weighted by the gate's scores. Over training, the gate discovers which experts are good at what and routes accordingly.

Only the two lit experts run. The other N−2 — the bulk of the model's parameters — cost nothing for this token.

Left to its own devices, the gate cheats. It finds a few experts that work well early on and keeps sending everything to them, while most experts starve — never getting enough tokens to learn anything. A model with 128 experts that only ever uses 4 is mostly dead weight.

The original Sparsely-Gated MoE built layers with up to thousands of experts and billions of parameters, beating strong dense baselines at a fraction of the compute. The later Switch Transformer simplified routing to a single expert per token and scaled to over a trillion parameters — while keeping per-token cost roughly flat.

The chain between capacity and cost — the thing that made scaling expensive — is broken. And it's two formulas, not a metaphor. Store every expert, but run only k of the E per token:

The only difference is E vs k — capacity scales with the expert count you store, compute scales with the few you fire. Plug in a Mixtral-8×7B-shaped config (d=4096, 32 layers, 8 experts, top-2):

Eightfold the capacity, identical per-token compute — the whole point of conditional computation, in two numbers.

Mixture of Experts is one of the main levers behind the frontier. Open models like Mixtral and DeepSeek are MoE, and several leading proprietary models are widely believed to be as well. When you hear a model has "hundreds of billions of parameters" but runs surprisingly cheaply, MoE is usually why — only a slice of it wakes up for each token.

The deeper idea — conditional computation, spending compute only where it's needed — is one of the most powerful tools we have for scaling models without scaling their bills. This paper is where it went from theory to practice.

The paper is direct about the costs sparsity introduces: routing tokens to experts that live on different devices is a communication problem, so the gains are contingent on the interconnect, and the auxiliary balancing loss is a term that is explicitly fighting the objective you actually care about — it exists to keep the mechanism healthy, not to make the model better at language.

One belief it establishes was later overturned. Routing to at least two experts was argued to be necessary for the gate to receive a useful learning signal, on the reasoning that you need to compare alternatives to know a choice was good. Rung 2 shows a single expert per token works, which removes roughly half the cost and most of the complexity.

How it held up: it is the architectural basis of the current frontier. The models at the top of this ladder and several on the open-weights ladder are mixtures of experts, and the reason is exactly the decoupling above — it is the only known way to buy capacity without buying proportional serving cost. What has been revised is every mechanism: top-1 replaced top-k, and rung 3 removes the auxiliary loss itself in favour of something that does not compete with the language objective.

Mixture-of-Experts answers a specific bind: you want a model with more parameters (more capacity to know things) without paying more compute per token. MoE does it by replacing a dense feed-forward layer with many expert sub-networks and a router that sends each token to just a few of them. The model can be huge in total parameters while only a small slice activates per token — capacity without proportional cost.

The number that matters is active vs total parameters. A model might hold, say, 600B parameters but activate only ~37B per token; it pays roughly the inference cost of a 37B model while carrying the knowledge of something far larger. That decoupling is why several frontier-class models are MoE under the hood — it is one of the main ways labs push capability up while keeping serving affordable.

The catch is that MoE buys cheaper compute, not cheaper memory: every expert's weights must still be loaded, so the full parameter count sits in VRAM even though most of it is idle each step. MoE trades memory for compute — great when you are compute- or latency-bound, less helpful when memory is the wall.

The hard part in practice is the router. If it sends too many tokens to a few favorite experts, the rest sit unused and capacity is wasted, so training needs load-balancing pressure to keep experts evenly fed. Getting routing stable and balanced — rather than the expert idea itself — is where much of the real engineering goes, and it is why MoE was tricky to train well for years before it became mainstream.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

Sparsity: route each token to a few experts and buy parameter count without paying for it at inference. 2017 idea, 2026 frontier default.

1. 1Mixture of Experts2017
2. 2Switch Transformer2022
3. 3DeepSeek-V32024
4. 4DeepSeek-V42026
5. 5Qwen4-Coder2026

All 110 papers, in 19 ladders →