Mixture of Experts,
explained.

In a normal dense network, every parameter is used for every input. That's a hard tax: to make the model twice as capable you double the parameters, and now every single token costs twice as much compute. Capacity and cost are chained together, so scaling up gets expensive fast.

But intuitively, that's wasteful. A token about French poetry and a token about Python syntax probably don't need the same billions of parameters. What if different parts of the model handled different kinds of input?

A Mixture of Experts replaces one big dense layer with many smaller sub-networks — the experts — plus a small gating network that acts as a receptionist. For each token, the gate picks just the top one or two experts to handle it. The rest sit idle for that token.

So the model can have dozens or hundreds of experts — an enormous total parameter count — while any single token only ever touches a couple of them. This is sparse activation: lots of capacity, little compute per token.

The gate is itself learned. For each token it scores every expert, keeps the highest few (top-k routing, often k=1 or 2), and sends the token only to those. Their outputs are blended, weighted by the gate's scores. Over training, the gate discovers which experts are good at what and routes accordingly.

Only the two lit experts run. The other N−2 — the bulk of the model's parameters — cost nothing for this token.

Left to its own devices, the gate cheats. It finds a few experts that work well early on and keeps sending everything to them, while most experts starve — never getting enough tokens to learn anything. A model with 128 experts that only ever uses 4 is mostly dead weight.

The original Sparsely-Gated MoE built layers with up to thousands of experts and billions of parameters, beating strong dense baselines at a fraction of the compute. The later Switch Transformer simplified routing to a single expert per token and scaled to over a trillion parameters — while keeping per-token cost roughly flat.

The chain between capacity and cost — the thing that made scaling expensive — is broken.

Mixture of Experts is one of the main levers behind the frontier. Open models like Mixtral and DeepSeek are MoE, and several leading proprietary models are widely believed to be as well. When you hear a model has "hundreds of billions of parameters" but runs surprisingly cheaply, MoE is usually why — only a slice of it wakes up for each token.

The deeper idea — conditional computation, spending compute only where it's needed — is one of the most powerful tools we have for scaling models without scaling their bills. This paper is where it went from theory to practice.