Switch Transformer,
explained.

Rung 1 sends each token to several experts and combines their outputs, on the argument that the gate needs a comparison to learn from. This rung's change is to send each token to exactly one, and to show that the argument was wrong.

Everything gets simpler at once, and the savings compound rather than adding: one expert's arithmetic per token instead of two or more, one set of activations to route rather than several, and — the part that dominates at scale — each token crosses the network to one destination, so the communication volume the routing generates is cut proportionally. Simplicity here is not elegance, it is the difference between a mechanism that scales across a large cluster and one that does not.

Two pieces of bookkeeping make top-1 safe, and both are worth knowing because they are where MoE models actually misbehave. Expert capacity caps how many tokens any one expert will accept in a batch, so a popular expert cannot stall everything behind it. And when that cap is hit, the excess tokens are dropped — they skip the expert layer entirely and pass through the residual connection unmodified. That is a real and slightly alarming property to hold on to: in a production MoE, some tokens receive less computation than others, determined by what else happened to be in the batch. The capacity factor is the dial trading wasted compute against how often that happens. Stated in units, this is the configuration that took the family past a trillion total parameters while keeping per-token compute tractable.

A mixture of experts replaces a transformer's single feed-forward layer with many parallel ones — the experts — and a small router that decides which expert each token should use. The payoff is sparse activation: the model can carry an enormous number of parameters, but each token only touches a few of them, so the compute per token barely rises. Capacity grows without the usual proportional cost in FLOPs.

The catch, before this paper, was that MoE was hard to train. Routing to two or more experts per token added complexity and communication overhead, gradients were noisy, and — worst of all — routers collapsed, funneling most tokens to a handful of favored experts while the rest sat idle. Switch Transformer's contribution was to make MoE simple and stable enough to scale reliably, and its central move was counterintuitive: route less, not more.

Earlier MoE work insisted you needed at least two experts per token — the belief was that routing needed a comparison to produce useful gradients. Switch Transformer challenged that and used top-1 routing: the router scores the experts for a token, and the token goes to the single highest-scoring one. Nothing else.

The simplification pays off three ways. Routing computation halves or better. The all-to-all communication that shuffles tokens to their experts across devices shrinks. And the implementation gets dramatically cleaner — one expert per token means fixed, predictable tensor shapes. The router still produces a probability for the chosen expert, which is used to weight the expert's output so gradients flow back to the router. Less machinery, same learning signal.

Left to itself, a router develops favorites — a rich-get-richer loop where a few experts get most tokens, train faster, and get chosen even more, while the rest waste their parameters. Switch counters this with a load-balancing auxiliary loss, added to the training objective, that is minimized only when tokens spread evenly across experts. With N experts, f_i the fraction of tokens routed to expert i, and P_i the router's mean probability for expert i:

The product f_i·P_i couples how many tokens an expert actually gets to how confident the router is about it, and the sum is smallest when both are spread uniformly (each 1/N), giving N·N·(1/N)² = 1. Pile everything onto one expert and it climbs toward N. Because it is differentiable through P_i, minimizing it during training actively pushes the router toward even utilization. The runnable version below computes it for balanced and skewed loads.

The balancing loss is a soft nudge; hardware needs a hard guarantee. On real accelerators every expert must be given a fixed-size buffer, so Switch sets an expert capacity — the maximum tokens one expert will process in a batch — as tokens-per-expert times a tunable capacity factor:

If routing still sends an expert more tokens than its capacity, the overflow tokens are dropped — they skip the expert entirely and pass through unchanged via the residual connection. A capacity factor above 1 adds slack to reduce dropping, at the cost of wasted compute on padding; below 1 saves compute but drops more. It is the explicit lever that trades a little accuracy for guaranteed, hardware-friendly balance.

Fedus, Zoph and Shazeer build the Switch Transformer on T5, the text-to-text encoder-decoder, and change only the feed-forward layers. A token's trip through the model and through training looks like this.

1. Replace feed-forward layers with Switch layers. In most configurations every other feed-forward block becomes a Switch layer: N expert feed-forward networks plus a router. Attention and everything else stay dense.
2. Route in float32. The router multiplies each token's representation by W_r, applies a softmax over experts, and sends the token to the single highest-probability expert. The model runs in bfloat16, but the router's input is cast to float32 inside the router and the result cast back before dispatch. A routing decision rests on small differences between scores, and the exponentials in the softmax amplify rounding error; the paper calls this fix selective precision.
3. Allocate fixed buffers. Each expert gets capacity = (tokens per batch ÷ experts) × capacity factor slots. Accelerators need static tensor shapes, so the buffer size is fixed before anyone knows how tokens will route.
4. Dispatch, drop, compute, return. An all-to-all exchange sends tokens to the devices holding their experts. Tokens beyond an expert's capacity are dropped: they skip the expert and continue on the residual connection. Each expert's output is multiplied by its router probability before it returns, and that multiplication is the path by which the router receives gradient.
5. Balance with an auxiliary loss. Add α·N·Σ f_i·P_i with α = 10^⁻², where f_i is the fraction of tokens sent to expert i and P_i the mean router probability for it.
6. Initialize small. Weights are drawn from a truncated normal with standard deviation √(s/n), n being the fan-in, and the scale s is cut from T5's 1.0 to 0.1. The authors found this improved quality and sharply reduced variation between runs.
7. Pre-train like T5. The same C4 corpus and span-corruption objective, against T5 baselines with the same FLOPs per token, so any gain is bought with parameters rather than compute.
8. Fine-tune with expert dropout. A sparse model has a huge parameter count relative to a small fine-tuning set, so it overfits. The paper uses a dropout rate of 0.1 in the non-expert layers and 0.4 inside the experts.
9. Lay the model out across devices. Data parallelism splits the batch, model parallelism splits each weight matrix across cores, and expert parallelism puts different experts on different cores. Switch-C, 1.6T parameters with 2,048 experts, uses only data and expert parallelism, which keeps its computation per token small; Switch-XXL, 395B parameters with 64 experts, combines all three to match the FLOPs per token of T5-XXL.

Why one expert is enough

The earlier argument for top-2 was that a router needs to compare experts to learn. The Switch layer still gives it a signal, because the chosen expert's output is scaled by its own probability: if the choice helped, the gradient raises that probability, and if it hurt, it lowers it. In exchange, top-1 halves the expert computation of top-2, roughly halves the tokens crossing the network, and lets each expert's buffer be smaller.

• Wr — the router weights; h(x) — one logit per expert; N — the number of experts.
• i* — the single chosen expert. The token's output is that expert's output times its own router probability. Without that factor the argmax would be a dead end for gradients; with it, the router learns whether boosting the chosen expert lowered the loss.
• fi — the fraction of the batch's tokens dispatched to expert i, a count with no gradient; Pi — the mean of p_i over the batch, which is differentiable, so the balancing gradient flows only through P_i.
• α — small enough not to swamp the language-model loss L_LM, large enough to keep routing even.

At the extremes: with perfectly uniform routing every f_i and P_i is 1/N and the auxiliary term equals α. If one expert took every token with probability near 1, it would approach α·N. Capacity adds a hard limit on top of this soft one: with a capacity factor of 1.0 and uneven routing, some tokens are dropped no matter what the loss says.

Every comparison holds FLOPs per token fixed against a T5 model, so the added parameters are the only change. Pre-training is on C4 with span corruption, and quality is measured as negative log perplexity.

The sample-efficiency result is the headline: up to about 7× faster to a target quality than a dense model at equal compute. Selective precision did what it was meant to: in the paper's comparison, a float32 router inside a bfloat16 model trained as stably as full float32 at bfloat16 speed, while a pure bfloat16 router diverged.

Negative results and limits

Switch-XXL, at 395B parameters, suffered sporadic training instability that Switch-C did not. Fine-tuning overfit, which is why expert dropout exists. Better pre-training perplexity carried over clearly to knowledge-heavy question answering but less reliably to reasoning-heavy fine-tuning tasks. And distillation recovered only part of the gain, so sparsity cannot be cashed out into a small dense model afterwards. One historical claim needs correcting: top-1 routing made sparse training simpler, but it did not become the standard. Mixtral routes each token to 2 of 8 experts, DeepSeek-V3 to 8 of 256 routed experts plus a shared one, and many current MoEs are dropless, computing every routed token rather than dropping overflow.

Two problems are reported honestly and both proved durable. Training instability: sparse models diverge more readily than dense ones at scale, and part of the mitigation is numerical — keeping the router's own arithmetic in higher precision than the rest of the model, because a routing decision is a discrete choice made from a small difference and is therefore unusually sensitive to rounding. And fine-tuning overfits more easily than the dense equivalent, which is intuitive once stated: a very large parameter count fine-tuned on a small task dataset has a lot of capacity to memorise with.

The paper also explores distilling a sparse model back down to a dense one and reports recovering only part of the gain — an honest negative result that delimits the claim. Sparsity is not a free lunch you can cash out into a small dense model afterwards.

How it held up: top-1 routing did not become the default. Most current mixture-of-experts models send each token to several experts — Mixtral to two of eight, DeepSeek-V3 to eight routed experts plus a shared one — and many train without dropping tokens. What survived is the case that sparse models scale at matched FLOPs, expert capacity as the unit of serving cost, and the auxiliary balancing loss, which DeepSeek-V3 then largely replaced with a bias-based, loss-free scheme. Worth noting the comparison that made the paper persuasive was at matched FLOPs rather than matched parameters — which is the correct comparison for this architecture and also the one that makes its numbers incomparable with dense-model headline figures, a confusion that persists in how MoE models are marketed.

With routing simplified and stabilized, the numbers got large. The Switch Transformer scaled to 1.6 trillion parameters — a headline figure at the time — while keeping the compute per token comparable to a far smaller dense model, because only one expert fires per token. That is the whole point of sparsity made concrete: parameters and FLOPs, usually chained together, come apart.

The trade is honest: you pay in memory and cross-device communication for parameters you activate only sparsely. But for large-scale training, spending memory to buy quality-per-FLOP was exactly the right exchange — and it reframed how people thought about scaling.

Switch Transformer took mixture-of-experts from a finicky research idea to a scalable, practical technique, and the modern MoE models that followed — from open trillion-scale releases to the sparse layers rumored inside frontier systems — build on its ideas: sparse routing to a few experts, an auxiliary balancing loss, and capacity limits. Its specific choices have been revised since — many newer systems route to more than one expert, and some drop the auxiliary loss, but the skeleton is the same.

Its lasting idea is the decoupling itself. For years, "bigger model" meant "more compute," full stop. Sparse expert routing broke that identity, letting parameter count and per-token compute scale on separate dials. That is why mixture-of-experts is now a central tool for building very large models economically — and Switch Transformer is the paper that made the tool dependable.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Sparsity: route each token to a few experts and buy parameter count without paying for it at inference. 2017 idea, 2026 frontier default.

1. 1Mixture of Experts2017
2. 2Switch Transformer2022
3. 3DeepSeek-V32024
4. 4DeepSeek-V42026
5. 5Qwen4-Coder2026

All 110 papers, in 19 ladders →