A trillion-parameter model that runs like a small one? The trick is a switchboard. Instead of pushing every token through every weight, a router sends each token to just a few of many experts — so the model stores a giant and pays for a sliver. Route the tokens, overload an expert, then balance the load.
Act 1 · The router picks
Each token, a few experts
The router scores every expert for the incoming token and picks the top-k (here 2 of 8). Only those experts run; the rest stay dark. Send tokens through and watch the switchboard light up — different tokens, different experts.
0
Tokens routed
2 / 8
Experts active / token
25%
Compute used / token
Two of eight experts run per token — so this layer does a quarter of the work a dense layer would, while holding 8 experts' worth of knowledge. Scale "8" to "256" and the sparsity gets dramatic.
Act 2 · Capacity vs compute
Store a giant, pay for a sliver
MoE decouples two numbers that are locked together in dense models: total parameters (the knowledge you store) and active parameters (the compute you pay per token). Slide the expert count and top-k, and watch them diverge.
Experts64
Top-k (experts per token)8
— B
Total parameters
stored in VRAM
— B
Active / token
compute + speed
—×
Sparsity ratio
This is the whole pitch. DeepSeek-V3 stores 671B parameters but activates ~37B per token — frontier knowledge at mid-model compute cost. Fine-grained MoE (many small experts) pushes the ratio further.
Act 3 · The failure mode
One expert does all the work
Left alone, the router plays favorites — a few experts get flooded while others sit idle and never learn. That's wasted capacity and a system bottleneck. Send a burst with balancing off to watch it collapse, then switch it on.
—
Busiest expert load
—
Idle experts
—
Imbalance (CV)
lower = more even
Balancing off: traffic piles onto favorites, the rest starve. On: tokens spread evenly, every expert learns and the busy one stops being a bottleneck. DeepSeek-V3 does this loss-free — a per-expert routing bias, no tax on the training objective.
Why sparse beats dense — for the money
In a dense transformer, every token flows through every parameter. Doubling the model doubles both its knowledge and the cost of every token — capacity and compute are welded together. Mixture-of-experts breaks the weld.
Replace one big feed-forward block with many parallel experts and a small router. For each token, the router scores the experts and activates only the top-k — the rest never run. So you can pile on experts to grow total capacity, while per-token compute stays pinned to just the k that fire. Store a giant; pay for a sliver.
Active vs total — the number that confuses everyone
Total parameters set memory (every expert sits in VRAM). Active parameters set speed and cost (only the shared layers plus the chosen experts run). DeepSeek-V3: 671B total, ~37B active. When a spec sheet quotes both, this is why.
The catch: balance
Sparsity's dark side is load imbalance. Routers naturally collapse onto a few favorite experts — those overload while others atrophy, wasting the capacity you paid for and jamming the GPUs that host the hot experts. The classic fix was an auxiliary loss penalizing imbalance, but it fights the main objective (you pay quality for balance). DeepSeek-V3's answer is loss-free: a per-expert bias used only for routing, nudged up for underused experts and down for overloaded ones — balance from a feedback controller, not a tax.
Why it's everywhere now
Scaling laws want bigger models; budgets don't want bigger bills. MoE resolves the tension, so several frontier open models — DeepSeek-V3, Kimi K2, Mixtral — are sparse. Fine-grained MoE (many small experts, higher top-k) refines the same idea, and the MoE paper traces the lineage.
Expert routing, step by step
One token through a router, how capacity and the balancing loss behave, and the active-versus-total arithmetic. The example uses 8 experts and top-2 so every number fits on the page; the widget defaults are 64 experts and top-8.
One token, four moves
Score. The router is a small linear layer: one logit per expert, z = W·x.
Pick.Softmax the logits and keep the top-k experts.
Renormalize. Divide the kept probabilities by their sum so the gate weights add to 1. Some designs skip this step.
Mix. Output y = Σ gᵢ · Eᵢ(x) over the chosen experts. The other experts never run for this token.
Worked example
Logits for experts 1 to 8: 2.0, 1.5, 0.5, 0.0, −0.5, −1.0, −1.5, −2.0. Softmax gives 46.6%, 28.3%, 10.4%, 6.3%, 3.8%, 2.3%, 1.4% and 0.9%.
Top-2 keeps experts 1 and 2, which hold 74.9% of the mass. Renormalized, the gates are 0.622 and 0.378.
Capacity. With 4,096 tokens, top-2 and 8 experts the average load is 1,024 tokens per expert. A capacity factor of 1.25 gives every expert room for 1,280; a token routed to a full expert is dropped from that expert and passes through on the residual connection alone.
Balance loss. The Switch Transformer penalty is α · N · Σ fᵢ·Pᵢ, with fᵢ the fraction of tokens sent to expert i and Pᵢ its mean router probability. With 4 experts and an even split it equals α. If 5 of 8 tokens pile onto one expert, with fractions (0.625, 0.125, 0.125, 0.125) and probabilities (0.4, 0.2, 0.2, 0.2), it rises to 1.3α.
Active versus total. DeepSeek-V3 activates about 37B of 671B parameters per token (5.5%); Mixtral 8x7B, per Mistral, about 12.9B of 46.7B (28%). Memory follows the total, speed follows the active.
Knobs and pitfalls
Raise k for quality and pay in compute; add experts for capacity and pay in memory. The balancing weight α is a trade: too small and experts collapse, too large and it hurts the main loss, which is what DeepSeek's bias approach avoids. In serving, experts are spread across GPUs, so every layer needs an all-to-all exchange of tokens, and a hot expert becomes the slowest GPU. Small batches touch many experts at once, so MoE's memory-bandwidth savings shrink at low batch sizes.
Check yourself
Four questions. The router will grade you.
See every answer, with the reasoning
In an MoE layer, how many experts run for each token?
Answer: Only the top-k the router picks (often 2) — Top-k routing: a few experts fire, the rest stay dark — that’s the sparsity.
What sets an MoE model’s memory footprint?
Answer: Total parameters — every expert sits in VRAM — Memory follows total params; compute and speed follow active params. DeepSeek-V3: 671B total, 37B active.
Why do MoE models need load balancing?
Answer: Routers collapse onto favorite experts, overloading some while others idle and never learn — Imbalance wastes capacity and jams the GPUs hosting hot experts.
What is DeepSeek-V3’s approach to balancing?
Answer: Loss-free: a per-expert routing bias nudged by usage, no tax on the training objective — A feedback controller on routing, so balance doesn’t come at the cost of quality.