LABS · 18  /  ARCHITECTURE

The Switchboard.

A trillion-parameter model that runs like a small one? The trick is a switchboard. Instead of pushing every token through every weight, a router sends each token to just a few of many experts — so the model stores a giant and pays for a sliver. Route the tokens, overload an expert, then balance the load.

Act 1 · The router picks
Each token, a few experts
The router scores every expert for the incoming token and picks the top-k (here 2 of 8). Only those experts run; the rest stay dark. Send tokens through and watch the switchboard light up — different tokens, different experts.
0
Tokens routed
2 / 8
Experts active / token
25%
Compute used / token
Two of eight experts run per token — so this layer does a quarter of the work a dense layer would, while holding 8 experts' worth of knowledge. Scale "8" to "256" and the sparsity gets dramatic.
Act 2 · Capacity vs compute
Store a giant, pay for a sliver
MoE decouples two numbers that are locked together in dense models: total parameters (the knowledge you store) and active parameters (the compute you pay per token). Slide the expert count and top-k, and watch them diverge.
Experts64
Top-k (experts per token)8
— B
Total parameters
stored in VRAM
— B
Active / token
compute + speed
—×
Sparsity ratio
This is the whole pitch. DeepSeek-V3 stores 671B parameters but activates ~37B per token — frontier knowledge at mid-model compute cost. Fine-grained MoE (many small experts) pushes the ratio further.
Act 3 · The failure mode
One expert does all the work
Left alone, the router plays favorites — a few experts get flooded while others sit idle and never learn. That's wasted capacity and a system bottleneck. Send a burst with balancing off to watch it collapse, then switch it on.
Busiest expert load
Idle experts
Imbalance (CV)
lower = more even
Balancing off: traffic piles onto favorites, the rest starve. On: tokens spread evenly, every expert learns and the busy one stops being a bottleneck. DeepSeek-V3 does this loss-free — a per-expert routing bias, no tax on the training objective.

Why sparse beats dense — for the money

In a dense transformer, every token flows through every parameter. Doubling the model doubles both its knowledge and the cost of every token — capacity and compute are welded together. Mixture-of-experts breaks the weld.

Replace one big feed-forward block with many parallel experts and a small router. For each token, the router scores the experts and activates only the top-k — the rest never run. So you can pile on experts to grow total capacity, while per-token compute stays pinned to just the k that fire. Store a giant; pay for a sliver.

Active vs total — the number that confuses everyone

Total parameters set memory (every expert sits in VRAM). Active parameters set speed and cost (only the shared layers plus the chosen experts run). DeepSeek-V3: 671B total, ~37B active. When a spec sheet quotes both, this is why.

The catch: balance

Sparsity's dark side is load imbalance. Routers naturally collapse onto a few favorite experts — those overload while others atrophy, wasting the capacity you paid for and jamming the GPUs that host the hot experts. The classic fix was an auxiliary loss penalizing imbalance, but it fights the main objective (you pay quality for balance). DeepSeek-V3's answer is loss-free: a per-expert bias used only for routing, nudged up for underused experts and down for overloaded ones — balance from a feedback controller, not a tax.

Why it's everywhere now

Scaling laws want bigger models; budgets don't want bigger bills. MoE resolves the tension, so several frontier open models — DeepSeek-V3, Kimi K2, Mixtral — are sparse. Fine-grained MoE (many small experts, higher top-k) refines the same idea, and the MoE paper traces the lineage.

Check yourself

Four questions. The router will grade you.

Finished this one? 0 / 22 Labs done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Labs