Mixture of Experts (MoE)
also: MoE
A model that routes each token to a few specialist sub-networks instead of the whole network.
A Mixture of Experts replaces a dense layer with many “expert” sub-networks and a router that activates only a few per token. This grows total parameters (capacity) while keeping the compute per token roughly constant — the design behind many frontier models.
Worked example: a model with 64 experts that activates 2 per token has huge total capacity but does roughly the per-token compute of a much smaller dense model — a “hundreds-of-billions” MoE can decode with the FLOPs of a ~tens-of-billions dense one. Gotcha: MoE saves compute, not memory — every expert must still be loaded in VRAM even though only a couple run per token, so the model is cheap to run but expensive to host, and the router becoming unbalanced (dead experts) is a real training failure mode.