Mixture of Experts (MoE)
also: MoE
A model that routes each token to a few specialist sub-networks instead of the whole network.
A Mixture of Experts replaces a dense layer with many “expert” sub-networks and a router that activates only a few per token. This grows total parameters (capacity) while keeping the compute per token roughly constant — the design behind many frontier models.