DeepSeek-V3,
explained.

V3 is a mixture-of-experts pushed toward its logical extreme: 671B total parameters, organized as many small experts, with a router activating only ~37B per token — about 5%. You store the knowledge of a giant; you pay the compute of a mid-size model. Each token consults 8 routed experts plus one shared expert that every token sees.

Fine-grained experts (many small ones rather than few big ones) give the router more precise choices — specialization without redundancy. The blueprint came from DeepSeek's own MoE work; V3's contribution was making this shape trainable and servable at 671B without the usual MoE taxes. Those taxes are what the next two sections remove.

Serving throughput dies on the KV cache — the per-token attention state that grows with context and batch. Standard fixes (GQA, MQA) share keys and values across heads: less memory, some quality loss. V3's multi-head latent attention goes further: compress keys and values into one small latent vector, cache only that, and reconstruct full per-head K/V from it at use time.

The result: KV memory an order of magnitude smaller than vanilla attention — longer contexts, bigger serving batches, cheaper tokens — with benchmark quality that matches or beats standard attention. It's the rare compression that costs nothing you can measure.

MoE's classic pathology: the router collapses onto a few favorite experts — they overload, the rest atrophy. The standard cure is an auxiliary loss punishing imbalance. But that loss competes with the language-modeling objective: you're paying quality to buy balance.

V3's fix is almost cheeky: auxiliary-loss-free balancing. Each expert carries a bias term used only for routing — not in the gradient, not in the output weights. Overloaded expert? Bias nudged down, fewer tokens routed there. Underused? Nudged up. Balance emerges from a feedback controller outside the training objective, so the model never trades quality for it.

Standard training predicts one next token per position. V3 adds a lightweight module predicting the token after next as well — multi-token prediction (MTP). During training this densifies the learning signal (every position teaches two lessons) and pushes representations to plan a little further ahead.

At inference the MTP head gets a second life: it drafts the next-next token for speculative decoding, with ~85% acceptance — meaningfully faster generation, using machinery training already paid for. One module, two dividends.

Most labs trained in BF16. V3 ran the bulk of its matrix math in FP8 — half the memory traffic, nearly double the arithmetic throughput on H800s — something widely considered too unstable at frontier scale. The stabilizers: fine-grained scaling (per-tile/per-block scale factors so outliers don't wreck a whole tensor) and high-precision accumulation (sums collect in FP32 before rounding back down).

Sensitive pieces — embeddings, norms, router, optimizer states — stay in higher precision. The headline stat that convinced everyone: zero irrecoverable loss spikes across the entire 14.8T-token run. Add DualPipe (overlapping the MoE's heavy cross-GPU communication behind compute) and the hardware efficiency story is complete.

The number needs its caveats — it excludes research, ablations, failed runs, salaries and the cluster itself — but the like-for-like comparison still landed like a thunderclap: contemporary frontier runs were estimated at 10–50× more compute spend. The efficiency stack above, compounding multiplicatively, is where the order of magnitude came from.

V3 is the paper that made training efficiency a first-class research axis — after it, "what does your compute buy?" became as important a question as "how much compute do you have?" It's also the foundation everything DeepSeek-famous stands on: R1's reasoning RL runs on the V3 base, and the GRPO lineage got its frontier-scale proving ground here.

Architecturally, its ideas went mainstream fast: MLA-style cache compression, aux-loss-free balancing and MTP heads now appear across open models — including the next entry in this library, Kimi K2, which scales the same skeleton to a trillion parameters. And strategically, V3 + R1 reset the world's assumptions about who can afford to build frontier AI. The moat, it turned out, was partly an engineering culture.