DeepSeek-V3,
explained.

Rungs 1 and 2 keep experts balanced with an auxiliary loss, and both admit it is a term pulling against the language objective. This rung's first change is to remove it: balance is maintained by adjusting a per-expert bias on the routing scores, nudged up for underused experts and down for overused ones, so the pressure is applied to the decision rather than added to the loss. The gradient the model learns from is then only about language. That is a small idea with a clean argument behind it, and it is the rung's most portable contribution.

The second change targets the cost that actually dominates serving rather than training. A transformer's key-value cache grows with every token in every sequence, and at long context and high concurrency it, not the weights, is what fills the accelerator's memory. The change is to project keys and values down into a much smaller latent representation and cache that instead, reconstructing what attention needs on the fly — trading a little arithmetic for a large reduction in the thing that limits how many users you can serve at once.

Stated in the ladder's units, and these are the numbers the page carries: 671B total parameters with about 37B active per token — a roughly 18× decoupling, which is rung 1's identity taken much further than rung 2 did — trained in FP8 rather than a 16-bit format, with an auxiliary multi-token prediction objective, for a reported training compute cost around $5.6M.

V3 is a mixture-of-experts pushed toward its logical extreme: 671B total parameters, organized as many small experts, with a router activating only ~37B per token — about 5%. You store the knowledge of a giant; you pay the compute of a mid-size model. Each token consults 8 routed experts plus one shared expert that every token sees.

Fine-grained experts (many small ones rather than few big ones) give the router more precise choices — specialization without redundancy. The blueprint came from DeepSeek's own MoE work; V3's contribution was making this shape trainable and servable at 671B without the usual MoE taxes. Those taxes are what the next two sections remove.

Serving throughput dies on the KV cache — the per-token attention state that grows with context and batch. Standard fixes (GQA, MQA) share keys and values across heads: less memory, some quality loss. V3's multi-head latent attention goes further: compress keys and values into one small latent vector, cache only that, and reconstruct full per-head K/V from it at use time.

The result: KV memory an order of magnitude smaller than vanilla attention — longer contexts, bigger serving batches, cheaper tokens — with benchmark quality that matches or beats standard attention. It's the rare compression that costs nothing you can measure.

The size difference is countable. Standard attention caches K and V for every head; GQA shares them across a few groups; MLA caches only the latent (plus a tiny decoupled RoPE key):

That's 57× smaller than MHA — a 128k-token cache drops from ~477 GB to ~8 GB. The runnable version below computes it for any config.

MoE's classic pathology: the router collapses onto a few favorite experts — they overload, the rest atrophy. The standard cure is an auxiliary loss punishing imbalance. But that loss competes with the language-modeling objective: you're paying quality to buy balance.

V3's fix is almost cheeky: auxiliary-loss-free balancing. Each expert carries a bias term used only for routing — not in the gradient, not in the output weights. Overloaded expert? Bias nudged down, fewer tokens routed there. Underused? Nudged up. Balance emerges from a feedback controller outside the training objective, so the model never trades quality for it.

Standard training predicts one next token per position. V3 adds a lightweight module predicting the token after next as well — multi-token prediction (MTP). During training this densifies the learning signal (every position teaches two lessons) and pushes representations to plan a little further ahead.

At inference the MTP head gets a second life: it drafts the next-next token for speculative decoding, with ~85% acceptance — meaningfully faster generation, using machinery training already paid for. One module, two dividends.

Most labs trained in BF16. V3 ran the bulk of its matrix math in FP8 — half the memory traffic, nearly double the arithmetic throughput on H800s — something widely considered too unstable at frontier scale. The stabilizers: fine-grained scaling (per-tile/per-block scale factors so outliers don't wreck a whole tensor) and high-precision accumulation (sums collect in FP32 before rounding back down).

Sensitive pieces — embeddings, norms, router, optimizer states — stay in higher precision. The headline stat that convinced everyone: zero irrecoverable loss spikes across the entire 14.8T-token run. Add DualPipe (overlapping the MoE's heavy cross-GPU communication behind compute) and the hardware efficiency story is complete.

The number needs its caveats — it excludes research, ablations, failed runs, salaries and the cluster itself — but the like-for-like comparison still landed like a thunderclap: contemporary frontier runs were estimated at 10–50× more compute spend. The efficiency stack above, compounding multiplicatively, is where the order of magnitude came from.

V3 is the paper that made training efficiency a first-class research axis — after it, "what does your compute buy?" became as important a question as "how much compute do you have?" It's also the foundation everything DeepSeek-famous stands on: R1's reasoning RL runs on the V3 base, and the GRPO lineage got its frontier-scale proving ground here.

Architecturally, its ideas went mainstream fast: MLA-style cache compression, aux-loss-free balancing and MTP heads now appear across open models — including the next entry in this library, Kimi K2, which scales the same skeleton to a trillion parameters. And strategically, V3 + R1 reset the world's assumptions about who can afford to build frontier AI. The moat, it turned out, was partly an engineering culture.

The technical report is a stack of mostly independent choices. In the order a token, and the training run, meets them:

1. Prepare 14.8T tokens. A corpus richer in math and code than DeepSeek-V2's, with more languages beyond English and Chinese, tokenized with a byte-level BPE of 128K entries. About 10% of documents are rearranged for fill-in-the-middle training, prefix and suffix first and the middle to predict, which teaches code infilling without hurting ordinary next-token prediction.
2. Stack 61 layers. The hidden size is 7,168. Attention is multi-head latent attention with 128 heads of size 128: keys and values are cached as a 512-dimensional latent plus a 64-dimensional decoupled key that carries rotary position information, and queries are compressed too, to 1,536 dimensions, to save activation memory in training.
3. Make the feed-forward layers MoE. Every feed-forward layer except the first three has 1 shared expert and 256 routed experts, each with an intermediate size of 2,048. A token uses the shared expert plus 8 routed ones. That yields 671B parameters in total and about 37B per token.
4. Route with sigmoid affinities and a bias. Each expert's affinity is a sigmoid of the token's dot product with that expert's centroid. A per-expert bias is added only when choosing the top 8; the gate weights that scale the chosen experts' outputs use the unbiased affinities, normalized over the 8. After each step an overloaded expert's bias falls by γ and an underloaded one's rises by γ, with γ = 0.001 for most of training and 0 for the last 500B tokens. A sequence-level balance loss with a tiny weight, 0.0001, guards against extreme imbalance inside a single sequence.
5. Limit each token to four nodes. The routed experts are spread over 8 nodes. For each token, nodes are ranked by the sum of the highest affinities among the experts they hold, and its 8 experts are picked only from the top 4 nodes, which caps the cross-node traffic each token creates. With balance maintained this way, no tokens are dropped in training or at inference.
6. Add one multi-token prediction module. Beside the main next-token loss, a sequential module predicts the token after next. It combines the main model's final hidden state at position i with the embedding of token i+1, passes them through one Transformer block, and predicts token i+2 through the shared output head. Its loss is weighted 0.3 for the first 10T tokens and 0.1 after that.
7. Train in FP8 with fine-grained scaling. Most matrix multiplies take FP8 (E4M3) inputs, with one scale factor per 1×128 tile of activations and per 128×128 block of weights, and partial sums are promoted to FP32 every 128 elements. The embedding, output head, gating, normalization and attention stay in BF16 or FP32, as do master weights and optimizer states.
8. Parallelize without tensor parallelism. On 2,048 H800 GPUs: 16-way pipeline parallelism with the DualPipe schedule, which overlaps the computation of paired micro-batches with their all-to-all and pipeline communication; 64-way expert parallelism across 8 nodes; and ZeRO-1 data parallelism.
9. Optimize. AdamW with β₁ 0.9, β₂ 0.95 and weight decay 0.1, on 4K-token sequences. The learning rate warms to 2.2 × 10^⁻⁴ over 2K steps, holds until 10T tokens, then decays on a cosine; the batch ramps from 3,072 to 15,360 sequences over the first 469B tokens.
10. Extend context, then post-train. Two YaRN stages stretch the context from 4K to 32K and then to 128K. Post-training is supervised fine-tuning on 1.5M examples, with reasoning data distilled from internal DeepSeek-R1 models and other data written by DeepSeek-V2.5 and checked by people, followed by reinforcement learning with GRPO, using rule-based rewards where an answer can be checked and a reward model elsewhere.

• ut — token t's input to the MoE layer; ei — expert i's centroid; σ — the sigmoid, so each affinity is computed independently rather than competing in a softmax.
• bi — the balancing bias. It decides which experts are chosen and never how much they count, and it moves by ±γ after each step according to observed load, not by gradient.
• Kr = 8 routed experts; their normalized weights sum to 1, and the shared expert FFN^^(s) is always added at full weight.
• λ — the MTP weight, 0.3 and then 0.1; LMTP^k — cross-entropy for predicting the token k+1 positions ahead at depth k.

At the extremes: with γ = 0 the biases freeze and routing follows affinity alone, and a very large γ would let load chase content out of the routing decision. With λ = 0, training reduces to ordinary next-token prediction. Because the bias never touches the gate weights, the language-modeling gradient is the same whether balance is good or bad.

The base model is compared with leading open models, including Qwen2.5 72B and Llama-3.1 405B; the chat model also with GPT-4o and Claude-3.5-Sonnet. Ablations were run on smaller MoE baselines, not on V3 itself.

V3 was the strongest open model on most of these at release. Against the closed models it led clearly on the math benchmarks, MATH-500 and AIME 2024, and trailed Claude-3.5-Sonnet on SWE-bench Verified and GPQA-Diamond.

The report also states its limits. The recommended unit for deploying V3 efficiently is large, a real burden for small teams, and generation speed still has room to improve. A comparison of batch-level and sequence-level balancing found similar loss, with batch-level balance risking imbalance within small batches and under domain shift at inference. And its cost figure covers only the final training run, not the research, ablations and failed runs that found the recipe.

The cost figure is the contested part, and the contest is about scope rather than honesty. It refers to the final training run priced at a market rate for GPU hours. It does not include the research that found the recipe, the failed runs, the ablations, the data pipeline, the salaries, or the capital cost of owning the cluster. The report is reasonably clear about what it is measuring; a great deal of the commentary that followed was not, and treated it as the total cost of producing a frontier model. Both the number and its scope are worth carrying, because the pair is the actual lesson.

What is not seriously disputed is that the efficiency work is real and that the resulting model was competitive with far more expensively-trained ones — which is a statement about how much headroom remained in training efficiency, and therefore about how much of prior spending was buying capability versus paying for inexperience.

How it held up: the two mechanisms were adopted quickly. Loss-free load balancing is a clean improvement with no obvious downside, and latent compression of the key-value cache addressed the constraint every serving operator actually feels — the same constraint paged attention attacks from the memory-management side rather than the representation side. Its largest effect was on expectations: after this report, a frontier-class result from a mid-sized budget stopped being assumed impossible.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Sparsity: route each token to a few experts and buy parameter count without paying for it at inference. 2017 idea, 2026 frontier default.

1. 1Mixture of Experts2017
2. 2Switch Transformer2022
3. 3DeepSeek-V32024
4. 4DeepSeek-V42026
5. 5Qwen4-Coder2026

All 110 papers, in 19 ladders →