DeepSeek-V4,
explained.

V4 takes the mixture-of-experts lesson of V3 and pushes the ratio harder: roughly 1.6T total parameters, of which only about 49B activate per token — near 3%. The store of knowledge more than doubles; the per-token compute barely moves. That is the whole point of a sparse MoE, and V3 already proved it trains and serves.

But a bigger brain is the easy part. The expensive part of a modern model is not the feed-forward experts — it's attention at long context. Double the parameters and you double a cost you already know how to shard. Take context from 128k to 1M and you hit a wall that grows with the square of the sequence. V4's real work is that wall, and the next three sections are how it comes down.

Attention costs you in two currencies. Memory: the KV cache stores a key and value for every token you've seen, so it grows linearly with context and batch. Compute: every new query scores itself against every stored key, so attention FLOPs grow with the square of the sequence. V4's hybrid attention attacks both at once, and the trick is that the two savings multiply.

Lever one shrinks the memory per token (a descendant of V3's multi-head latent attention). Lever two shrinks the number of tokens scored (only the most relevant blocks of the past, not all of it). Neither alone gets you to a cheap million-token context; together they compound. The two sections below take them one at a time, then the runnable version proves the multiplication.

V3 introduced multi-head latent attention — cache one small latent vector per token instead of full per-head keys and values, and reconstruct the heads at use time. V4 keeps that idea and leans on it harder, because at a million tokens the cache is the serving cost.

The saving is countable and, crucially, it does not depend on how long the context gets — you compress each token the same way whether the sequence is a thousand tokens or a million:

A latent of 512 against a 4096-wide K/V is an 8× smaller cache — the same 8× at 1k tokens and at 1M. The runnable below checks that invariance.

Dense attention has every query score every past key — that squared term. But most of that scoring is wasted: for any given query, the vast majority of a million past tokens are irrelevant. Compressed sparse attention chops the past into fixed blocks and, per query, selects only the top-k blocks worth scoring. Below the budget nothing is dropped; above it, the work stops growing with the sequence.

Once n > k·b, the scored-key count freezes at k·b while a dense model keeps paying n. Compression's d_c/width then multiplies on top.

R1 earned its reasoning with large-scale reinforcement learning: sample answers, score them with a verifiable reward, push the policy up the gradient. It works, but RL at frontier scale is expensive and finicky — reward design, instability, compute. V4's report swaps in on-policy distillation.

The student generates its own answers — on-policy, in its own distribution — and a stronger teacher grades or corrects those exact trajectories. The signal is the teacher's judgement of the student's own work, not a scalar reward feeding a policy-gradient update. You keep the on-policy virtue that made RL good (train on the states the model actually visits) while dropping the reward machinery that made it hard.

These are the technical report's headline figures, and the mechanism above is why they're plausible: a per-token cache discount that holds at any length, times a scored-key count that stops growing once the sequence outruns the selection budget. The runnable model below reproduces the shape of those savings from the two levers — a mechanism you can change and re-measure, not a benchmark to take on faith.

V3 made training efficiency a first-class axis. V4 does the same for serving at long context — the cost that actually bites once agents read whole codebases and month-long histories. Its hybrid of compression and selection is the shape the whole field is converging on, and it pairs naturally with an explicit memory layer like Titans-style surprise memory for what should persist beyond the window.

The training story matters too: if a teacher grading a student's own trajectories can stand in for a full RL loop, the reasoning recipe that R1 made famous gets cheaper and steadier to reproduce. Read alongside Kimi K2, the sibling trillion-scale open model, V4 is the clearest sign that the frontier's real competition is now efficiency engineering — cache, sparsity, and cheaper post-training — not raw parameter count.