DeepSeek-V4,
explained.

Rung 3 makes the feed-forward layers sparse and compresses what attention stores. It leaves the attention computation itself dense: every token still attends to every earlier token. At a context of a million tokens that is the dominant cost, so this rung's change is to make attention sparse too — and to do it with two complementary moves rather than one.

The first is compression: store each past token in a smaller form (a compact latent, as in V3's multi-head latent attention), so the long history costs far less to keep and read. The second is selection: pick a limited set of positions that this query will attend to at full fidelity. Taken together the pattern is content-dependent rather than fixed, which is the difference from a sliding window — the model chooses where to look rather than being restricted to its neighbourhood, so long-range interaction survives the sparsification instead of having to be composed through depth.

Stated in rung 3's units, as the page carries them: 1.6T total parameters with roughly 49B active, up from 671B and 37B — the decoupling ratio widening again — serving a 1M-token context at a reported fraction of V3's cost per token. The other reported change is to post-training rather than architecture: on-policy distillation in place of large-scale reinforcement learning, which means training the student on its own outputs corrected by a stronger teacher rather than on a reward signal — cheaper and more stable than an RL loop, and a direct alternative to what the reasoning ladder does.

V4 takes the mixture-of-experts lesson of V3 and pushes the ratio harder: roughly 1.6T total parameters, of which only about 49B activate per token — near 3%. The store of knowledge more than doubles; the per-token compute barely moves. That is the whole point of a sparse MoE, and V3 already proved it trains and serves.

But a bigger brain is the easy part. The expensive part of a modern model is not the feed-forward experts — it's attention at long context. Double the parameters and you double a cost you already know how to shard. Take context from 128k to 1M and you hit a wall that grows with the square of the sequence. V4's real work is that wall, and the next three sections are how it comes down.

Attention costs you in two currencies. Memory: the KV cache stores a key and value for every token you've seen, so it grows linearly with context and batch. Compute: every new query scores itself against every stored key, so attention FLOPs grow with the square of the sequence. V4's hybrid attention attacks both at once, and the trick is that the two savings multiply.

Lever one shrinks the memory per token (a descendant of V3's multi-head latent attention). Lever two shrinks the number of tokens scored (only the most relevant blocks of the past, not all of it). Neither alone gets you to a cheap million-token context; together they compound. The two sections below take them one at a time, then the runnable version proves the multiplication.

V3 introduced multi-head latent attention — cache one small latent vector per token instead of full per-head keys and values, and reconstruct the heads at use time. V4 keeps that idea and leans on it harder, because at a million tokens the cache is the serving cost.

The saving is countable and, crucially, it does not depend on how long the context gets — you compress each token the same way whether the sequence is a thousand tokens or a million:

A latent of 512 against a 4096-wide K/V is an 8× smaller cache — the same 8× at 1k tokens and at 1M. The runnable below checks that invariance.

Dense attention has every query score every past key — that squared term. But most of that scoring is wasted: for any given query, the vast majority of a million past tokens are irrelevant. Compressed sparse attention chops the past into fixed blocks and, per query, selects only the top-k blocks worth scoring. Below the budget nothing is dropped; above it, the work stops growing with the sequence.

Once n > k·b, the scored-key count freezes at k·b while a dense model keeps paying n. Compression's d_c/width then multiplies on top.

R1 earned its reasoning with large-scale reinforcement learning: sample answers, score them with a verifiable reward, push the policy up the gradient. It works, but RL at frontier scale is expensive and finicky — reward design, instability, compute. V4's report swaps in on-policy distillation.

The student generates its own answers — on-policy, in its own distribution — and a stronger teacher grades or corrects those exact trajectories. The signal is the teacher's judgement of the student's own work, not a scalar reward feeding a policy-gradient update. You keep the on-policy virtue that made RL good (train on the states the model actually visits) while dropping the reward machinery that made it hard.

These are the figures this page reports (it cites no primary source, so they cannot be checked here), and the mechanism above is why they're plausible: a per-token cache discount that holds at any length, times a scored-key count that stops growing once the sequence outruns the selection budget. The runnable model below reproduces the shape of those savings from the two levers — a mechanism you can change and re-measure, not a benchmark to take on faith.

V3 made training efficiency a first-class axis. V4 does the same for serving at long context — the cost that actually bites once agents read whole codebases and month-long histories. Its hybrid of compression and selection is the shape the whole field is converging on, and it pairs naturally with an explicit memory layer like Titans-style surprise memory for what should persist beyond the window.

The training story matters too: if a teacher grading a student's own trajectories can stand in for a full RL loop, the reasoning recipe that R1 made famous gets cheaper and steadier to reproduce. Read alongside Kimi K2, the sibling trillion-scale open model, V4 is the clearest sign that the frontier's real competition is now efficiency engineering — cache, sparsity, and cheaper post-training — not raw parameter count.

This page has no primary source to check against, so the V4 figures it quotes are reported, not verified. What can be described with confidence is the mechanism family the page attributes to V4, because its published ancestors are documented: multi-head latent attention from DeepSeek-V2 and V3; Native Sparse Attention (NSA), from researchers at DeepSeek and Peking University in 2025; and DeepSeek Sparse Attention (DSA), introduced with DeepSeek-V3.2-Exp later in 2025. For one query, an attention layer that compresses and then selects works like this.

1. Cache compressed tokens. As in V3's multi-head latent attention, each past token is stored as one small latent vector instead of full per-head keys and values, and the heads are reconstructed from it when needed. Memory per token shrinks by the same factor at every context length.
2. Group the past into blocks. The cached sequence is cut into fixed-size blocks of consecutive tokens, the unit of selection.
3. Score cheaply. A lightweight scorer rates how relevant each part of the past is to the current query. In NSA the scores come from attention over compressed block representations, so the coarse view doubles as the selector. In DSA a separate lightning indexer, a few small heads run in FP8, scores individual past tokens.
4. Select the top k. Keep the k highest-scoring blocks; NSA also always keeps a sliding window of the most recent tokens, which are almost always relevant. Everything else is skipped by the expensive attention.
5. Attend at full fidelity to the selection. Ordinary softmax attention runs over the selected positions only, so the cost per query is set by the selection budget rather than by the context length.
6. Combine branches. NSA mixes its compressed, selected and sliding-window outputs with learned gates. The page describes V4's design as two levers, compression and selection, without a gating detail.
7. Train through the top-k. Selection is a hard choice with no gradient, so the scorer must learn some other way. NSA derives its selection scores from the compressed branch, which the language-model loss trains anyway. DSA trains its indexer with a separate KL-divergence loss that pulls the indexer's scores toward the main attention's distribution: first in a short dense warm-up with the rest of the model frozen, then during sparse training, with the indexer's input detached so that loss never reaches the main model.
8. Read memory in blocks. Fetching whole contiguous blocks, shared by the heads that share keys, keeps memory access efficient; positions scattered one by one across a million-token cache would waste bandwidth. NSA's kernels are designed around this.

The post-training change

The page also says V4 replaces large-scale reinforcement learning with on-policy distillation. The general method is documented: the student generates its own answers, a stronger teacher scores every token of those answers with its own next-token probabilities, and the student is trained to reduce the divergence between the two, typically the reverse KL. Generalized knowledge distillation (Agarwal and colleagues, 2024) and MiniLLM (Gu and colleagues, 2024) are two published versions.

• qt — the current query; score — the cheap relevance estimate for each block; k — the number of blocks kept; St — the positions inside the kept blocks, plus any window that is always kept.
• ks, vs — keys and values reconstructed from the cached latents, for positions in S_t only; d — the head dimension.
• πθ — the student; πT — the teacher; y ∼ πθ — sequences sampled from the student itself, which is what makes the method on-policy.
• KL(πθ ‖ πT) — the reverse KL, which is mode-seeking: the student is pushed to put its probability where the teacher puts most of its own, rather than spreading over everything the teacher allows.

These are the standard forms of each idea; the page does not give V4's exact scorer or loss. At the extremes: if k covers every block, S_t is the whole past and the layer is dense attention again. If the student already matches the teacher on every token, the loss is zero. Unlike a reinforcement-learning reward, which arrives once per answer, this objective gives a signal at every token.

There is no primary source behind this page, so the V4 entries below are reported claims to verify, not results.

The design's ancestors do have published evidence. The NSA paper trained a 27B-parameter model from scratch and reported quality matching or exceeding full attention on general benchmarks, long-context tests and chain-of-thought reasoning, with large speedups at 64K tokens in decoding and in the forward and backward passes. DeepSeek-V3.2-Exp, which added DSA to V3.1-Terminus, reported benchmark results roughly on par with the dense-attention model at substantially lower long-context cost.

Negative results and missing evidence

The NSA authors report a negative result that matters for any selector: in their pilot runs, alternatives that trained block selection with a separate auxiliary loss, or chose tokens by clustering keys, reached a worse loss than NSA's design or ran slower. Selection is a bet that the model can tell in advance which positions matter. What would test that bet for V4 is long-context evaluation that requires aggregating information scattered across the whole context, rather than retrieving it from a few places, and none is cited here.

This is recent and the numbers here are vendor-reported, so the appropriate reading is that the architecture's direction is more informative than its benchmark table. Treat the parameter counts and the cost ratios as claims from the party with an interest in them until independent evaluation exists.

The direction is nonetheless the one the whole site has been pointing at: after four rungs, every major axis of a frontier model is now conditional — which experts run, what the cache stores, which positions attention reads. That is a coherent end state for the idea rung 1 introduced, and worth noticing as a shape even if the specific model's figures move.

What would settle it: whether the selected-attention pattern holds up on tasks that need to aggregate across a long context rather than retrieve from it. Selection is a bet that the model can tell in advance which positions matter, and that bet is safest when the answer lives in a few places — which is also the regime long-context benchmarks mostly measure, as the rung above on the context ladder argues. Independent reproduction of the on-policy distillation claim matters too, because "cheaper than RL and as good" is a large claim about the most expensive stage of modern post-training.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Sparsity: route each token to a few experts and buy parameter count without paying for it at inference. 2017 idea, 2026 frontier default.

1. 1Mixture of Experts2017
2. 2Switch Transformer2022
3. 3DeepSeek-V32024
4. 4DeepSeek-V42026
5. 5Qwen4-Coder2026

All 110 papers, in 19 ladders →