Ask a model to read a whole codebase and dense attention has to weigh everytoken against every other — memory grows with the context, compute grows with its square. At a million tokens that bill is the whole cost of serving. Pull two levers — compress each token's key/value, and select only the blocks that matter — and watch it collapse.
Act 1 · The wall
Every token, weighed against all of them
Dense attention caches a key/value for every token, and each new query scores against all of them. Add tokens and the cache grows linearly while the scoring grows with the square. Watch the keys pile up — every one is stored and scored.
KV grows as n, scoring as n². Scale n from 8 to a million and both numbers explode — that's the wall a long-context model hits. The next act attacks both.
Act 2 · The two levers
Compress every token, score only a few blocks
Lever one — compress: cache a small latent per token instead of the full width (yellow). Lever two — select: for this query, score only the top-k blocks (green), skip the rest (dimmed). The dense row scores 32 keys; the sparse row scores far fewer, at a fraction of the memory each.
Dense — score all 32 keys, full width each
Sparse — compressed keys, top-2 blocks selected
Green = the blocks this query actually scores; dimmed = skipped. Each stored key is a compressed latent, not the full width — so you pay less memory per key and score fewer of them. Two discounts, stacked.
Act 3 · Scale it to a million
Where the two levers multiply
KV ratio is d_c / width — a fixed per-token discount. FLOPs ratio is (scored keys / n) × (d_c / width) — the selection ratio keeps shrinking as context grows, and the two multiply. Drag the context to a million and watch both ratios fall.
Context length n1,000,000
Latent width d_c (of 4096)512
Blocks kept k (× block 256)64
—
KV cache vs dense
—
Attn FLOPs vs dense
—
Scored keys (min n, k·b)
Two currencies, two levers
Attention costs you twice. Memory: the KV cache stores a key and value for every token, so it grows linearly with context and batch. Compute: every query scores against every stored key, so attention FLOPs grow with the square of the sequence. At a million tokens both are the dominant serving cost — this is the wall long-context models hit.
Compressed sparse attention (the shape in DeepSeek-V4) attacks both, and the trick is that the two savings multiply. Compression is a constant discount on every token; selection is a growing discount as context lengthens.
The math
KV ratio = d_c / width (per-token, context-invariant). Scored keys = min(n, k·b) — once n > k·b the count freezes while dense keeps paying n. Attention-FLOPs ratio = (min(n,k·b) / n) × (d_c / width) — the product of the two levers. A fixed 8× compression times a selection ratio that keeps shrinking lands the total far below the dense curve.
Doesn't selection lose information?
It bets that for any query, only a small slice of a million past tokens is actually relevant — so scoring the top-k blocks keeps what matters and skips the rest. Below the budget nothing is dropped; above it, the work stops growing with the sequence. It's an approximation, but a well-chosen one, which is why quality holds. Contrast this with the KV-cache visualizer, where the cache grows dense and unbounded.
Compression alone isn't new
The compression lever descends from multi-head latent attention (cache a small latent, reconstruct heads at use time). What compressed sparse attention adds is the second lever — periodic KV compression plus top-k block selection — so the discounts stack instead of competing. Read the full breakdown in DeepSeek-V4.
Sparse attention, step by step
One decode step under compressed sparse attention, and the Act 3 arithmetic at the widget's defaults: a million tokens, a 512-wide latent against a 4,096-wide key, and 64 blocks of 256 kept.
One query, four moves
Compress. As tokens arrive, store each one's key and value as a small latent of width d_c instead of the full width. This is a fixed discount per token.
Group. Cut the stored sequence into blocks of b tokens (256 here) and keep one summary per block.
Select. Score the query against the block summaries, a list n/b long, and keep the top k blocks.
Attend. Run ordinary softmax attention over only the tokens in those blocks, at most k·b keys.
Worked example: one million tokens
Dense baseline. One query scores 1,000,000 keys; scoring the whole sequence is about n²/2 = 5 × 10¹¹ query-key pairs per head per layer, and the cache holds all 1,000,000 keys at full width.
Selection alone. k·b = 64 × 256 = 16,384 scored keys, a ratio of 16,384 / 1,000,000 = 1.64%, about 61 times fewer. Of the roughly 3,906 blocks, 64 are kept.
Compression alone. d_c / width = 512 / 4096 = 0.125, so the cache is 8 times smaller: if a key and a value are each 4,096 wide in FP16 that is 16.4 GB per layer for a million tokens, down to 2.05 GB.
Together. 0.0164 × 0.125 = 0.00205 of dense attention work, about 488 times less.
The selection ratio is not constant. It is 100% up to 16,384 tokens, 25% at 65,536, 6.25% at 262,144 and 1.6% at a million, so the combined ratio runs 12.5%, 3.1%, 0.78% and 0.20%. The savings grow with context, which is the point.
Sliding the knobs. Cutting k to 16 scores 4,096 keys, 0.05% of dense work. Doubling d_c to 1,024 halves the cache saving to 4× and gives 244×.
Knobs and pitfalls
The cache still grows linearly, so memory capacity matters even when compute does not. Selection is an approximation: a small k can miss a needed block, so test long-range retrieval tasks rather than perplexity alone. Training must match inference, since a model trained with dense attention does not tolerate selection added afterward. Sparse patterns only pay off with kernels that read blocks contiguously, which is why block size is a hardware choice as much as a modeling one.
Check yourself
Four questions. Don't attend to all of them — just the relevant ones.
See every answer, with the reasoning
Why is dense attention expensive at a million tokens?
Answer: KV memory grows linearly (n) and attention compute grows with the square (n²) of the context — Store a key/value per token, and score every query against every key — both explode with length.
What are the two levers of compressed sparse attention?
Answer: Compress each token's K/V to a latent, and select only the top-k blocks per query — Compression shrinks memory per token; selection shrinks how many tokens are scored.
Why do the two savings multiply?
Answer: FLOPs scale with scored-keys × per-key work, so the ratio is (scored/n) × (d_c/width) — A fixed compression ratio times a shrinking selection ratio lands far below the dense curve.
What happens to scored keys once n exceeds the selection budget k·b?
Answer: They freeze at k·b while a dense model keeps paying n — Above the budget, selection caps the work — that's the growing discount as context lengthens.