LABS · 51  /  ATTENTION

The million-token bill.

Ask a model to read a whole codebase and dense attention has to weigh every token against every other — memory grows with the context, compute grows with its square. At a million tokens that bill is the whole cost of serving. Pull two levers — compress each token's key/value, and select only the blocks that matter — and watch it collapse.

Act 1 · The wall
Every token, weighed against all of them
Dense attention caches a key/value for every token, and each new query scores against all of them. Add tokens and the cache grows linearly while the scoring grows with the square. Watch the keys pile up — every one is stored and scored.
KV cache — every token stored & scored (dense)
0
Tokens (context n)
0
KV keys cached
0
Score ops (∝ n²)
KV grows as n, scoring as . Scale n from 8 to a million and both numbers explode — that's the wall a long-context model hits. The next act attacks both.
Act 2 · The two levers
Compress every token, score only a few blocks
Lever one — compress: cache a small latent per token instead of the full width (yellow). Lever two — select: for this query, score only the top-k blocks (green), skip the rest (dimmed). The dense row scores 32 keys; the sparse row scores far fewer, at a fraction of the memory each.
Dense — score all 32 keys, full width each
Sparse — compressed keys, top-2 blocks selected
Green = the blocks this query actually scores; dimmed = skipped. Each stored key is a compressed latent, not the full width — so you pay less memory per key and score fewer of them. Two discounts, stacked.
Act 3 · Scale it to a million
Where the two levers multiply
KV ratio is d_c / width — a fixed per-token discount. FLOPs ratio is (scored keys / n) × (d_c / width) — the selection ratio keeps shrinking as context grows, and the two multiply. Drag the context to a million and watch both ratios fall.
Context length n1,000,000
Latent width d_c (of 4096)512
Blocks kept k (× block 256)64
KV cache vs dense
Attn FLOPs vs dense
Scored keys (min n, k·b)

Two currencies, two levers

Attention costs you twice. Memory: the KV cache stores a key and value for every token, so it grows linearly with context and batch. Compute: every query scores against every stored key, so attention FLOPs grow with the square of the sequence. At a million tokens both are the dominant serving cost — this is the wall long-context models hit.

Compressed sparse attention (the shape in DeepSeek-V4) attacks both, and the trick is that the two savings multiply. Compression is a constant discount on every token; selection is a growing discount as context lengthens.

The math

KV ratio = d_c / width (per-token, context-invariant). Scored keys = min(n, k·b) — once n > k·b the count freezes while dense keeps paying n. Attention-FLOPs ratio = (min(n,k·b) / n) × (d_c / width) — the product of the two levers. A fixed 8× compression times a selection ratio that keeps shrinking lands the total far below the dense curve.

Doesn't selection lose information?

It bets that for any query, only a small slice of a million past tokens is actually relevant — so scoring the top-k blocks keeps what matters and skips the rest. Below the budget nothing is dropped; above it, the work stops growing with the sequence. It's an approximation, but a well-chosen one, which is why quality holds. Contrast this with the KV-cache visualizer, where the cache grows dense and unbounded.

Compression alone isn't new

The compression lever descends from multi-head latent attention (cache a small latent, reconstruct heads at use time). What compressed sparse attention adds is the second lever — periodic KV compression plus top-k block selection — so the discounts stack instead of competing. Read the full breakdown in DeepSeek-V4.

Check yourself

Four questions. Don't attend to all of them — just the relevant ones.

Finished this one? 0 / 51 Labs done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Labs