KV Cache
also: key-value cache
The stored attention state that lets a model generate each new token without re-reading the whole prompt.
During generation, a Transformer reuses the keys and values it computed for all previous tokens instead of recomputing them per new token. That store is the KV cache. It converts quadratic re-work into linear generation — but it eats GPU memory in proportion to context length, which is why KV-cache memory, not compute, usually caps how many users fit on one GPU, and why long chats cost more.