KV Cache
also: key-value cache
The stored attention state that lets a model generate each new token without re-reading the whole prompt.
During generation, a Transformer reuses the keys and values it computed for all previous tokens instead of recomputing them per new token. That store is the KV cache. It converts quadratic re-work into linear generation — but it eats GPU memory in proportion to context length, which is why KV-cache memory, not compute, usually caps how many users fit on one GPU, and why long chats cost more.
Worked example: a 7B model serving a 4,000-token context in fp16 holds a few hundred MB of keys and values per request — multiply by concurrent requests and it is VRAM, not compute, that runs out first. Gotcha: the cache grows with every generated token, so long outputs quietly eat memory mid-request; paging (vLLM), eviction, or shorter contexts are the levers.