PagedAttention
Managing the KV-cache in fixed-size, non-contiguous memory pages — like an OS managing virtual memory — instead of one big reserved block per request.
A naive KV-cache implementation reserves one large contiguous memory block per request sized for the worst case, wasting huge amounts of GPU memory to fragmentation. PagedAttention (from the vLLM paper) borrows the operating-system idea of paging: the KV-cache is split into small fixed-size blocks allocated on demand and referenced indirectly, eliminating fragmentation and letting a GPU serve far more concurrent sequences from the same memory.