vLLM vs TGI vs SGLang: choosing an LLM serving engine

vLLM, TGI (Text Generation Inference), and SGLang are all open-source LLM serving engines: they take model weights and turn them into a high-throughput, batched inference server. They all do continuous batching — dynamically packing in-flight requests onto the GPU — so on that axis they rhyme. Where they split is their signature approach to the KV cache, the per-request memory of attention keys and values that dominates GPU memory and, therefore, how many requests you can serve at once.

PagedAttention (vLLM’s idea) treats the KV cache like virtual memory: instead of one contiguous block per request — which fragments and wastes memory as sequences grow and finish — it stores the cache in fixed-size pages, so memory is packed with almost no waste. More usable KV memory means larger effective batches, which means higher throughput. It is the reason vLLM became the default.

RadixAttention (SGLang’s idea) attacks a different waste: when many requests share a prefix — the same system prompt, the same few-shot examples, the same conversation so far — recomputing and re-storing that prefix’s KV per request is pure duplication. SGLang keeps a radix tree of KV caches and shares the common prefix across requests, so it is computed once and reused. On prefix-heavy workloads (agents, few-shot, multi-turn) that is an enormous win the paged approach alone does not capture.

Benchmarks that crown one engine are almost always measuring one workload. On generic, low-overlap traffic, vLLM’s paged memory management makes it a throughput leader. On workloads where requests share big prefixes, SGLang’s prefix reuse can pull decisively ahead — because it is skipping work the others repeat. TGI, meanwhile, optimizes for being a robust, well-integrated production server, which is a different objective than topping a synthetic throughput chart.

So the honest framing is not "X is fastest" but "fastest for what." The variables that flip the result are prefix sharing (favors SGLang), model and hardware coverage (favors vLLM’s breadth), and how much you value production ergonomics and HF integration (favors TGI). All three are fast; the right one is the one whose signature optimization matches the shape of your traffic. Benchmark on your workload, not someone else’s.

Serve a RAG endpoint where every request stuffs different retrieved passages into the prompt: prefixes barely overlap, so prefix reuse buys little and the win comes from packing the KV cache tightly and supporting your model well. vLLM is the natural default — broad support, strong general throughput, huge community. Nothing exotic required.

Now serve an agent platform where thousands of requests share the same long system prompt, the same tool definitions, and the same few-shot scaffold, differing only in the tail: prefixes overlap massively. Here SGLang’s RadixAttention computes that shared prefix once and reuses it across the swarm, a win the paged approach alone cannot match — often the single biggest throughput lever available. Same models, same GPUs; the right engine is decided by whether your traffic repeats itself. And if the deciding factor were instead "we standardize everything on the Hugging Face stack," TGI would win on integration, not benchmark.