A serving engine that performs well on independent, short prompts may behave differently when requests share long documents or generate structured output. Choosing between vLLM and SGLang requires matching their scheduling and cache-reuse behavior to the traffic you plan to serve.
Both engines support high-throughput LLM serving. This comparison examines their memory management, prefix reuse, deployment features, and benchmark conditions to help you choose what to test on your own workload. Tensormesh Platform integrates with vLLM for persistent cache reuse.
The short version, before the evidence:
| If your traffic | Start with | Because |
|---|---|---|
| Shares long prefixes: chat, agents, few-shot, RAG | SGLang | Cross-request reuse is what its paper measures |
| Needs the widest documented model list | vLLM | 200+ architectures, the project’s own count |
| Is reinforcement learning rollouts | SGLang | Its README calls it a proven rollout backend |
| Uses Tensormesh Platform | vLLM | Platform integrates with vLLM |
What are vLLM and SGLang?
vLLM is an open-source LLM inference and serving engine originally developed in UC Berkeley’s Sky Computing Lab. SGLang is a production inference framework designed for low latency and high throughput, from a single GPU to distributed clusters. Both serve open-weight models.
The engines support many of the same optimizations. SGLang’s README lists prefix caching, continuous batching, paged attention, four kinds of parallelism, speculative decoding, and chunked prefill. vLLM’s docs list most of the same ideas. These are two implementations of one body of serving research with different defaults, and the latency either produces has a direct commercial cost.
vLLM vs SGLang: architecture and core technology
PagedAttention: allocation
PagedAttention addresses KV cache allocation. As the 2023 vLLM paper explains, cache requirements change with each request, and fragmentation or duplicated state can waste enough memory to limit batch size.
So stop allocating one contiguous region per request. The cache splits into fixed-size blocks, and a sequence’s blocks need not be adjacent in GPU memory. The paper reports near-zero KV memory waste and supports sharing cached state within and across requests. This describes the original design, and the docs home now names FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton as the optimized kernels. For current implementation details, consult the version you deploy.
RadixAttention: retention and lookup
RadixAttention retains KV state for prompts and generated output after a request finishes, using a radix tree to support prefix lookup, reuse, insertion, and eviction. The SGLang paper describes this as a systematic way to reuse cache state across requests.
A radix tree maps token sequences to their KV tensors, so a later request sharing a prefix finds them instead of recomputing. Eviction takes least-recently-used leaves first, keeping shared ancestors alive until they become leaves. This describes the design in the June 2024 paper.
The correction most comparisons need
RadixAttention does not replace PagedAttention. SGLang’s paper describes it as compatible with continuous batching, paged attention, and tensor parallelism, and its README lists both RadixAttention and PagedAttention as runtime features.
They sit at different layers. Paging is allocation: how blocks are handed out and reclaimed inside a request, so fragmentation doesn’t cap your batch size. Radix-tree indexing is retention: what survives a request, and how the next one finds it. Both engines do both. vLLM’s design docs note that public APIs such as OpenAI and Anthropic, and open-source frameworks including SGLang, use prefix caching.
The differences are in the surrounding machinery. vLLM supports optional per-request salts that restrict cache reuse to requests with the same salt. SGLang pairs its cache with a scheduling policy that orders requests to raise the hit rate. For the mechanics underneath, see prefill and decode.
vLLM vs SGLang: performance and throughput benchmarks
The figures below come from the original vLLM and SGLang papers, published in 2023 and 2024. They use different baselines and do not provide a head-to-head comparison of the versions you deploy.
| Engine | Best for | Published throughput | Published TTFT | Conditions you cannot drop |
|---|---|---|---|---|
| vLLM | General-purpose serving, many model architectures | 2–4x throughput at equal latency versus FasterTransformer and Orca, from the 2023 PagedAttention paper | No comparable TTFT figure; tune max_num_batched_tokens for the workload | A 2023 result for vLLM as a whole serving system, not for PagedAttention alone, against 2023-era baselines. The reported gains were larger for longer sequences, larger models and more complex decoding. |
| SGLang | Heavy prefix overlap: chat, agents, few-shot, RAG | Up to 6.4x higher throughput versus Guidance, vLLM and LMQL, from the SGLang paper | Average 1.7x lower first-token latency on Vicuna-33B, month-long production deployment | Measured mid-2024, mostly on A10G 24GB with some A100 80GB runs. The paper attributes the gains to KV cache reuse, parallel execution within a program and faster constrained decoding together. |
The 2–4x and 6.4x gains apply to the systems and baselines evaluated in those papers. Neither predicts performance on a newer deployment.
SGLang’s authors measured RadixAttention in Chatbot Arena over one month: cache hit rates were 52.4% for LLaVA-Next-34B and 74.1% for Vicuna-33B. Reuse came from shared system messages, repeated example images and multi-turn history, reducing first-token latency by an average of 1.7x for Vicuna-33B. Across the synthetic suite, hit rates ranged from 50% to 99%, measured as cached prompt tokens divided by prompt tokens.
The 1.7x TTFT improvement applies to Vicuna-33B in that production measurement; results will vary with prefix reuse in your traffic.
Why benchmark numbers for vLLM vs SGLang disagree so much
Four factors can change the result.
Version drift. vLLM shipped v0.22.0 through v0.28.0 between late May and late August 2026, roughly a release every ten days. On the SGLang side you often cannot tell which build was measured at all.
Hardware. The SGLang paper ran most experiments on A10G 24GB cards, and 24GB results don’t transfer to an H100 fleet.
Prefix overlap, the largest of the four. Hit rate ran from 50% to 99% across the paper’s own suite, and it decides how much prefill work either engine skips.
Configuration. Smaller chunked-prefill values in vLLM improve inter-token latency by reducing prefill interference; higher values improve TTFT. Tuning for different metrics can therefore change the ranking on identical hardware. The SGLang paper excludes optimizations that alter computation results, another condition to check when comparing benchmarks.
Benchmark your own deployment. We publish an open CLI for this, and comparing serving stacks with the tmesh benchmark tool points it at any OpenAI-compatible endpoint, including a vLLM or SGLang deployment you already run.
When to choose vLLM vs when to choose SGLang
The deciding variable is workload shape, so this is a checklist, not a feature grid.
| Signal in your workload | What it points to | Workload fit |
|---|---|---|
| High prefix overlap | SGLang | The workloads measured in its paper, including shared system messages, reused example images and multi-turn history, where cross-request reuse can help. |
| Mostly unique prompts | Either, decide on operations | Reuse cannot help where nothing repeats. The SGLang paper reports negligible RadixAttention memory and time overhead on cache misses, but that is a low overhead rather than a reuse benefit. |
| Broad model coverage | vLLM | vLLM lists support for more than 200 model architectures. |
| Non-NVIDIA hardware | Check both against your parts | Both publish non-NVIDIA support. SGLang’s README lists AMD GPUs, Intel Xeon CPUs, Google TPUs and Ascend NPUs alongside NVIDIA. vLLM’s GPU installation page has four tabs: NVIDIA CUDA, AMD ROCm, Intel XPU and Apple Silicon. Neither list is a measured comparison. |
| Reinforcement learning rollouts | SGLang | Its README lists use as a rollout backend for model training in AReaL, Miles, slime, Tunix and verl. |
| Several models behind one endpoint | vLLM: one model per server process. Confirm SGLang’s model-serving limits for your deployment. | vLLM hosts one base model per server. Check SGLang’s deployment configuration for your model-serving requirements. |
Two operational settings need attention before deployment.
The memory controls require tuning in both engines. vLLM lists a default of 0.92 for –gpu-memory-utilization, while SGLang selects –mem-fraction-static using heuristics rather than a single documented default.
# vLLM: fraction of GPU memory for weights plus KV cache, documented default 0.92--gpu-memory-utilization 0.92# SGLang: the same trade, no default published in the docs--mem-fraction-static <value>Both engines support separating prefill and decode to manage latency. vLLM marks disaggregated prefilling as experimental and describes it as a way to tune TTFT and inter-token latency independently, rather than improve raw throughput. SGLang uses PD disaggregation to address prefill interference and data-parallel attention imbalance. For the architecture, see how prefill and decode split apart.
Where TensorRT-LLM and TGI fit in
TensorRT-LLM: peak throughput on NVIDIA hardware
NVIDIA’s TensorRT LLM accelerates inference on NVIDIA GPUs. Its optimizations include in-flight batching, paged KV cache with block reuse, chunked prefill, speculative decoding and beta disaggregated serving. In-flight batching is another name for continuous or iteration-level batching.
The trade-off is hardware. Its supported-hardware page lists Blackwell, Hopper, Ada Lovelace, and Ampere, NVIDIA and nothing else. NVIDIA claims that FP8 on H100 and later can double performance and halve memory use relative to 16-bit floating point, a vendor claim with no model, benchmark, or baseline. vLLM’s docs put the throughput half at up to 1.6x. These figures use different conditions and should not be compared directly.
What happened to TGI?
Hugging Face has put TGI into maintenance mode. As of September 5, 2026, its README limits ongoing work to small bug fixes, documentation updates, and lightweight maintenance. It recommends vLLM, SGLang and compatible local engines such as llama.cpp or MLX for new work.
That is the maintainer’s recommendation, not a benchmark, and it sets no end date. If you run TGI, your migration targets are the two engines compared here.
The KV cache problem behind all four engines
Sort the techniques above, and only some of them avoid recomputing tokens. Prefix reuse does. Paging, batching, disaggregation, and memory flags mostly change how work is allocated, scheduled, moved, or executed. Separate the two and one shared limit stands out.
The built-in prefix caches live in memory and are subject to eviction. vLLM reclaims cached blocks from its free queue under memory pressure, while SGLang applies LRU eviction to radix-tree leaves. Cached prefixes can also be lost on restart or redeploy, and a request routed to a cold replica cannot use another replica’s local cache.
Connectors can extend reuse beyond the built-in cache. vLLM’s disaggregated prefilling interface includes LMCacheConnectorV1, which can connect instances to a standalone LMCache server holding shared KV state. External storage can retain that state beyond the engine process.
Engine choice and caching strategy are two decisions, not one. A layer that owns the KV cache can carry it across requests, sessions and replicas, a lever no single-engine benchmark captures. That is what Tensormesh Platform is: our self-hosted software, built on the open-source LMCache project our founders created. Owning the context caching lifecycle covers what persistence changes, and LMCache’s peer-to-peer architecture covers sharing it between replicas.
Tensormesh Platform supports vLLM. For engine, model, and feature compatibility, consult the compatibility matrix before deploying.
Our partner program covers integration opportunities. Our separate Serverless Inference service charges nothing for cached tokens, so greater context reuse can reduce the cost per request.
Choosing an engine
Pick vLLM for the broadest documented model coverage and a release you can pin. Pick SGLang when much of your traffic shares long prefixes, or for reinforcement learning rollouts. Pick TensorRT-LLM if you are committed to NVIDIA hardware. For TGI, follow the maintainer’s recommendation and plan a move.
Measure your prefix reuse ratio on your own traffic to assess how much repeated computation either engine can avoid.
Contact our team to discuss caching for your inference stack.
Frequently asked questions
Is SGLang faster than vLLM?
No first-party benchmark answers that for current versions of both. The SGLang paper‘s reported gain of up to 6.4x is from mid-2024, mostly on A10G GPUs, against Guidance, vLLM, and LMQL, and it credits three optimizations together rather than RadixAttention alone. The vLLM it beat is not the vLLM shipping today. Where prefixes overlap heavily, SGLang has a real structural argument. Where they do not, benchmark current releases on your own model, hardware, and concurrency: performance depends on those deployment conditions.
Is RadixAttention better than PagedAttention?
They are not alternatives. SGLang’s paper lists RadixAttention as compatible with paged attention, and its runtime ships both. Paging governs allocation inside a request; radix-tree indexing governs reuse across requests.
Does vLLM support TensorRT?
Not as a serving backend. vLLM’s docs list TRTLLM-GEN among its optimized attention kernels, but that is a kernel, not the TensorRT LLM engine. Running TensorRT-LLM means running NVIDIA’s stack instead of vLLM.
Is TGI dead?
It is in maintenance mode, accepting small bug fixes, documentation updates, and lightweight maintenance work. Hugging Face’s README recommends vLLM and SGLang for new work and doesn’t set a retirement date.
Can I run Tensormesh Platform with SGLang?
Tensormesh Platform supports vLLM. For a SGLang deployment, check the compatibility matrix for engine support and evaluate SGLang’s built-in prefix cache for your workload.