A long prompt arriving while other requests are generating answers can increase latency for both. Prefill needs to process many input tokens, while decode needs to keep producing output tokens at a steady pace. Running them together makes those workloads compete for the same GPU resources.
Disaggregated serving places prefill and decode on separate GPU pools so each can be tuned independently. That separation adds KV cache transfer overhead, so its value depends on prompt length, network capacity, and latency targets. Here is how the architecture works and how to decide whether the tradeoff suits your workload.
What is disaggregated serving?
Disaggregated serving runs the two phases of an LLM request on separate GPU pools. A prefill pool processes the prompt and builds the KV cache. A decode pool receives that cache over the network and generates tokens without recomputing the prompt. Each pool is sized for its own bottleneck.
Every stack names it differently: PD disaggregation in SGLang’s and Tensormesh’s docs, DPD at AWS, disaggregated prefilling in vLLM’s. It exists because prefill is compute-bound while decode is memory-bandwidth-bound, covered in our two phases of inference and our serving pillar.
Disaggregated vs. aggregated serving
| Aggregated (colocated) | Disaggregated | |
|---|---|---|
| GPU pools | One, shared by both phases | Two, each sized for its own bottleneck |
| Latency control | One aggregate number | TTFT and inter-token latency tuned separately |
| Fits | Short prompts, low concurrency, batch | Long prompts, high concurrency, separate SLOs |
Disaggregation separates prompt processing from token generation, adding a KV transfer whose cost must be justified by workload and hardware.
The router decides, per request
llm-d asks how much of the prompt is already cached on the decode side before committing to the disaggregated path. AWS makes the same call on one number: a routing threshold defaulting to 4,096 tokens, below which requests bypass the prefiller. The router, not the engine, decides whether you see any benefit.
The transport is where it bites
The second requirement is the KV transfer, prefill to decode, over RDMA. The common transport is NIXL, NVIDIA’s Inference Xfer Library, which abstracts over backends like UCX and libfabric and the fabrics beneath them. It does not remove the hardware underneath. AWS’s floor is one prefill node and one decode node with RDMA-capable EFA, on instance types supporting both NVLink and EFA with RDMA read and write, in one Availability Zone. P5 and P6 are its named examples; G6, G6e, and G7e support the same RDMA modes but bottleneck on PCIe between GPUs.
The Kubernetes network configuration matters too. Under a standard overlay CNI, the pod’s GID table can be empty, and UCX initialization can fail. Our PD disaggregation deployment therefore requires host networking.
When it isn’t worth doing
Each implementation has workload-dependent limits.
| Situation | Deployment guidance |
|---|---|
| One model on one GPU | NVIDIA Dynamo: a single model on one GPU will often need only the inference engine. |
| Batch, low-concurrency or short-prompt traffic | AWS: colocated serving is simpler when GPU contention is low, including low-concurrency or short-prompt traffic and batch or offline jobs focused on TTFT. |
| No RDMA fabric | llm-d: without RDMA, NIXL falls back to TCP, which is intended for testing and development rather than efficient production transfers. |
Inside the envelope, you still pay. AWS records a modest increase in time to first token from the transfer, and currently supports one decoder replica with multiple prefiller replicas, so prefill is the only side you scale out. llm-d’s stated condition is long context at roughly a 10:1 input-to-output ratio on medium-to-large models.
Where the KV cache layer fits
Once you split the phases, the data path between them sets your latency. vLLM labels disaggregated prefilling as experimental and depends on third-party connectors for production deployments. LMCache, the open-source project we commercialize, ships one of those connectors.
Tensormesh Platform provides prefill/decode disaggregation, so those phases can run on separate GPU pools sized for their workloads. Peer-to-peer KV cache sharing complements it, fetching warm state from a peer instead of recomputing it, over RDMA where the network supports it and over ordinary TCP where it does not.
Tensormesh Platform supports vLLM. Check our compatibility matrix for model and configuration details before planning a deployment. Our separate Serverless Inference service does not charge for cached tokens; self-hosted teams can instead focus on owning their context caching lifecycle.
Before building it, measure your prompt lengths against your router’s threshold. Below it, you’re buying a fabric you won’t use.
Contact our team to discuss prefill and decode disaggregation for your workload.
Frequently asked questions
Does disaggregated serving improve throughput?
Not on its own. vLLM describes disaggregated prefill as a latency-control feature rather than a raw-throughput improvement: you tune TTFT and inter-token latency separately. Even DistServe, the OSDI 2024 paper that argued for splitting the phases, reports its headline 7.4x as goodput, the request rate served inside both latency SLOs at over 90% attainment, not raw throughput.
Which inference engines support disaggregated serving?
vLLM, SGLang, and TensorRT-LLM each expose disaggregation, though vLLM documents its own as experimental. NVIDIA Dynamo and llm-d orchestrate them, with support that varies by backend and connector: in llm-d v0.9, the routing sidecar drives vLLM through a two-phase sequential protocol and SGLang through an asynchronous one.