Prefill/Decode Disaggregation for LLM Serving Architecture (2026)

If you run a large language model in production, you have almost certainly watched one slow request poison a whole batch. A user pastes a 30,000-token document, the GPU spends 800 milliseconds crunching that prompt, and during those 800 milliseconds every other user’s token stream stutters. That is not a tuning problem you can fix with a bigger batch size. It is a structural consequence of running two fundamentally different workloads on the same silicon. Prefill decode disaggregation is the architecture that stops fighting that structure and instead splits inference into two independently scaled pools — one built for the compute-bound prompt phase, one built for the memory-bound generation phase.

This pattern moved from research papers to production defaults over 2024 and 2025, and in 2026 it ships natively in the major serving stacks. It is not free, and it is not always the right call. This post explains the mechanism from first principles, the systems that pioneered it, the KV-cache transfer problem at its core, and the honest conditions under which it beats co-located serving.

What this covers: the two-phase resource split, why co-location causes interference, how the KV cache moves between pools, SLO isolation and goodput, pool-ratio tuning, real systems (DistServe, Splitwise, Mooncake, vLLM, NVIDIA Dynamo), and a decision framework for 2026.

Context and Background

Autoregressive transformer inference has always had two phases, but for years we served them as one. A request arrives, the model reads the prompt, and then it emits tokens one at a time until it hits a stop condition. The first step — reading the prompt and populating the key/value attention cache — is called prefill. The second — generating each subsequent token — is called decode. Early serving systems ran both on the same GPU in the same process, because that is the obvious thing to do and because a single request naturally flows from one phase into the other.

The breakthrough that made high-throughput serving practical was continuous batching, popularized by Orca and then vLLM, which dynamically adds and removes requests from an in-flight batch rather than waiting for a fixed batch to finish. Paired with PagedAttention — vLLM’s block-based KV-cache allocator — continuous batching pushed GPU utilization far above the naive baseline. If you want the full mechanics of that, see our companion piece on continuous batching for LLM inference. Continuous batching is still the foundation; disaggregation sits on top of it.

But continuous batching has a limit it cannot cross while both phases share a GPU. Prefill and decode do not just have different latency profiles — they stress different parts of the hardware. The academic case for separating them was made most cleanly by the DistServe paper (Zhong et al., 2024), which framed the goal as goodput rather than raw throughput and reported up to 7.4x more goodput than co-located baselines. Microsoft’s Splitwise and Moonshot AI’s Mooncake took the same idea into large production fleets. By 2026 the question is no longer “does this work” but “when is the overhead worth it.”

The industry trajectory is worth naming, because it explains why the pattern is only now becoming a default. Splitwise, presented at ISCA 2024, came from Microsoft Azure’s need to serve enormous, heterogeneous traffic economically, and its key finding was that the prefill and decode phases have such different hardware sweet spots that you can profitably build the two pools out of different GPU generations. Mooncake, from the team behind Kimi, reported handling on the order of 115% and 107% more requests on A800 and H800 clusters respectively versus their previous co-located system, and up to a 525% throughput gain in some simulated scenarios while still honoring SLOs. On the open-source side, vLLM added disaggregated-prefill support and NVIDIA shipped Dynamo, a serving framework built from the ground up around disaggregation with a dedicated KV transfer layer. When both the leading open stack and the dominant GPU vendor ship the pattern natively, it has crossed from research into infrastructure.

The Reference Architecture: Two Workloads, Two Pools

Disaggregated serving runs prefill on one dedicated pool of GPUs and decode on another. A request is prefilled on the prefill pool, its KV cache is transferred to a decode worker, and that worker generates the output tokens. Each pool is batched, scheduled, and scaled independently to satisfy its own service-level objective. That is the entire idea, and everything else is a consequence of it.

Figure 1: Co-located serving forces prefill and decode to compete on one shared GPU pool, coupling their latency. Disaggregated serving routes each request through a compute-bound prefill pool and a memory-bound decode pool, letting each hit its own SLO.

In the co-located design on the left, one request’s long prefill occupies the streaming multiprocessors that other requests’ decode steps need, so a single large prompt inflates everyone’s inter-token latency. In the disaggregated design on the right, the prefill pool is provisioned for raw floating-point throughput, the decode pool is provisioned for memory bandwidth and cache capacity, and a transfer step in the middle hands the KV cache from one to the other. The two pools no longer contend for the same resource, so a burst of long prompts scales the prefill pool without touching decode latency.

In one paragraph: Prefill decode disaggregation separates the compute-bound prompt-processing phase and the memory-bandwidth-bound token-generation phase onto distinct GPU pools connected by a KV-cache transfer path. This eliminates prefill/decode interference, lets each pool be scaled and batched to its own latency target, and typically improves SLO-attained throughput — goodput — under mixed and long-prompt workloads, at the cost of transferring the KV cache between pools.

Prefill is compute-bound and bursty

During prefill the model processes every prompt token in parallel in a single forward pass. Because all positions are available at once, the attention and feed-forward matrix multiplications are large, dense, and highly parallel. The GPU’s arithmetic units are the bottleneck; a modern accelerator runs prefill near its compute roofline. Prefill cost scales roughly with the number of prompt tokens for the projections and, because of self-attention, closer to quadratically in sequence length for the attention term. This is why one long prompt is so disruptive: it is a large, indivisible slug of compute that lands all at once.

The output of prefill is the KV cache — the per-layer key and value tensors for every prompt token — plus the first generated token. That cache is the state the decode phase needs. It is also, as we will see, the thing that must physically move between pools.

Decode is memory-bandwidth-bound and latency-sensitive

Decode generates one token per step. Each step reads the entire KV cache for that sequence, computes attention against it, produces a single new token, and appends one new KV entry. The matrix multiplications are tiny — one token wide — so the arithmetic units sit mostly idle while the GPU streams the growing KV cache and the model weights out of high-bandwidth memory. Decode is therefore memory-bandwidth-bound, not compute-bound. Its cost per step scales with the current sequence length because the KV cache keeps growing.

Decode is also where the user experiences latency as smoothness. The metric that matters is inter-token latency (ITL), also called time-per-output-token (TPOT). A user reading a streamed answer notices a stall of 200 milliseconds between tokens far more than they notice which GPU produced them. Because decode steps are cheap and memory-bound, they batch beautifully — you can pack many sequences into one decode step and amortize the weight read across all of them.

That batching behavior is the economic engine of decode. Reading a multi-hundred-gigabyte-per-second stream of weights out of HBM costs the same whether you decode one sequence or two hundred, so throughput per GPU rises almost linearly with batch size until the KV cache fills memory. The binding constraint on decode is therefore not compute but KV-cache capacity: how many concurrent sequences you can hold before you run out of HBM. This is a completely different provisioning question from prefill, where the constraint is raw FLOP throughput and a single request can saturate the device. Two different bottlenecks — capacity versus compute — arguing for two different hardware profiles is the deepest reason the phases want to live apart.

The two phases want opposite things

Here is the crux. Prefill wants to run alone and fast so time-to-first-token (TTFT) is low. Decode wants to run in a large steady batch so per-GPU throughput is high and ITL is smooth. On a shared GPU you cannot have both. If you prioritize a fresh prefill, you interrupt the decode batch and ITL spikes. If you protect the decode batch, incoming prompts queue and TTFT balloons. Continuous batching mitigates this by interleaving at the step boundary — chunked prefill, which slices a long prompt into smaller pieces so it can share steps with decode, is the best co-located answer — but it is a compromise, not an elimination. You are still time-slicing one resource between two workloads that each want the whole thing. Disaggregation removes the shared resource, and with it the compromise.

How a Request Flows Through a Disaggregated System

Walking a single request end to end makes the moving parts concrete. The lifecycle spans a router, the prefill pool, a transfer, and the decode pool.

Figure 2: A request is routed to a prefill worker, which builds the KV cache in one parallel pass and emits the first token, transfers the cache to a decode worker, which then streams the remaining tokens autoregressively.

The sequence begins when the client sends a prompt to a router or global scheduler. The router picks a prefill worker — usually the least-loaded one that can fit the prompt, sometimes one that already holds a prefix of this prompt in cache. The prefill worker runs the single parallel forward pass, producing the full KV cache and the first output token. That first token can be streamed back to the client immediately, which is why TTFT is essentially the prefill latency plus queueing.

Then the KV cache is transferred to a decode worker. The decode worker attaches the sequence to its running continuous batch and generates tokens step by step, streaming each one to the client until an end-of-sequence token or the max-length limit. The whole time, the decode worker is also serving dozens or hundreds of other sequences in the same batch. From the client’s perspective it is one coherent stream; underneath, two different machines built the front and back halves.

The router is doing real work

The router is not a dumb load balancer. It has to decide which prefill worker gets a request based on prompt length and current queue depth, whether a prefix cache hit lets it skip or shorten prefill, and which decode worker has KV-cache headroom and batch capacity. Mooncake (Qin et al., 2024) makes this KV-cache-centric scheduling the heart of its design, and even adds a prediction-based early-rejection policy that refuses requests it forecasts cannot meet their SLO — better to reject fast than to accept and blow the latency budget for everyone. Good routing is a large fraction of what separates a disaggregated system that hits its numbers from one that thrashes.

TTFT and TPOT become independently controllable

Because prefill owns TTFT and decode owns TPOT, and the two live on different hardware, you can tune each without disturbing the other. Need faster first tokens? Add prefill GPUs or use more aggressive tensor parallelism on the prefill pool to cut single-prompt latency. Need smoother streaming under load? Grow the decode batch or add decode GPUs. In a co-located system these knobs fight each other; in a disaggregated system they are genuinely separate. This is the SLO isolation that gives the pattern its value.

The KV-Cache Transfer Problem

Disaggregation replaces prefill/decode compute interference with a new cost: you must physically move the KV cache from the prefill GPU to the decode GPU. This transfer is the pattern’s central engineering challenge, and it is where most of the real-world difficulty lives.

Figure 3: The KV cache moves layer by layer from the prefill GPU’s HBM across an RDMA or NVLink fabric into the decode GPU’s KV store, optionally staged through a pooled CPU-DRAM and SSD tier.

The KV cache is not small. Its size is roughly two (keys and values) times the number of layers, times the sequence length, times the hidden dimension per layer, times the bytes per element. For a large model with a long prompt, a single request’s cache can run from tens of megabytes to several gigabytes. Multiply by the request rate and the aggregate bytes-per-second the fabric must carry becomes the thing that decides whether disaggregation helps or hurts.

A worked illustration makes the scale tangible (these numbers are illustrative, not a benchmark). Take a model with 80 layers and an attention key/value width of 8,192 elements per layer per token, stored in 16-bit precision. That is 2 × 80 × 8,192 × 2 bytes ≈ 2.6 MB of KV state per token. A 16,000-token prompt then produces roughly 42 GB of KV cache for a single request — more than fits in one GPU, which is why long-context serving already leans on paging and tiering. Even a more modest 4,000-token prompt lands near 10 GB. Now suppose 20 such requests arrive per second: the fabric must move on the order of 200 GB/s just to keep the decode pool fed. That is comfortable on NVLink but pushes hard against a single RDMA NIC, which is exactly why topology, not raw GPU count, is often the true capacity ceiling of a disaggregated deployment. Grouped-query and multi-query attention shrink these figures substantially by sharing key/value heads, which is one reason modern long-context models adopt them — smaller KV means cheaper transfer.

Bandwidth and topology dictate feasibility

Where the two pools sit relative to each other changes everything. Within a single node, GPUs connected by NVLink share hundreds of gigabytes per second of bidirectional bandwidth, and a KV transfer is nearly free relative to the compute it unblocks. Across nodes you fall back to the network — InfiniBand or RoCE with GPUDirect RDMA — which is fast but an order of magnitude below NVLink, and now the transfer latency starts to eat into your TTFT budget. This is why serious deployments treat interconnect as a first-class design constraint rather than an afterthought, and why disaggregation took off precisely as high-bandwidth RDMA fabrics became standard in GPU clusters.

Layer-by-layer and overlapped transfer

The naive approach — finish all of prefill, then send the whole cache, then start decode — serializes three expensive steps. Production systems overlap them. Because prefill computes the KV cache layer by layer, you can begin transmitting layer 0’s KV as soon as it is ready while the GPU computes layer 1, pipelining the transfer under ongoing compute so most of it is hidden. Systems like Mooncake and NVIDIA’s Dynamo inference framework build dedicated transfer engines for exactly this, moving KV blocks over RDMA with the transfer overlapped against computation so the effective added latency is a fraction of the raw copy time.

Paged, tiered, and reusable caches

Because vLLM’s PagedAttention already stores the KV cache as fixed-size blocks, those blocks are the natural unit of transfer and of reuse. If two requests share a prompt prefix — a common system prompt, a shared document — the decode pool or a shared cache tier can hold those blocks once and reuse them, so prefill is skipped entirely on a hit. Mooncake extends this into a disaggregated KV-cache pool that spans the cluster’s underused CPU DRAM and SSD, trading cheap storage for expensive recomputation. Our deep dive on KV-cache optimization for LLM inference covers paging, quantization, and prefix reuse in detail; in a disaggregated system all of those techniques double as transfer-reduction techniques, because the cheapest cache to move is the one you did not have to build.

Independent Autoscaling and the Goodput Metric

The reason to accept the transfer cost is that the two pools can now be sized, scaled, and measured independently. This is where disaggregation pays back, and it starts with choosing the right metric.

Figure 4: A workload monitor tracks TTFT and TPOT separately; a breach of the prefill SLO scales the compute-heavy prefill pool, a breach of the decode SLO scales the memory-heavy decode pool, and a rebalancer tunes the prefill-to-decode ratio.

Throughput — tokens per second — is the wrong headline number, because a system can post huge throughput while violating latency SLOs for half its users. The right metric is goodput: the request rate a system sustains while still meeting its SLO targets, typically expressed as SLO attainment above some percentile threshold. DistServe’s central result is stated in goodput for exactly this reason — it reports up to 7.4x higher per-GPU goodput, or the ability to hold 12.6x tighter SLOs at fixed load, versus co-located baselines. Optimizing throughput can actively hurt goodput if it means fatter batches that miss latency targets. Disaggregation optimizes goodput directly by giving each latency target its own controllable pool.

Pool ratio is the key tuning knob

The prefill-to-decode GPU ratio is the parameter you will spend the most time on. It depends on your workload’s shape: the ratio of prompt tokens to generated tokens. A retrieval-augmented or document-summarization workload with 20,000-token prompts and 200-token answers is prefill-heavy and wants more prefill GPUs. A chatbot with short prompts and long, chatty answers is decode-heavy and wants the ratio tilted the other way. Get the ratio wrong and one pool sits idle while the other is the bottleneck — which is worse than co-location, because now you are wasting whole GPUs instead of time-slices. Splitwise (Patel et al., 2024) studied this on Azure production traces and showed that even mixing older, cheaper GPUs into the prefill or decode pool can improve cost per query, because the two phases have different hardware sweet spots.

Autoscaling reacts to each SLO separately

With separate pools and separate metrics, autoscaling becomes tractable. Monitor TTFT; if it breaches, add prefill capacity. Monitor TPOT; if it breaches, add decode capacity. A rebalancer periodically retunes the ratio as the traffic mix drifts across the day — more long-document traffic during business hours, more conversational traffic in the evening. This closed loop is far cleaner than trying to autoscale a co-located pool where a single scale-out action changes both latencies at once in coupled, hard-to-predict ways. The independence is the whole point: two simple controllers instead of one impossible one.

Independent parallelism strategies

Capacity and cost in practice

The capacity math follows directly from the two bottlenecks. Prefill-pool capacity is governed by aggregate FLOP throughput and scales with the prompt-token rate; decode-pool capacity is governed by HBM capacity and bandwidth and scales with the number of concurrent sequences and their lengths. Sizing the fleet means solving both constraints at your target load and taking the max, then adding transfer headroom on the fabric between them. The cost advantage is real but conditional: DistServe’s up-to-7.4x per-GPU goodput and Splitwise’s cost-per-query reductions came from eliminating stranded capacity and from placing each phase on hardware it actually uses. In a co-located pool sized for peak prefill, decode-bound intervals leave FLOPs idle; sized for peak decode, prefill bursts queue. Disaggregation lets you buy exactly the compute-heavy and memory-heavy hardware each phase needs, and to buy them in different quantities — which is where the per-query savings come from. The flip side is that the savings evaporate the moment pool utilization drops, because two under-filled pools waste more than one under-filled pool.

The pools can even run different model-parallel layouts. Prefill, being compute-bound on large matrices, often benefits from more tensor parallelism to cut single-prompt latency. Decode, being memory-bound, may prefer a layout that maximizes KV-cache capacity and batch size per GPU. In a co-located system you must pick one parallelism plan for both phases and compromise. Disaggregation lets each phase run the plan that suits it, which is a second, subtler source of the efficiency gain beyond interference removal.

Trade-offs, Gotchas, and What Goes Wrong

Disaggregation is a genuine architectural improvement for the right workload, but it introduces failure modes that co-located serving simply does not have. Deploy it without respecting these and you will spend more money for worse latency.

The transfer can become the bottleneck. If your interconnect is thin or your pools span nodes, KV-cache transfer latency lands directly on TTFT and transfer bandwidth caps your request rate. On slow fabrics the copy can cost more than the interference you removed. Always measure the achievable KV bytes-per-second against your aggregate cache production rate before committing; if the fabric cannot keep up, disaggregation makes things worse.

Short prompts and low load lose. The pattern’s benefit comes from removing interference and independently batching two heavy phases. If prompts are short, prefill is cheap and barely interferes with decode, so there is little interference to remove — and you have added a transfer plus a network hop for no gain. At low utilization you may be running two half-empty pools where one full co-located pool would have been cheaper and faster. Disaggregation shines under high load with mixed or long prompts, not under light or uniform-short traffic.

Pool-ratio drift causes stranded capacity. Set the prefill-to-decode ratio for a morning traffic mix and it will be wrong by evening. A mistuned ratio strands whole GPUs idle in one pool while the other saturates. This demands active monitoring and a rebalancing loop; a static ratio is a slow leak of money. It is the single most common operational failure in production disaggregated fleets.

Failure handling is harder. A request now has state on two machines. If a decode worker dies mid-generation, you must either recover its KV cache from a replica or a cache tier, or recompute prefill from scratch. If a prefill worker dies after transfer, the decode side may be fine; if it dies before, the request restarts. The distributed-systems surface area is strictly larger, and a naive implementation drops requests on any single-node failure.

Small deployments should not bother. If you are serving from a handful of GPUs, the operational complexity — a router, a transfer engine, two autoscalers, cross-pool observability — almost never pays for itself. Chunked-prefill co-located serving in vLLM will get you most of the way with a fraction of the moving parts.

The baseline you compare against matters. Many disaggregation wins are quoted against a naive co-located system that batches prefill and decode together with no chunking. That is not the fair comparison in 2026. Chunked prefill — slicing long prompts so they interleave with decode steps — closes a large part of the interference gap on a single pool, and it is the honest baseline. When you evaluate disaggregation, tune the co-located system first: enable chunked prefill, size the chunk to protect ITL, turn on prefix caching, and only then measure goodput on both. Some workloads that look like clear disaggregation wins against a naive baseline turn into ties once the co-located system is properly tuned, and a tie does not justify the extra machinery. Reserve disaggregation for the cases where the gap survives a well-tuned baseline.

Practical Recommendations

Start by profiling your real traffic, not a synthetic benchmark. Measure the distribution of prompt lengths and output lengths, and compute the prefill-to-decode compute ratio your workload actually implies. That single number tells you both whether disaggregation will help and roughly what pool ratio to start from. If your prompts are uniformly short, stop here and use chunked prefill on a co-located pool.

If your workload is mixed or prefill-heavy and you are latency-constrained at scale, prototype disaggregation on your existing stack before building anything custom. vLLM ships disaggregated-prefill support and NVIDIA Dynamo provides a production-grade disaggregated serving framework with a purpose-built KV transfer engine; either lets you test the pattern without writing a transfer engine yourself. Measure goodput — SLO attainment at your target percentile — not raw throughput, and compare it honestly against a well-tuned chunked-prefill baseline on the same hardware.

A short deployment checklist:

• Measure interconnect bandwidth first. Confirm your NVLink or RDMA fabric can carry your aggregate KV-production rate with headroom.
• Co-locate pools within a node when you can, to keep transfer on NVLink rather than the network.
• Overlap transfer with compute layer by layer; never serialize prefill, transfer, and decode.
• Instrument TTFT and TPOT separately and alert on each SLO independently.
• Run a rebalancing loop that retunes the prefill-to-decode ratio as traffic drifts.
• Enable prefix caching so shared prompts skip prefill entirely — the cheapest cache to transfer is the one you never built.
• Plan failure recovery for state that now lives on two machines.

For the underly