Chunked Prefill and Prefill-Decode Interference in LLM Serving

A single 8,000-token prompt can freeze every other user on a GPU for a third of a second. Nothing is broken: the server is simply doing a large, legitimate piece of work, and the scheduler has no way to say “later.” In an interactive chat product, that stall shows up as a token stream that hiccups for no visible reason, and as a p99 inter-token latency that no amount of extra batching can fix. The mechanism behind it is chunked prefill and its absence, and understanding it is the cleanest way to understand how modern LLM serving schedulers actually think.

This matters now because long prompts are the norm: retrieval-augmented context, agent transcripts, and code repositories routinely push prompts into the tens of thousands of tokens. Every serious engine has converged on the same remedy, but each defaults and tunes it differently, and the knobs interact with prefix caching and disaggregation in ways the docs only hint at. You will leave with a first-principles model of prefill-decode interference, the Sarathi-Serve stall-free algorithm, the exact vLLM and SGLang settings, and a worked arithmetic example you can reproduce on your own hardware numbers.

What this covers: why prefill and decode fight, how a token budget per iteration resolves the fight, chunk-size trade-offs between time to first token (TTFT) and time per output token (TPOT), vLLM and SGLang configuration, and when to prefer prefix caching or disaggregation instead.

Context and Background

An LLM request has two computationally different phases. Prefill processes the entire prompt in one forward pass to build the key-value (KV) cache and emit the first token. Decode then generates one token per step, each step reading the whole weight set plus the accumulated KV cache. Prefill pushes thousands of tokens through every weight matrix at once, so the arithmetic units are the bottleneck: it is compute-bound. Decode pushes one token per sequence through the same matrices, so the time goes to streaming weights from high-bandwidth memory (HBM): it is memory-bound. This asymmetry is the root of everything that follows.

Early serving systems scheduled at the request level, which wasted GPUs because requests finish at different times. The Orca system (OSDI 2022) introduced iteration-level scheduling, now called continuous batching: the scheduler re-forms the batch every forward pass, so finished sequences leave and new ones join immediately. We cover that mechanism in depth in our continuous batching architecture guide, and this article assumes it. vLLM’s PagedAttention then removed the memory fragmentation that limited batch sizes, which is what made big mixed batches practical.

Continuous batching created a new problem, though. When a new request is admitted, its whole prompt is prefilled in the very next iteration. That iteration is now enormously more expensive than a pure-decode iteration, and every running decode request waits for it. This is prefill-decode interference. The research response came in two lines. The SARATHI paper (Agrawal et al., arXiv 2308.16369, August 2023) proposed splitting prefills into equal chunks and piggybacking decodes onto them. Its successor, “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve” (arXiv 2403.02310; OSDI 2024, pages 117 to 134), generalized this into stall-free scheduling with a per-iteration token budget. The other line, DistServe (arXiv 2401.09670), removes interference by running prefill and decode on different GPUs entirely.

By 2026 the first line has won the default position. According to the vLLM documentation, chunked prefill is enabled by default in the V1 engine whenever possible, and SGLang exposes it through a chunk-size flag. The practical questions are no longer “should I chunk?” but “how big should the budget be, and what does it cost me?” For the bigger picture of engine choice, see our LLM inference benchmark across vLLM, TGI, SGLang and Triton. The primary references are the Sarathi-Serve paper on arXiv and the vLLM optimization documentation.

Two latency metrics that pull in opposite directions

Two metrics govern user experience. Time to first token is the delay from request arrival to the first generated token; it is dominated by queueing plus prefill. Time per output token, also called inter-token latency (ITL) in the vLLM docs, is the gap between successive tokens once streaming starts. Production SLOs usually bound TTFT at some percentile and TPOT at another.

Prefill-decode interference damages TPOT: a long prefill inflates one decode gap from a few milliseconds to hundreds. Naively avoiding interference by never mixing the phases damages TTFT: new prompts wait behind running decodes. The whole design space is the tension between these two, and chunked prefill is a dial between them rather than a free lunch.

Why Prefill and Decode Collide: A Reference Model

This section builds the mental model. The short version is the direct answer below, and the rest of the article elaborates it.

Direct answer: Prefill-decode interference occurs because a continuous-batching scheduler runs one forward pass per iteration and that pass is only as fast as its heaviest request. A long prompt makes the iteration compute-heavy and slow, so every co-scheduled decode, which needs only a few milliseconds of memory-bound work, waits for the whole pass. Chunked prefill caps the per-iteration work with a token budget, bounding the stall.

Figure 1: Without chunking, one long prefill dominates an iteration and stalls decodes. With chunked prefill, the iteration is bounded and decodes advance every step.

Figure 1 contrasts the two schedules. In the unchunked path, the prefill iteration lasts as long as the prompt requires. In the chunked path, each iteration is fused: decode tokens first, then a slice of the prompt sized to the leftover budget. The long-description version: the left branch shows a single long step that blocks running users, while the right branch shows repeated bounded steps in which decoding never pauses.

The roofline view: why the phases differ

A GPU has a compute ceiling (FLOPs per second) and a memory-bandwidth ceiling (bytes per second). The ratio of the two is the ridge point, the arithmetic intensity (FLOPs per byte moved) at which a kernel transitions from memory-bound to compute-bound. For modern datacenter GPUs, the ridge point sits at a few hundred FLOPs per byte for bf16 dense matrix multiplication. Check your own part’s datasheet; the exact value depends on the GPU, so treat any single number as hardware-specific.

A linear layer processing T tokens against a weight matrix of W bytes does roughly 2 x (parameters) x T FLOPs while moving W bytes of weights once. Its arithmetic intensity therefore scales with T. With bf16 weights, intensity is roughly T FLOPs per byte (two FLOPs per parameter per token, divided by two bytes per parameter). A decode batch of 32 sequences has T of 32, far below the ridge point, so the pass is memory-bound and time is set by weight bandwidth. A prefill of 8,192 tokens has T of 8,192, well above the ridge, so it is compute-bound and time is set by the FLOP rate.

This explains the most important structural fact for schedulers: below the ridge point, adding tokens to a batch is nearly free, because the weights are being read anyway. Decode batches therefore leave compute idle. Prefill chunks are the perfect filler. That is the efficiency half of the Sarathi insight: piggyback decodes onto a chunk, or equivalently, fill idle compute during decode with prefill work.

Head-of-line blocking at iteration granularity

Head-of-line (HOL) blocking is usually a queueing term. Here it appears inside the iteration. Continuous batching removes request-level HOL blocking, since a short request no longer waits for a long one to finish generating. But it reintroduces HOL blocking at the level of a single step: the step cannot return until the slowest member’s work is done, and no member can emit a token until the step returns.

Consider an engine running 64 decoding users with a 6 ms step. If a prompt of 8,192 tokens arrives and the scheduler prefills it whole alongside them, all 64 users see a gap of roughly the prefill time. At p99 TPOT, an engine under steady arrival of long prompts will show that spike constantly. Mean TPOT can look healthy while the tail is awful, which is exactly why teams that watch only averages miss this problem.

The token budget: the single control variable

Sarathi-Serve’s central abstraction is a token budget per iteration, a cap on the total number of tokens, prefill and decode combined, processed in one forward pass. The scheduler fills the batch in priority order. Every ongoing decode contributes one token and is admitted first. The leftover budget goes to prefill tokens, from partially processed prompts first and then from new ones. A prompt that does not fit is cut at the budget boundary; the remainder waits for the next iteration.

Figure 2 shows this loop. The crucial property is the one the paper calls stall-free batching: new requests join batches without pausing ongoing decodes. Because decodes are always admitted before prefill tokens, a running user never loses a step to a newcomer. Because every iteration processes roughly the same number of tokens, iteration times are also nearly uniform, which matters again for pipeline parallelism.

Figure 2: The iteration scheduler reserves one token per running decode, spends the remaining token budget on a prefill chunk, and keeps unfinished prompts in the running set.

Chunking is exact, not approximate

A natural worry is whether splitting a prompt changes the output. It does not, mathematically. Causal attention means token i attends only to tokens up to i. When chunk k runs, the KV entries of chunks 1 to k-1 are already in the cache, and the chunk’s queries attend to them plus the earlier tokens of their own chunk. The result is identical to a monolithic prefill, up to floating-point reordering effects in the kernels. What changes is cost accounting, not semantics, and we quantify the cost below.

Worked Example: Prefill Chunks Versus a Frozen Batch

All numbers in this section are illustrative, built from round hardware figures so you can swap in your own. They are not benchmarks. The point is the structure of the arithmetic.

Assume a dense 8-billion-parameter model in bf16 (16 GB of weights), served on one accelerator with an effective 400 TFLOP/s of dense bf16 throughput and 3 TB/s of HBM bandwidth. Those are plausible sustained figures for a current datacenter part but are assumptions, not measurements.

Decode step cost. A pure-decode step must read all 16 GB of weights: 16 GB divided by 3 TB/s is about 5.3 ms. KV cache reads add more depending on context length; call the total about 6 ms. Compute per step is tiny: 64 tokens times 16 GFLOP per token is about 1 TFLOP, or 2.6 ms of compute, hidden under the memory time.

Whole-prompt prefill cost. An 8,192-token prompt costs about 8,192 times 16 GFLOP, roughly 131 TFLOP (ignoring attention’s quadratic term, which adds modestly at this length). At 400 TFLOP/s that is about 330 ms. Unchunked, those 64 decoders each lose roughly 330 ms: their gap is about 55 times the normal 6 ms step.

Chunked with a 512-token prefill allotment. Let the scheduler’s budget be 576 tokens per iteration: 64 decode tokens plus a 512-token chunk. Compute is 576 times 16 GFLOP, about 9.2 TFLOP, which at 400 TFLOP/s is roughly 23 ms. Memory time is still about 6 ms, so the pass is now compute-bound at roughly 23 ms. The prompt needs 16 chunks, so it finishes prefill after about 16 x 23 = 368 ms.

Compare outcomes. The new request’s TTFT rose slightly (368 ms versus 330 ms, about 12 percent) because each chunk iteration also carries the decoders’ work. But every running decoder now sees a worst-case gap of about 23 ms instead of 330 ms, a 14-fold reduction in the stall. Those 16 iterations also produced 16 tokens for each of the 64 users that would have been frozen, so total useful work in those 368 ms went from 0 decode tokens to 1,024.

Chunk-size sweep and the KV re-read overhead

Now vary the budget. With a 2,048-token budget (64 decode plus roughly 1,984 prefill), compute per iteration is 2,112 times 16 GFLOP, about 34 TFLOP, or roughly 85 ms. The prompt needs about 4.1 chunks, so about 5 iterations, finishing in roughly 4 x 85 plus one smaller step, around 350 ms. The stall for decoders is 85 ms, 3.7 times worse than the 23 ms case, but TTFT is closer to the unchunked 330 ms.

There is a real cost the simple model hid. Chunk k must read the KV cache of all earlier chunks from HBM. Take a Llama-3-8B-style layout: 32 layers, 8 KV heads, head dimension 128, bf16. That is 2 (K and V) x 32 x 8 x 128 x 2 bytes = 131,072 bytes, or 128 KiB per token. The full 8,192-token prompt is 1 GiB of KV. With 512-token chunks, chunk k re-reads (k-1) x 512 tokens of earlier KV. Summed over 16 chunks, that is 512 x (0 + 1 + … + 15) = 512 x 120 = 61,440 token-reads, about 7.5 GiB of extra HBM traffic. At 3 TB/s that is roughly 2.5 ms spread across the whole prefill. That is under 1 percent of the 368 ms, so for this model and length the re-read cost is negligible.

The cost grows quadratically with prompt length and linearly with chunk count, so it is not always negligible. At 128,000 tokens with 512-token chunks, there are 250 chunks, and the re-read volume is about 512 x (250 x 249 / 2) = 15.9 million token-reads, roughly 1.9 TiB at 128 KiB per token, or about 0.7 seconds of pure HBM time against an ideal prefill of several seconds. That is meaningful, and it is one reason long-context workloads use larger chunks or dedicate hardware to prefill. Attention kernels also cache-block, so the real overhead differs from this simple bound. Treat the figures as order-of-magnitude, and measure on your own stack.

Sarathi-Serve: Stall-Free Batching in Detail

Having the intuition, we can now describe the algorithm in the terms of the paper. The authors state the problem as a throughput-latency tradeoff. Systems like vLLM historically favored prefill (admit new requests aggressively), which gives good throughput and TTFT but generation stalls. Systems like FasterTransformer favored decode (finish running requests before starting new ones), which gives low TPOT but poor throughput and long queues. Sarathi-Serve claims to dissolve the tradeoff by making batches uniform and bounded.

The scheduling loop

Per iteration, the scheduler follows an order the paper describes as stall-free. First, add all running decode requests. Second, add any partially prefilled request, contributing its next chunk. Third, admit new requests while budget and KV memory remain, adding as many tokens of each as the leftover budget allows. If a new request’s whole prompt does not fit, it contributes only a chunk and carries over.

Decodes always get in, so TPOT is bounded by the iteration time, which the budget bounds. The token budget is selected from the TPOT SLO: the largest budget whose iteration time still satisfies the latency target maximizes the prefill share and therefore throughput. The paper profiles iteration time versus tokens to choose it, which means the budget is a function of model, hardware, and SLO, never a universal constant.

Reported results

The abstract of arXiv 2403.02310 reports serving capacity under tail-latency constraints relative to vLLM: 2.6 times higher on Mistral-7B on a single A100, up to 3.7 times on Yi-34B across two A100 GPUs, and up to 5.6 times end-to-end capacity for Falcon-180B with pipeline parallelism. These are the authors’ numbers on their workloads and software versions from early 2024; vLLM has changed substantially since, including adopting chunked prefill itself, so do not expect the same ratio against a current vLLM. The original SARATHI paper reported, for LLaMA-13B on an A6000, up to 10 times decode throughput and 1.33 times end-to-end throughput, and for GPT-3 with pipeline parallelism 6.29 times fewer bubbles and 1.91 times end-to-end throughput.

Why uniform batches also fix pipeline bubbles

Pipeline parallelism splits layers across devices and streams micro-batches through stages. If micro-batches take very different times (one carries a 4,000-token prefill, the next only decodes), stages idle waiting on the slowest neighbor: pipeline bubbles. A fixed token budget makes every micro-batch cost nearly the same, so stages stay busy. This is why the pipeline-parallel gains in the paper are the largest, and why SGLang added a dynamic chunking option for pipeline parallelism, discussed below.

How vLLM and SGLang Implement It

Figure 3: A new 8,192-token prompt is prefilled over 16 iterations while the existing decoding user receives a token on every one of them.

Figure 3 traces the walk-through from the worked example as a message sequence. The long-description version: request A arrives, and for sixteen iterations the scheduler fuses one chunk of A with a decode step for user B; only after the last chunk does A emit its first token, and from iteration 17 both decode together.

vLLM

The vLLM V1 documentation describes chunked prefill as splitting large prefills into smaller chunks and batching them with decode requests, enabled by default in V1 whenever possible. Its scheduling policy is exactly the stall-free order: all pending decode requests are batched first, then the remaining token budget is allocated to pending prefills, and a prefill that does not fit is chunked automatically. The documentation names the benefit in the same two terms used here: better inter-token latency because decodes are not stalled, and better GPU utilization because compute-bound prefill is mixed with memory-bound decode in one batch.

The main tuning knob is max_num_batched_tokens, the per-iteration token budget (the --max-num-batched-tokens server argument). The docs say that smaller values, with 2048 cited as an example, improve ITL because fewer prefill tokens slow down decodes, and larger values improve TTFT because more prefill tokens fit in each batch. For throughput, the docs recommend values above 8192, especially for smaller models on large GPUs. Setting the budget equal to max_model_len roughly approximates the older V0 default policy, though decodes still keep priority. The docs also warn that when chunked prefill is disabled, max_num_batched_tokens must be larger than max_model_len, or the server may fail at startup.

Two verification notes. First, the optimization page I consulted does not document an enable_chunked_prefill flag for V1; older vLLM versions did expose --enable-chunked-prefill, and you should check vllm serve --help for your installed version before copying it from a blog or a forum answer. Second, the page gives no single default for the budget; defaults have varied across versions and hardware, so read the value your engine logs at startup rather than assuming. A related concurrency limit, max_num_seqs, caps how many sequences are in a batch and interacts with the budget: a budget of 8,192 with 256 sequences leaves 7,936 tokens of prefill headroom even with every slot decoding.

# Illustrative vLLM launch; verify flag names with --help for your version
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-num-batched-tokens 2048 \
  --max-num-seqs 128 \
  --enable-prefix-caching

SGLang

SGLang exposes the same idea through --chunked-prefill-size, documented as the maximum number of tokens in a chunk for chunked prefill, with the default shown as None in the server-arguments reference and -1 disabling chunking. Because the default is documented as unset, the effective chunk size can be chosen per hardware by the engine; confirm in the startup log rather than assuming. A separate flag, --max-prefill-tokens (documented default 16384), bounds the number of tokens in a prefill batch, and --schedule-policy selects among fcfs (default), lpm, random, dfs-weight, lof, priority and routing-key. The lpm (longest prefix match) policy reorders the queue to favor requests with long cache hits, which matters for the radix-tree prefix cache discussed below.

For pipeline parallelism, SGLang documents --enable-dynamic-chunking (default off): chunk sizes are computed from a fitted function so execution time stays consistent across chunks. This is the same insight as Sarathi’s uniform batches taken one step further. Attention cost grows with prefix length, so a fixed token count produces later chunks that run slower than earlier ones; dynamic chunking shrinks later chunks so the time per chunk, not the token count, stays level. I did not find --enable-mixed-chunk in the server-arguments page I consulted, so I do not describe it here. Mixed-chunk behavior has differed across SGLang releases, and you should consult the release notes for your version.

A scheduler comparison

Aspect	Sarathi-Serve (paper)	vLLM V1	SGLang
Control variable	Token budget per iteration	`max_num_batched_tokens`	`--chunked-prefill-size`, `--max-prefill-tokens`
Decode priority	Always admitted first	Decodes batched first	Policy selected by `--schedule-policy`
Chunk uniformity	Fixed budget, equal chunks	Budget leftover after decodes	Optional dynamic chunking for PP
Default state	Research system	On by default when possible	Check startup log for your version
Cache awareness	Not a focus of the paper	Optional prefix caching	Radix cache with `lpm` ordering

Choosing the Chunk Size: The TTFT-TPOT Frontier

What the budget actually trades

Return to the sweep in the worked example. A 512-token chunk gave a 23 ms worst-case decode gap and a TTFT of 368 ms; a 2,048-token chunk gave an 85 ms gap and a TTFT near 350 ms. The relationship generalizes to three regimes. At very small budgets, per-iteration overheads (kernel launches, scheduling, sampling) dominate and prefill throughput drops, because many tiny iterations replace a few efficient ones. In the middle, iteration time is compute-bound and scales roughly linearly with budget. At very large budgets, you approach the unchunked behavior and the stall returns.

The practical optimum is the largest budget whose iteration time still satisfies your TPOT SLO. That is the Sarathi-Serve recipe and a good default procedure: measure iteration latency as a function of batch tokens on your hardware, choose the budget at the knee below your TPOT target, and re-measure whenever the model, quantization or GPU changes.

Budget interacts with queueing

TTFT is not just prefill. Under load, a new request first waits in the queue. Under a small budget, the prefill capacity per iteration is small, so a burst of prompts queues longer. Consider ten 8,192-token prompts arriving together on the example hardware. Total prefill work is 10 x 330 ms = 3.3 s of compute either way. With a 512-token allotment the engine drains them at about 512 tokens per 23 ms, roughly 22,000 prefill tokens per second, so the last prompt waits about 3.7 s for its first token, a figure that includes the decode work riding along. Chunking did not create this queue; compute scarcity did. What chunking decides is whether decoders suffer during that drain.

This distinction matters for capacity planning. If your TTFT SLO is tight and the arrival pattern is bursty, no chunking policy rescues you; you need more prefill capacity, which leads to disaggregation (below) or simply more replicas. Chunked prefill redistributes pain between TTFT and TPOT but cannot create FLOPs.

Figure 4: A decision flow for choosing among prefix caching, chunked prefill and disaggregated prefill-decode when latency SLOs are missed.

Scheduling order inside the prefill queue

Within the prefill share of the budget, ordering matters. First-come-first-served is fair but lets one huge prompt delay several short ones, which is HOL blocking again, now in the prefill queue. Shortest-job-first style policies reduce mean TTFT but can starve long prompts. Some systems prioritize partially completed prefills so memory held by half-processed prompts is released sooner. Remember that a partially prefilled request already holds KV blocks. Too many concurrent partial prefills can exhaust KV memory and trigger preemption, which costs recomputation. A sound rule is to bound the number of concurrently partial prompts rather than letting the budget scatter across many.

Relation to Prefix Caching and Disaggregated Serving

Chunked prefill does not live alone. Two neighbors, prefix caching and prefill-decode disaggregation, address overlapping problems with different cost structures, and picking the right one is mostly a question of workload shape.

Prefix caching: skip the prefill instead of scheduling it

Prefix caching reuses KV blocks for prompt prefixes that were computed before: system prompts, few-shot examples, shared documents, and previous conversation turns. A cache hit means those tokens skip prefill entirely, which reduces both TTFT and the interference they would have caused. vLLM keys blocks by content hash; SGLang’s RadixAttention stores prefixes in a radix tree. The mechanisms compose with chunking naturally: a cached prefix reduces the number of tokens that need chunking, and chunking then handles the uncached suffix.

The interaction to watch is chunk boundaries and block alignment. KV caches are managed in fixed-size blocks, and only full blocks can be shared by hash. A scheduler that cuts chunks at arbitrary token counts is still correct, but sharing granularity is the block size. Also, prefix-aware ordering such as SGLang’s lpm policy can conflict with fairness: it favors requests that hit the cache, which is great for throughput and hostile to unique prompts under load.

Disaggregation: separate the phases onto different GPUs

DistServe (arXiv 2401.09670) takes the opposite view: if prefill and decode interfere, put them on different GPUs, size and parallelize each independently, and transfer the KV cache between them. Its abstract reports serving up to 7.4 times more requests, or meeting up to 12.6 times tighter SLOs, than existing systems while keeping more than 90 percent of requests within latency constraints. Those are the authors’ reported results on their evaluation setup. Disaggregation removes interference outright and lets you tune tensor-parallel degree separately per phase, at the cost of KV transfer over the interconnect and more complicated cluster management. We cover production implementations in our guide to NVIDIA Dynamo, LMCache and disaggregated serving.

The Sarathi-Serve authors positioned chunked prefill as a way to obtain much of that benefit without the transfer. That is a reasonable engineering view, with a caveat worth stating plainly as opinion: chunking bounds interference but never zeroes it, since decode still shares the iteration with compute-heavy chunk work. For very tight TPOT SLOs on long-prompt workloads, or when you want to scale prefill and decode capacity independently, disaggregation is the cleaner tool. For a single-node or small-cluster deployment, chunking has no transfer cost and no extra moving parts.

Combining all three

The strongest production pattern in 2026 is layered. Prefix caching removes redundant prefill. Chunked prefill bounds the interference from the remaining prefill. Disaggregation then applies at scale when one cluster must serve both strict TTFT and strict TPOT. Even in a disaggregated deployment, chunked prefill is still useful on the prefill workers to keep their own iterations bounded and to interleave very long prompts fairly. Prefill workers that process a 100,000-token prompt monolithically block every other prompt queued on that worker, which is the same HOL problem one level up.

Deeper Analysis: Memory, Kernels and Pipeline Effects

KV memory pressure from partial prefills

A chunk-scheduled prompt holds its KV allocation across many iterations. Early in a prefill, memory is low; by the final chunk it equals the full prompt. A system running many concurrent long partial prefills can therefore hold more KV memory than the same system running them to completion one at a time, because the one-at-a-time schedule frees each prompt’s transient state quickly. Admission control should account for the final KV footprint of a prompt, not its current one. Otherwise, the engine admits a prompt, spends several chunks on it, then must preempt, wasting all that compute.

Using the example layout (128 KiB per token), 20 simultaneous 8,192-token prompts need 20 GiB of KV, which is a significant slice of an 80 GB card after 16 GB of weights. Add 64 decoders at 4,000 tokens of context each, another 32 GiB, and the budget is under pressure. This is arithmetic from assumed values, but it shows why max_num_seqs and KV-aware admission matter as much as the token budget.

Attention kernels and ragged batches

A fused batch of decodes and one prefill chunk is a ragged batch: some sequences contribute one query token against long contexts, one contributes hundreds of query tokens against a growing prefix. Attention kernels must handle both efficiently. Modern backends such as FlashAttention variants and FlashInfer expose paged, variable-length interfaces that accept this mixture in one launch; historically, engines launched separate kernels for prefill and decode parts and summed their latency. The efficiency gain from fusing depends on the kernel path your engine selects, so confirm it in the version you run. Without a fused path, the iteration time is closer to the sum of the parts than the max, and the roofline argument above overstates the benefit.

Pipeline parallelism and uniform micro-batches

With pipeline depth P, a micro-batch that is 3 times slower than its neighbors leaves other stages idle for the difference. Fixed token budgets bound this variance; attention growth with prefix length still introduces drift across chunks, which is why SGLang offers dynamic chunking for pipeline parallelism. Sarathi-Serve reported its largest gains in this regime (5.6 times capacity on Falcon-180B with pipeline parallelism, per the paper). If you run tensor parallelism within a node only, the uniformity argument matters less, and the main benefit is the decode-stall bound.

Speculative decoding and quantization

Two other features change the arithmetic. Speculative decoding verifies several draft tokens per sequence per step, increasing tokens per decode sequence and thus the decode share of the budget; budget accounting must count verification tokens, not one per sequence. Quantization (FP8 or INT4 weights) shrinks the weight bytes, making decode faster and moving the ridge point, so the same budget now yields a different iteration time. Re-profile after any such change; a budget tuned for bf16 will be wrong for FP8.

Trade-offs, Gotchas, and What Goes Wrong

TTFT regression for short prompts under load. Chunking helps long prompts coexist with decoders, but a short prompt behind a decode-heavy batch now competes for a bounded prefill share. If the budget is small and decoders are many, the decode tokens consume most of it, leaving little for prefill. In the extreme, a budget of 256 with 256 running decoders leaves zero prefill tokens and starves new requests entirely. Make sure the budget exceeds max_num_seqs with real headroom.

Overhead of many small iterations. Each iteration has fixed costs: scheduler time, input preparation, kernel launches, sampling and detokenization. At very small budgets, these costs are paid more times per prompt. CUDA graphs, which engines use to cut launch overhead for pure-decode batches, generally apply to fixed shapes, so mixed prefill-decode iterations may not benefit. That is a version-dependent detail; verify against your engine’s behavior with profiling.

Chunk-boundary cost on long contexts. As computed above, KV re-reads scale quadratically with prompt length for fixed chunk size. At 100,000-plus tokens, small chunks can add large fractions of a second. Increase the chunk size for long-context workloads, or scale the chunk with the prefix length.

Metrics that hide the problem. Mean TPOT hides stalls; watch p99 TPOT and the maximum inter-token gap. Likewise, a TTFT average conflates queueing with prefill. Instrument queue wait, prefill time and decode time separately. Without that decomposition, teams tune the wrong knob.

Tuning to a benchmark. A budget chosen on a synthetic benchmark with uniform 1,000-token prompts will not transfer to production mixes with a long tail. Replay real traffic distributions, including the 99th percentile prompt length, before settling on a value.

Interaction with preemption. When KV memory runs out, engines preempt running requests and either swap or recompute them. A half-finished chunked prefill that is preempted loses its work if recomputed. The result is wasted compute at exactly the moment the system is saturated. Avoid it with conservative KV admission rather than relying on preemption as a safety net.

Non-determinism. Chunking changes floating-point reduction order, so outputs can differ in the last bits from an unchunked run, and greedy decoding may diverge on rare ties. This is expected behavior, not a bug, but it affects regression testing; compare with tolerance rather than exact string equality.

Fairness versus throughput. Cache-aware ordering (lpm) and shortest-first policies raise throughput and reduce mean TTFT but can starve unusual requests. Put a ceiling on how long any request can be deferred.

Practical Recommendations

Start from the SLO, not the default. Write down the TTFT and TPOT percentiles you must meet, the prompt-length distribution, and the arrival burstiness. Then work through the controls in order of their blast radius.

First, turn on prefix caching if your workload has shared prefixes, since it is the cheapest way to cut prefill work. Second, keep chunked prefill enabled, which is the default in vLLM V1 when possible, and set the token budget from a measured iteration-time curve: pick the largest budget whose iteration time sits under your TPOT target. The vLLM docs offer 2048 as an example of a latency-oriented setting and values above 8192 for throughput-oriented ones; treat those as starting points for your sweep, not answers. Third, size max_num_seqs and KV capacity so decodes never crowd out the prefill share. Fourth, if TTFT remains out of SLO because of raw prefill load, add prefill capacity or evaluate disaggregation instead of shrinking the budget further.

A short checklist you can run on any deployment:

Log queue wait, prefill time and decode time separately, and track p99 TPOT and maximum inter-token gap.
Sweep the token budget on replayed production traffic and plot TTFT and TPOT percentiles against it.
Confirm the budget exceeds the maximum concurrent decode count by a healthy margin.
Check flag names with --help on the exact engine version; do not copy from older posts.
Re-profile after changing model, quantization, GPU type, tensor-parallel degree, or enabling speculative decoding.
Cap concurrent partial prefills and use KV-aware admission to avoid recompute-causing preemption.
Reserve disaggregation for cases where TTFT and TPOT SLOs both bind and interconnect bandwidth is available.

Frequently Asked Questions

What is chunked prefill in LLM inference?

Chunked prefill splits a long prompt’s prefill into smaller pieces, each processed in a different scheduler iteration, and batches each piece together with ongoing decode requests. Because every iteration processes a bounded number of tokens, running users keep receiving output tokens instead of waiting for the entire prompt to finish. It was formalized in the SARATHI and Sarathi-Serve papers and is now enabled by default in vLLM V1 whenever possible.

Does chunked prefill change model outputs?

Not mathematically. Causal attention means each chunk attends to the cached keys and values of earlier chunks plus its own earlier tokens, which equals a monolithic prefill. In practice, floating-point reduction order differs, so tiny numerical differences are possible and greedy outputs can occasionally diverge on near-ties. Treat this as normal numerical variation and use tolerant comparisons in your regression tests rather than expecting bit-identical strings.

How do I choose the chunk size or token budget?

Pick the largest per-iteration token budget whose measured iteration time still meets your TPOT target, since larger budgets improve prefill throughput and TTFT. The vLLM docs describe smaller values such as 2048 as better for inter-token latency and values above 8192 as better for throughput. Always sweep on replayed production traffic, and re-tune after changing model, quantization or hardware, because the iteration-time curve shifts.

Does chunked prefill increase time to first token?

It can, modestly, for the request being chunked, because each of its iterations also carries decode work and its prefill is spread over more passes. In the illustrative example above, TTFT rose about 12 percent while the worst-case decode stall fell roughly 14-fold. Under bursty load, queueing usually dominates TTFT, and that is a capacity problem chunking cannot solve.

Chunked prefill or disaggregated prefill: which should I use?

Start with chunked prefill: it has no KV transfer cost and no cluster changes, and it bounds most interference. Choose disaggregation when both TTFT and TPOT SLOs are strict, when you want to scale prefill and decode independently, or when long-prompt traffic still violates TPOT after tuning. DistServe reports large goodput gains from disaggregation, but on its own evaluation setup; validate on your workload and interconnect.

Does chunked prefill work with prefix caching?

Yes, and the two compose well. Prefix caching removes the cached portion of a prompt from prefill entirely, and chunked prefill then schedules the remaining uncached suffix. Be aware that sharing happens at KV block granularity, and cache-aware queue ordering such as SGLang’s longest-prefix-match policy can favor hits over unique prompts under load, so monitor tail TTFT for uncached requests.