Continuous Batching for LLM Inference: Architecture and Throughput

A single A100 running a 13B model with naive request-at-a-time serving might push 30–40 tokens per second and sit 90% idle. The same GPU, same model, under continuous batching llm inference, can serve dozens of concurrent users and drive aggregate throughput more than an order of magnitude higher — without buying a single extra card. That gap is not a kernel optimization or a quantization trick. It is a scheduling decision: whether you batch work at the granularity of a whole request or at the granularity of a single decode iteration.

This matters now because inference, not training, is where most production GPU dollars go. The economics of an LLM endpoint are dominated by how many concurrent sequences you can keep on the card and how full each forward pass runs. Get the scheduler wrong and you pay for silicon that computes padding.

What this covers: why static batching wastes GPUs, how iteration-level scheduling reclaims that waste, how it interacts with the KV cache and paged attention, the TTFT-versus-TPOT trade-off, admission control and preemption, and how vLLM, TensorRT-LLM, SGLang, and TGI implement the pattern in 2026.

Context and Background

Autoregressive text generation has an awkward shape for hardware. A transformer forward pass is a dense, GPU-friendly burst of matrix multiplies. But decoding emits one token at a time, and each token depends on all the tokens before it. So generation is a long sequence of small forward passes, each processing a single new position per active request. On a modern GPU, one such pass for one request uses a sliver of the available compute. The card is memory-bandwidth bound, waiting on weights and KV cache reads, while thousands of tensor cores idle.

The classical fix is batching: run many requests through the same forward pass so the matrix multiplies get fat and the fixed cost of streaming weights from HBM is amortized across more useful work. This is standard practice everywhere in deep learning serving, and for fixed-shape workloads like image classification it is nearly optimal. You collect a batch, pad to a common shape, run it, return results.

LLMs break the assumption that made static batching clean. Requests arrive at unpredictable times, prompts have wildly different lengths, and — crucially — outputs have different and unknown lengths. You cannot know in advance that one request will emit 12 tokens and another 900. A batch of requests grouped at submission time is a batch of sequences that finish at wildly different iterations. The incumbent serving stacks before 2022 (early FasterTransformer deployments, TorchServe wrappers, bespoke Flask loops) largely inherited static batching from vision, and paid for it in idle silicon. The pivot came with Orca (OSDI 2022), which introduced iteration-level scheduling, and with vLLM (SOSP 2023), which paired it with PagedAttention to solve the memory-management half of the problem. Those two papers define the architecture nearly every serious LLM server ships today. For the memory side of the story, see our companion piece on KV cache optimization for LLM inference; the canonical source is the vLLM PagedAttention paper.

Why Static Batching Wastes the GPU

Continuous batching is a solution, so start with the problem it solves. Static batching — sometimes called request-level batching — groups a set of requests, runs them together from first prefill token to last decode token, and only then admits the next group. Two structural inefficiencies fall out of that choice, and both are severe under realistic traffic.

Figure 1: Static batching pads a batch to its longest sequence and blocks new work until every member finishes; continuous batching schedules at the token level, freeing and refilling slots as sequences complete.

Figure 1 contrasts the two regimes. On the static side, four requests enter together, the batch is padded to the longest sequence, all four run to completion, and the short sequences occupy slots long after they have produced their final token. New arrivals wait behind the whole batch. On the continuous side, the scheduler treats each decode iteration as the unit of work: a finished sequence leaves immediately, its slot is handed to a waiting request, and the batch composition changes every step.

Head-of-line blocking freezes short requests behind long ones

In static batching, throughput and latency are hostage to the longest member of the batch. Imagine a batch where three requests need 20 output tokens and one needs 800. All four occupy the GPU for 800 decode iterations. The three short requests finished at iteration 20 but cannot return to the client, and their batch slots cannot be reused, until the long request completes at iteration 800. That is head-of-line blocking. A user asking for a one-line answer waits for a neighbor writing an essay.

Worse, any request that arrives at iteration 1 must wait for the entire current batch to drain before it can even start prefill. Under bursty traffic — which is every real chat workload — this inflates tail latency dramatically. The 99th-percentile time-to-first-token becomes a function of the longest generation currently in flight, which the scheduler cannot predict.

Padding turns ragged lengths into wasted FLOPs

The second waste is spatial rather than temporal. To run a batch as one tensor, sequences must share a shape. Static batching pads shorter prompts up to the longest prompt in the batch, and during decode the finished-but-still-resident sequences continue to consume a batch row. Every padded position is a full column of matrix multiply that produces nothing useful. If prompt lengths in a batch range from 50 to 2,000 tokens, the prefill tensor is sized for 2,000 and the 50-token request wastes 97.5% of its rows.

Consider an illustrative example (numbers illustrative, chosen to show the mechanism). Suppose a batch of 8 has an average useful occupancy of 40% once you account for length skew and early-finishing sequences. Then even if each forward pass runs at peak kernel efficiency, your effective throughput is 40% of the batch’s nominal capacity. The GPU is busy; it is just busy computing padding and running spent sequences. Static batching can keep utilization counters high while delivering a fraction of the achievable tokens per second.

The combined effect: static batching forces a bad choice between small batches (low latency, low throughput, poor GPU use) and large batches (better amortization but brutal head-of-line blocking and worse tails). There is no batch size that is simultaneously good on both axes, because the batch is the wrong unit of scheduling.

The Reference Architecture: Iteration-Level Scheduling

Continuous batching schedules at the granularity of a single decode iteration rather than a whole request. After every forward pass, the server re-evaluates which sequences are still active, evicts the ones that just emitted a stop token, and pulls waiting requests into the freed slots — so the running batch is recomposed each step and the GPU rarely runs a sequence that has nothing left to do. This is the entire idea, and everything else is machinery to make it efficient.

The pattern goes by several names that mean the same thing. Orca called it iteration-level scheduling. Practitioners say continuous batching or in-flight batching (NVIDIA’s term in TensorRT-LLM). Some docs say dynamic batching, though that term is overloaded — in Triton it historically meant server-side request coalescing, which is not the same as recomposing a batch every token. Whatever the label, the defining property is: the scheduler runs a loop, and each loop iteration corresponds to one model forward step, not one request.

Figure 2: The continuous batching scheduler loop. Each new request reserves KV cache blocks and runs a prefill; then every decode iteration batches all ready sequences, streams one token each, evicts finished sequences, and admits new ones into freed slots.

Figure 2 traces the loop. A request submits a prompt; the scheduler reserves KV cache space and runs prefill, which produces the first token and reports time-to-first-token. Then the decode loop runs: each iteration batches every ready sequence, executes one forward step, streams one new token per sequence to its client, evicts any sequence that hit its stop condition, and admits new sequences into the vacated slots. The batch is a living set, not a fixed cohort.

Orca’s two contributions: iteration scheduling and selective batching

Orca (Yu et al., OSDI 2022) named the problem and gave it two mechanisms. The first is iteration-level scheduling itself — return control to the scheduler after each token so admission and eviction happen at token granularity. The second, subtler contribution is selective batching. The trouble with recomposing a batch every iteration is that different sequences are at different positions, so their tensors have different shapes; a naive implementation cannot batch them.

Orca’s insight is that the operations in a transformer layer split into two classes. The token-wise operations — the QKV projections, the feed-forward MLP, the output projection — are the same computation applied independently per token, so they can be flattened across all sequences into one big matrix multiply regardless of each sequence’s length. Only attention itself is sequence-specific, because each token attends to its own history. Orca batches the token-wise ops together and handles attention per sequence. That is selective batching, and it is what makes a mixed-length, mixed-position batch runnable as one efficient forward pass. The result reported in the Orca paper was a large throughput improvement at matched latency versus FasterTransformer-style static serving; treat the exact multiplier as workload-dependent and consult the Orca paper directly for its methodology.

Prefill and decode are two different workloads

To schedule an LLM well you must see that generation has two phases with opposite hardware profiles, and the scheduler juggles both.

Prefill processes the entire prompt in one pass. Every prompt token is known up front, so prefill is a big parallel matrix multiply over N positions — compute-bound, tensor-core-hungry, and efficient. A 1,000-token prompt is one fat forward pass that fills the GPU.

Decode processes one new token per sequence per step. It is memory-bandwidth bound: the arithmetic is tiny (a single new position), but you must stream the entire weight matrix and the growing KV cache from HBM on every step. Decode is where the GPU sits underused unless you batch many sequences together so the streamed weights are amortized across many concurrent tokens.

This asymmetry is why continuous batching is worth so much on decode specifically. Batching decode steps across, say, 64 sequences turns 64 tiny memory-bound passes into one pass that reads the weights once and does 64 tokens of useful work. The weights get amortized; throughput scales with batch size until you hit a memory-bandwidth or KV-capacity wall. Prefill, already compute-bound, benefits less from batching and can even hurt latency when a big prefill barges into a stream of decodes — a tension we return to under chunked prefill.

The batch size that matters is dynamic, not fixed

Under continuous batching there is no single batch size. There is a running set whose membership changes every iteration and whose maximum is governed by an admission limit (vLLM’s max_num_seqs) and by KV cache capacity. The scheduler’s job each step is to run the largest batch it can that respects the memory budget and the latency SLOs. When many sequences are short, more fit; when sequences accumulate long histories, KV pressure forces the batch smaller. The system self-tunes toward high occupancy, which is exactly what static batching could not do.

Deeper Analysis: KV Cache, Paged Attention, and Chunked Prefill

Iteration-level scheduling recovers the time and shape waste, but it creates a memory-management problem that is just as hard. The moment you let sequences enter and leave a batch arbitrarily, you need to allocate and free their KV cache arbitrarily — and the KV cache is the single largest and most volatile consumer of GPU memory in LLM serving.

Why the KV cache is the binding constraint

Every token a transformer has processed leaves behind a key and value vector in every layer, cached so future tokens can attend to it without recomputation. That cache grows linearly with sequence length and with the number of concurrent sequences. For a model with L layers, H KV heads, head dimension d, in 16-bit precision, the KV cache per token is roughly 2 (key and value) × L × H × d × 2 bytes. For a 13B-class model that lands on the order of hundreds of kilobytes to around a megabyte per token, depending on architecture and whether grouped-query attention shrinks the KV head count. Multiply by thousands of tokens across dozens of sequences and the KV cache, not the weights, is what fills the card. Deployment cost math follows directly from this; we work the numbers in the vLLM cost economics deep dive.

Because output length is unknown, you cannot pre-size a sequence’s cache. Classic serving reserved a contiguous buffer sized to the maximum possible length for every sequence — catastrophic waste, since most sequences use a fraction of it, and that reserved-but-unused space is memory you cannot lend to another sequence. The vLLM paper measured that this internal and external fragmentation left large fractions of KV memory unusable in practice, capping how many sequences could co-reside.

PagedAttention: virtual memory for the KV cache

vLLM’s PagedAttention borrows the operating-system idea of paging. Instead of one contiguous KV buffer per sequence, the cache is split into fixed-size blocks (say, 16 tokens each). A per-sequence block table maps the sequence’s logical token positions to physical blocks scattered anywhere in a shared GPU pool — exactly like page tables mapping virtual to physical memory.

Figure 3: PagedAttention maps each sequence’s logical KV blocks through a block table into a shared, non-contiguous physical pool. When free blocks run out the scheduler preempts a sequence, either recomputing its KV later or swapping it to CPU memory.

Figure 3 shows the indirection. Three consequences fall out, all of which make continuous batching practical:

First, no over-reservation. A sequence holds exactly the blocks it has filled, plus at most one partly filled block. Internal fragmentation drops to under one block per sequence, so far more sequences fit in the same HBM and the running batch can be much larger.

Second, blocks are allocated and freed at token-block granularity, which is precisely the churn that iteration-level scheduling demands. Admitting and evicting sequences every step becomes a matter of updating block tables, not moving or reallocating large buffers.

Third, blocks can be shared. Two requests with a common prompt prefix — a shared system prompt, a few-shot preamble, a branching beam — can point their block tables at the same physical blocks with copy-on-write semantics. This is the foundation of prefix caching, which cuts prefill cost for repeated prefixes and is a major lever behind semantic and prefix caching architectures.

Chunked prefill: stop letting big prompts stall decode

There is a scheduling conflict between the two phases. A long prefill (say a 4,000-token prompt) is one heavy forward pass. If it runs as a monolithic step, every sequence currently decoding stalls for the duration — a visible latency spike, a stutter in every open stream. Naive continuous batching that prioritizes prefills starves decodes; one that prioritizes decodes starves prefills and inflates time-to-first-token.

Chunked prefill resolves it by splitting a long prefill into fixed-size token chunks and interleaving those chunks with ongoing decode steps in the same batch. Instead of one 4,000-token prefill pass, the scheduler runs, for example, chunks of 512 prefill tokens, and it can co-schedule decode tokens from other sequences alongside each chunk to fill the batch’s token budget. The effect is twofold: decode latency stops spiking because no single step is dominated by one giant prefill, and GPU utilization stays high because compute-bound prefill chunks and memory-bound decode tokens are mixed into a balanced pass. vLLM and SGLang both expose this, and vLLM has made variants of chunked prefill the default scheduling behavior in recent releases; see the vLLM optimization docs for the current knobs.

A short, illustrative intuition for why mixing helps (numbers illustrative): a pure-decode step might use 20% of the GPU’s compute while saturating memory bandwidth, and a pure-prefill step the reverse. Interleave them and each step can approach full utilization on both resources, because the prefill chunk supplies the arithmetic intensity that decode lacks and decode supplies the concurrency that a lone prefill chunk does not need.

Trade-offs, Gotchas, and What Goes Wrong

Continuous batching is close to a free lunch on throughput, but it introduces real operational hazards. Ignoring them produces a server that benchmarks beautifully and falls over in production.

The throughput-latency knee, and how max-num-seqs sets it

The core tension is TTFT versus TPOT — time-to-first-token versus time-per-output-token, the two latency numbers users actually feel. Raising the admission ceiling (max_num_seqs) packs more sequences into every decode step, lifting aggregate token throughput. But a larger batch makes each decode step take longer, so per-user TPOT rises and streams feel slower. Push admission too high and you also lengthen the queue that new prompts wait behind, inflating TTFT. There is a knee: below it, adding sequences buys throughput almost for free; above it, throughput flattens while both latency metrics degrade. The right operating point is workload- and SLO-specific, and it is the single most important tuning decision. Set max_num_seqs and the token budget so the batch sits just below the knee for your traffic, not at the theoretical maximum.

Memory pressure, preemption, and the recompute-versus-swap choice

Because output length is unknown, the scheduler can over-admit: it accepts sequences that later grow long enough that their combined KV cache exceeds capacity. When free blocks run out mid-generation, the scheduler must preempt a sequence to free memory, then resume it later. There are two recovery strategies, both shown in Figure 3.

Recompute (the common default) simply drops the preempted sequence’s KV cache and, when it is rescheduled, re-runs prefill over its prompt-plus-generated tokens to rebuild the cache. It costs compute but no extra memory traffic to CPU, and prefill is efficient, so for shorter contexts it is often cheaper.

Swap moves the preempted sequence’s KV blocks to CPU memory over PCIe or the interconnect and copies them back on resume. It costs no recomputation but pays bandwidth, and for long contexts the copy can be slower than just recomputing. The right choice depends on context length and interconnect speed. The failure mode to watch is thrashing: if admission is too aggressive relative to real memory, the scheduler spends its time preempting and resuming rather than making forward progress, and effective throughput collapses. Conservative admission control prevents it.

Prefill-decode disaggregation: when colocation is the wrong answer

Colocating prefill and decode on one engine — even with chunked prefill — means one GPU pool tuned as a compromise between a compute-bound and a memory-bound workload. At scale, some operators split them.

Figure 4: Colocated serving runs prefill and decode on the same GPU and uses chunked prefill to protect decode latency; disaggregated serving runs separate prefill and decode pools, shipping the KV cache between them, so each pool is tuned for its own bottleneck.

Figure 4 contrasts the two. Prefill/decode disaggregation runs prefill on one pool of GPUs and decode on another, transferring the KV cache from the prefill node to the decode node over a fast fabric. Each pool is then tuned for its own bottleneck: prefill nodes optimize for compute and TTFT, decode nodes optimize for memory capacity and TPOT, and neither interferes with the other. Systems like DistServe and Mooncake demonstrated meaningful gains from this split, and it has moved into production stacks and vLLM’s roadmap. The cost is architectural: you now move gigabytes of KV cache across the network per request, so disaggregation only pays when the interconnect is fast enough (NVLink, InfiniBand, or comparable) and the traffic is large and steady enough to keep both pools busy. For small deployments it is over-engineering; colocation with chunked prefill is simpler and usually sufficient.

Other sharp edges

Fairness is not automatic: a naive first-come scheduler can let a flood of long generations starve short interactive requests, so production servers add priority or fair-queuing policies on top of the loop. Streaming semantics complicate cancellation — a client that disconnects mid-generation should free its slot promptly, or you leak KV capacity. And benchmark honesty matters: a throughput number quoted at a batch size that violates your latency SLO is meaningless. Always report throughput at a fixed TTFT/TPOT bound, not the peak tokens-per-second the hardware can emit when latency is ignored.

Practical Recommendations

Start from a server that implements continuous batching natively rather than bolting it onto a training-style inference loop. In 2026 that means vLLM, NVIDIA TensorRT-LLM (its “in-flight batching”), SGLang, or Hugging Face TGI. All four schedule at the iteration level; they differ in kernel maturity, quantization support, and scheduler sophistication, not in the core pattern.

Tune in this order. First, size the KV cache — give the engine as much GPU memory for cache as you safely can, because KV capacity, not compute, usually caps concurrency. Second, set the admission ceiling (max_num_seqs and the token budget) to sit just below the throughput-latency knee for your traffic; find it by sweeping load and watching TTFT and TPOT, not just tokens per second. Third, enable chunked prefill if long prompts are causing decode stutter. Fourth, enable prefix caching if your workload has shared system prompts or repeated preambles — it is often the largest single win for chat and RAG. Only consider prefill/decode disaggregation once you are multi-node, interconnect-rich, and have proven that colocation is the bottleneck.

A minimal vLLM configuration that turns on the levers discussed above:

from vllm import LLM, SamplingParams

# Continuous batching is always on in vLLM; these args tune it.
llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    gpu_memory_utilization=0.90,   # more HBM for KV cache = higher concurrency
    max_num_seqs=256,              # admission ceiling: cap on concurrent sequences
    max_num_batched_tokens=8192,   # per-step token budget (prefill + decode)
    enable_chunked_prefill=True,   # interleave long prefills with decode
    enable_prefix_caching=True,    # reuse KV for shared prompt prefixes
    kv_cache_dtype="fp8",          # shrink KV cache to fit more sequences
)

params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(["Explain continuous batching in one paragraph."], params)
print(outputs[0].outputs[0].text)

Checklist before you call a deployment tuned:

[ ] Server schedules at iteration level (vLLM / TRT-LLM in-flight / SGLang / TGI), not request level.
[ ] gpu_memory_utilization set high enough that KV cache — not weights — is the concurrency limit.
[ ] max_num_seqs and token budget swept to the knee, with throughput reported at a fixed TTFT/TPOT SLO.
[ ] Chunked prefill enabled if long prompts stall decode streams.
[ ] Prefix caching enabled for shared system prompts or RAG preambles.
[ ] Preemption strategy (recompute vs swap) chosen for your context lengths; admission conservative enough to avoid thrashing.

Frequently Asked Questions

What is the difference between continuous batching and dynamic batching?

Continuous batching (iteration-level scheduling) recomposes the running batch after every decode step, admitting and evicting sequences at token granularity. Classic dynamic batching, as in NVIDIA Triton’s dynamic batcher, coalesces incoming requests into a batch at submission time, then runs that batch to completion — it is still request-level. The terms get conflated because both are “dynamic,” but only continuous batching returns control to the scheduler between tokens, which is what eliminates head-of-line blocking and padding waste for autoregressive generation.

Does continuous batching increase per-user latency?

It can, and the trade-off is controllable. Packing more sequences into each decode step raises aggregate throughput but lengthens each step, so per-token latency (TPOT) rises with batch size. Below the throughput-latency knee the effect is small; above it, latency degrades sharply for little throughput gain. You manage this with the admission ceiling (max_num_seqs) and token budget, tuning to your SLO. Chunked prefill further protects decode latency by preventing large prompts from stalling active streams.

How does continuous batching relate to PagedAttention and the KV cache?

They are complementary halves. Continuous batching solves the time-and-shape waste by scheduling per iteration; PagedAttention solves the memory-management problem that iteration-level admission creates. Because sequences enter and leave the batch every step, their KV cache must be allocated and freed dynamically. PagedAttention stores the KV cache in fixed-size blocks mapped through per-sequence block tables, so allocation is cheap, fragmentation is near zero, and shared prefixes can be deduplicated. Without paging, dynamic admission would fragment memory badly and cap concurrency.

What is prefill/decode disaggregation and when should I use it?

Disaggregation runs the compute-bound prefill phase on one GPU pool and the memory-bound decode phase on another, shipping the KV cache between them over a fast interconnect. Each pool is tuned for its own bottleneck, so prefill and decode stop interfering. Use it when you are multi-node with NVLink/InfiniBand-class fabric and steady, high-volume traffic that keeps both pools busy. For single-node or bursty workloads it adds KV-transfer cost and complexity without payback; colocation with chunked prefill is simpler and usually enough.

Do vLLM, TensorRT-LLM, SGLang, and TGI all implement continuous batching the same way?

They share the core pattern — iteration-level scheduling with selective batching and a paged KV cache — but differ in the details. vLLM originated PagedAttention and has a mature scheduler with chunked prefill and prefix caching. NVIDIA TensorRT-LLM calls it in-flight batching and pairs it with heavily optimized fused kernels. SGLang adds RadixAttention for aggressive prefix sharing across requests. TGI provides a production-hardened server with continuous batching built in. Choose on kernel and quantization support, ecosystem fit, and scheduler features — not on the batching pattern, which is common to all.

Can continuous batching help if I only ever serve one request at a time?

Not much. The throughput win comes from amortizing weight streaming across many concurrent sequences in each memory-bound decode step. With a single active request there is nothing to batch, so decode stays memory-bandwidth bound and the GPU stays underused. You would instead reach for latency-oriented techniques: speculative decoding, quantization, tensor parallelism, or a smaller model. Continuous batching’s value scales with concurrency; it is a throughput optimization for multi-tenant or high-QPS endpoints, not a single-stream latency trick.

Continuous Batching for LLM Inference: Architecture and Throughput (2026)