vLLM vs SGLang vs TensorRT-LLM in 2026: The Serving Engine Pick

Picking a serving engine used to be a coin flip between “whatever the model card recommends” and “whatever the last blog post said was fastest.” That doesn’t hold up anymore. In 2026, vllm vs sglang vs tensorrt-llm is a decision with real, measurable consequences for cost per million tokens, tail latency under bursty traffic, and how much engineering time you spend babysitting a compilation pipeline instead of shipping features. All three engines have converged on the same baseline — continuous batching, paged KV-cache memory, FP8/FP4 quantization — which means the differentiators are no longer “does it support the basics” but “how does it behave at your actual concurrency, your actual prefix-reuse ratio, and your actual hardware budget.” What this covers: the core memory-management mechanism behind each engine (PagedAttention, RadixAttention, ahead-of-time kernel compilation), how continuous batching and quantization differ in practice, a results table with methodology caveats, the trade-offs and failure modes engineers actually hit in production, and a decision framework organized by workload shape rather than by marketing claims.

Context and Background

The LLM serving landscape split into three durable camps because each optimized for a different bottleneck. vLLM emerged from UC Berkeley’s Sky Computing Lab in 2023 with a single insight: KV-cache memory was being wasted through internal and external fragmentation the same way early operating systems wasted physical RAM before virtual memory. PagedAttention fixed that by paging the KV cache into fixed-size blocks, and the ecosystem effect — HuggingFace-native model loading, OpenAI-compatible APIs, day-one support for nearly every open-weight architecture — made vLLM the default choice for teams that need broad model coverage without engineering overhead.

SGLang, from the same academic lineage (LMSYS/SGLang team, later spun into an independent project), targeted a different bottleneck: redundant prefill computation. Multi-turn chat, agentic tool-calling loops, and RAG pipelines all re-send large shared prefixes — system prompts, tool schemas, retrieved documents — on every request. RadixAttention indexes the KV cache in a radix tree so those shared prefixes are computed once and reused, which is why SGLang wins decisively on prefix-heavy traffic and only marginally on traffic with no shared structure.

TensorRT-LLM took the opposite path: instead of a general-purpose Python-first serving stack, NVIDIA built a compiler. It ahead-of-time fuses and optimizes CUDA kernels for a specific model, precision, and GPU target, trading flexibility for the highest achievable tokens/sec on NVIDIA silicon. For architecture and infrastructure background on how these engines fit into a broader serving stack, see our related coverage of continuous batching in LLM inference architecture, which underlies all three engines discussed here.

By 2026, the three projects have also converged organizationally in ways that matter for procurement and support decisions. vLLM sits under the PyTorch Foundation with backing from a broad multi-vendor coalition, which makes it the closest thing the ecosystem has to a neutral standard — useful if you’re wary of single-vendor lock-in or you serve on mixed hardware (AMD MI300-class accelerators, in addition to NVIDIA GPUs, are increasingly first-class vLLM targets). SGLang is stewarded by the SGLang project team with heavy production usage inside frontier labs and inference clouds, and its release cadence tracks new attention-kernel research closely — RadixAttention was itself a research contribution before it became production infrastructure. TensorRT-LLM is NVIDIA-owned and NVIDIA-prioritized, which is exactly what you want if your fleet is homogeneous H100/H200/B200 and exactly what you don’t want if you need portability across vendors or need to run the same serving stack on cloud instances that mix accelerator types. None of these governance differences show up in a benchmark chart, but they show up in a two-year total cost of ownership model, and they’re worth weighing alongside raw throughput.

PagedAttention vs RadixAttention vs Compiled Kernels: The Core Mechanisms

Direct answer: vLLM’s PagedAttention pages KV-cache memory into fixed 16-token blocks to eliminate fragmentation and enable near-100% memory utilization; SGLang’s RadixAttention additionally indexes those blocks in a shared radix tree so identical token prefixes across requests are computed once and reused; TensorRT-LLM instead compiles a fused, static execution graph ahead of time for a specific model/precision/GPU combination, sacrificing runtime flexibility for the lowest per-token latency achievable on that hardware.

These are not competing implementations of the same idea — they’re solutions to three different problems that happen to intersect at “make GPU inference faster.” Understanding which problem each one solves is what makes the workload-fit decision later in this piece tractable instead of arbitrary.

Figure 1: PagedAttention allocates non-contiguous, fixed-size KV-cache blocks per sequence and frees them on completion, similar to OS virtual memory paging. RadixAttention builds on the same block-based allocator but indexes blocks in a shared radix tree keyed by token sequence, so two requests with an identical 2,000-token system prompt point at the same physical cache blocks instead of each allocating and computing their own copy. TensorRT-LLM (not pictured as a tree) uses a simpler paged KV-cache manager but gains its advantage from ahead-of-time kernel fusion rather than cache topology.

How PagedAttention Actually Manages Memory

Before PagedAttention, most serving frameworks pre-allocated the maximum possible sequence length as contiguous memory per request — a request that generates 50 tokens still reserved space for 4,096. That wasted 60-80% of KV-cache memory in typical production traffic. PagedAttention borrows the OS virtual-memory trick directly: it splits the cache into fixed-size blocks (16 tokens is the vLLM default), maintains a block table per sequence mapping logical to physical blocks, and allocates new blocks only as generation proceeds. Blocks are freed immediately when a sequence finishes, and near-100% of allocated memory is provably in use at any point, versus roughly 20-40% under naive pre-allocation. This is the mechanism that let vLLM claim 2-4x higher throughput than the serving stacks that predated it — not smarter attention math, just better memory bookkeeping.

How RadixAttention Extends the Idea

RadixAttention keeps PagedAttention’s block-based allocator but adds a global index: a radix tree where each path from root to a node represents a token sequence, and each node holds a pointer to the physical KV-cache blocks for that prefix. When a new request arrives, the scheduler walks the tree to find the longest matching prefix already cached — across different requests, not just within one — and only computes the suffix that diverges. For a multi-turn agent session where a 3,000-token tool-definition block precedes every call, this turns what would be a full prefill into a cache hit plus a short incremental computation. SGLang’s own published benchmarks report up to a 6.4x throughput gain in workloads engineered to maximize prefix sharing; that number is a ceiling for high-overlap traffic, not a general expectation.

TensorRT-LLM’s Ahead-of-Time Compilation Trade

TensorRT-LLM doesn’t build a shared-prefix cache index at all in its core engine path. Instead, it takes the model graph and, for a declared target GPU, precision, and batch-size range, fuses operators (attention, layer norm, MLP projections) into optimized CUDA kernels and locks in the execution plan before serving starts. That removes dispatch overhead, maximizes tensor-core utilization for the specific precision (FP8 on Hopper, FP4/NVFP4 on Blackwell), and is why it wins on raw sustained tokens/sec — reported in the 15-30% range over vLLM on H100s in several independent 2026 benchmarks. The cost is that a compiled engine is tied to the model checkpoint, precision, and hardware target it was built for; swap any of the three and you rebuild.

It’s worth being precise about where that compiled advantage actually comes from, because it’s frequently misattributed to “NVIDIA writes better kernels,” which understates the real mechanism. Python-first schedulers like vLLM’s and SGLang’s pay a per-step dispatch cost: launching each kernel involves Python-level orchestration, even with CUDA graphs capturing much of the repeated work. TensorRT-LLM’s build step captures the entire forward pass — attention, normalization, projections, even some scheduling logic — into a fixed graph with no interpreter in the loop. On short-decode-step workloads where the per-token compute is tiny relative to dispatch overhead, that fixed-graph advantage compounds because there’s simply less overhead to amortize away. On long-prefill-dominated workloads, the gap narrows because the actual matrix multiplication time dwarfs dispatch overhead regardless of which engine is issuing it. This is also why TensorRT-LLM’s advantage is most visible in the “many short generations, high QPS” scenario and least visible in “few long generations” scenarios — a distinction none of the headline throughput numbers usually make explicit.

Batching, Quantization, and Deployment Ergonomics

Direct answer: all three engines now implement continuous (iteration-level) batching as a baseline, so that is no longer a differentiator; the real gaps are in how much precision flexibility each exposes at runtime versus compile time, and how much operational tooling (autoscaling hooks, multi-LoRA, structured output) ships out of the box.

Continuous batching itself has become table stakes — vLLM popularized it, SGLang inherited it, and TensorRT-LLM added its in-flight batching equivalent years ago. What differs in 2026 is scheduling granularity and how gracefully each engine handles heterogeneous request lengths mixed with long-context prefill.

Figure 2: All three engines schedule at the iteration level rather than the request level — new requests join a running batch as soon as GPU capacity frees up, instead of waiting for the whole batch to drain. vLLM (V1) defaults to an 8,192-token chunk size for chunked prefill; SGLang layers its radix-tree cache lookup into the same scheduling loop; TensorRT-LLM’s in-flight batching operates inside the compiled execution graph, giving it less runtime scheduling flexibility but lower per-step dispatch overhead.

Quantization is where the engines diverge more sharply. vLLM supports the widest matrix of formats at runtime — GPTQ, AWQ, FP8, and increasingly FP4 — loadable without a separate compile step, which matters when you’re iterating on a model or serving many different checkpoints from one fleet. SGLang tracks vLLM closely here since it shares much of the same kernel ecosystem (both lean on FlashInfer and cutting-edge open kernels). TensorRT-LLM’s quantization is the most mature on raw numbers — NVIDIA’s Model Optimizer toolchain produces FP8 and NVFP4 checkpoints specifically tuned for Hopper and Blackwell tensor cores, and Blackwell’s roughly 18,000 sparse FP4 TFLOPS versus roughly 9,000 sparse FP8 TFLOPS is a real hardware ceiling the compiled path is built to hit — but every precision change means re-running the build pipeline, which can take anywhere from minutes to over an hour depending on model size.

Multi-LoRA and Structured Generation

If you’re serving many fine-tuned adapters off one base model, vLLM’s multi-LoRA support (dynamic adapter swapping per request) and SGLang’s structured-generation constraints (JSON-schema-guided decoding integrated into the scheduler) are meaningfully ahead of TensorRT-LLM’s LoRA story, which exists but is less dynamic given the compiled-graph model. SGLang in particular was built with agentic and structured-output workloads as a first-class use case — its constrained decoding is implemented at the token-mask level inside the same scheduler that handles RadixAttention, so you don’t pay a separate latency tax for enforcing a JSON schema.

Ecosystem and Model Coverage

vLLM ships day-one or near-day-one support for the overwhelming majority of new open-weight model releases, backed by the largest contributor base of the three. SGLang’s model coverage has closed most of the gap but still typically trails vLLM by days to weeks for brand-new architectures. TensorRT-LLM supports a curated, NVIDIA-validated model list — smaller, but each entry is deeply optimized; a model outside that list may need a custom conversion path or may not be supported at all.

Observability and Autoscaling Integration

Serving engine choice also determines how much glue code your platform team writes. vLLM and SGLang both expose Prometheus-compatible metrics endpoints out of the box — queue depth, KV-cache utilization, time-to-first-token percentiles — that plug directly into standard Kubernetes horizontal-pod-autoscaler setups keyed on custom metrics rather than raw CPU/GPU utilization, which is a poor proxy for LLM serving load. TensorRT-LLM’s metrics surface is comparably rich when run through NVIDIA’s Triton Inference Server as the serving frontend (the common production pattern), but that adds Triton as an additional operational layer to learn and maintain on top of the compiled engine itself. Teams that already run Triton for other model types (vision, recommendation) find this a wash or even a net simplification; teams new to the NVIDIA stack find it a meaningful onboarding cost layered on top of the compile pipeline.

Results Table and Methodology Notes

The table below synthesizes patterns from multiple independent 2026 benchmark write-ups (RunPod, Spheron Network, and vendor-published numbers). These are illustrative figures, not numbers we measured ourselves — hardware, model, batch size, sequence length, and precision all move these numbers substantially, and different publications report meaningfully different absolute values even for similar setups. Treat directionality (which engine wins under which condition) as more reliable than the precise magnitudes.

Scenario (illustrative)	vLLM	SGLang	TensorRT-LLM	Notes
Throughput, unique prompts, Llama 3.3 70B FP8, H100	Baseline	+1–4%	+15–30%	TensorRT-LLM’s compiled-kernel edge is largest here; SGLang’s RadixAttention has nothing to cache
Throughput, high prefix-overlap traffic (multi-turn/agentic), Llama 3.1 8B	Baseline (~12.5k tok/s)	+~29% (~16.2k tok/s)	Comparable to vLLM or better, no native prefix reuse advantage	SGLang’s gain is attributable almost entirely to RadixAttention cache hits
Time-to-first-token (TTFT), cached prefix	Moderate reduction (prefix caching added in recent versions)	Largest reduction on repeated prefixes	Sub-100ms reported at peak on H100 FP8, but static graph, no cross-request cache reuse	TTFT gains from prefix caching only materialize with real overlap in traffic
Memory efficiency (KV-cache utilization)	Near-100% via PagedAttention blocks	Near-100%, plus dedup from shared prefixes	High, but tuned per compiled batch-size profile	SGLang can serve more concurrent sessions on the same GPU when prefixes overlap heavily
Time-to-first-deploy for a new open-weight model	Hours (usually plug-and-play)	Hours (usually plug-and-play)	Days (compile, validate, tune per precision/batch profile)	This is the most consistently reported qualitative gap across sources

Methodology caveats: none of these figures come from a single controlled study run across all three engines on identical hardware, model, and traffic simultaneously — such studies are rare because each vendor benchmarks favorably for their own engine. Prefix-overlap percentage is the single biggest hidden variable: a benchmark that doesn’t disclose it is not comparable to one that does. GPU generation (Hopper vs Blackwell) changes the TensorRT-LLM advantage substantially since its quantization story is tuned per architecture. Always benchmark on your own traffic shape before committing.

Figure 3: A qualitative summary of where each engine tends to win. TensorRT-LLM leads on raw throughput and TTFT for compiled, single-model, high-QPS deployments. SGLang leads on both throughput and TTFT specifically when traffic has meaningful prefix overlap. vLLM is the most balanced generalist, trailing both specialists in their respective strong suits but rarely losing badly anywhere, and it wins decisively on deployment velocity for new or frequently-changing models.

Trade-offs, Gotchas, and What Goes Wrong

vLLM’s flexibility has a tail-latency cost. Because it stays general-purpose — Python-first scheduler, runtime quantization loading, broad model support — it doesn’t hit the same peak tokens/sec as a compiled TensorRT-LLM engine under sustained maximum load. Teams that assume “vLLM is fast enough for everything” sometimes discover a 15-30% throughput gap only after they’ve scaled traffic and the GPU bill makes it visible.

SGLang’s headline numbers are a prefix-caching illusion if your traffic doesn’t share prefixes. The 6.4x and ~29% figures floating around are ceiling numbers for workloads engineered around prefix reuse. If you’re serving unique, unrelated single-turn prompts — classification, one-shot summarization of distinct documents — RadixAttention has nothing to deduplicate and SGLang converges toward vLLM-level performance, sometimes with slightly more scheduler overhead from radix-tree maintenance that doesn’t pay for itself.

TensorRT-LLM’s compile step is a deployment liability, not just a startup delay. Every model update, precision change, or batch-size-profile adjustment triggers a rebuild. Teams running frequent fine-tune iterations or A/B testing multiple checkpoints find themselves maintaining a compile pipeline as a first-class piece of infrastructure — CI for CUDA graphs, effectively. Getting this wrong means either stale compiled engines silently serving an old checkpoint, or blocking releases on multi-hour builds.

KV-cache memory pressure is universal, but the failure mode differs. Under memory pressure, vLLM and SGLang can preempt and evict low-priority sequences, recomputing them later — a soft degradation. A compiled TensorRT-LLM engine sized for a specific max-batch profile will hard-reject or queue requests that exceed its planned envelope, since the memory layout was fixed at compile time. If your traffic is bursty and unpredictable, that rigidity needs explicit capacity planning, not just autoscaling.

Multi-GPU and disaggregated serving add engine-specific complexity. All three now support tensor and pipeline parallelism, and prefill/decode disaggregation is increasingly common at scale — see our deeper coverage of prefill-decode disaggregation architecture for the mechanics. TensorRT-LLM’s disaggregation setup is the most rigid to configure correctly because parallelism strategy is baked into the compiled graph; vLLM and SGLang can adjust more dynamically but at some cost to peak efficiency.

Mixture-of-experts models change the calculus again. If you’re serving MoE architectures with expert parallelism, engine choice interacts with routing strategy in ways none of the above table captures — our expert-parallel MoE inference serving architecture piece covers that separately, and it’s worth reading before locking in an engine for an MoE deployment specifically.

Structured-output correctness bugs surface differently across engines. JSON-schema-constrained decoding is now a common requirement for agentic and tool-calling workloads, and all three engines support some version of it, but the implementation details differ enough to matter. SGLang’s token-mask-based constrained decoding is integrated tightly with its scheduler and has the most mileage in production agentic deployments. vLLM’s guided-decoding support (built on outlines/lm-format-enforcer-style backends) is functionally solid but has historically lagged SGLang’s on latency overhead for complex schemas — a gap that’s narrowed but not fully closed as of 2026. TensorRT-LLM’s structured-output story is the least mature of the three; teams needing strict JSON-schema enforcement on a TensorRT-LLM deployment often end up validating and retrying at the application layer rather than relying on engine-level guarantees, which adds latency variance that a benchmark focused purely on token throughput won’t surface.

Cold-start and autoscaling behavior diverges sharply. vLLM and SGLang both load HuggingFace-format checkpoints directly, so scaling out a new replica is bounded mostly by model download and GPU memory allocation time — typically tens of seconds to a few minutes depending on model size and storage bandwidth. A TensorRT-LLM replica needs either a pre-built compiled engine artifact staged and ready to load (fast, but requires you to have built and stored it ahead of time for every precision/GPU combination you might scale onto) or an on-the-fly compile (slow, often minutes to tens of minutes, entirely impractical for reactive autoscaling). Production TensorRT-LLM deployments almost always pre-bake compiled engines into container images or shared storage specifically to avoid this trap; teams that skip that step discover it the first time a traffic spike triggers autoscaling and new replicas take far longer to become ready than the spike lasted.

Practical Recommendations

Match the engine to the shape of your traffic and your operational tolerance for compile pipelines, not to whichever benchmark you saw most recently. If you’re not sure what your prefix-overlap ratio actually is, instrument it before you decide — log the shared-prefix length as a percentage of total prompt tokens across a representative traffic sample for a week. That single number predicts more about SGLang’s real-world advantage than any published benchmark will.

Treat the engine choice as reversible rather than a one-way door, at least initially. Standardizing your serving layer behind an OpenAI-compatible API from day one — regardless of which engine sits behind it — means swapping vLLM for SGLang, or piloting TensorRT-LLM for a single high-traffic model, is a routing change rather than a client-side rewrite. This matters more than it sounds: teams that hard-couple application code to an engine’s native client library end up with a migration cost that quietly biases them toward staying with a suboptimal engine long after the traffic pattern that justified the original choice has changed. Budget a recurring quarterly review of traffic composition against engine choice, especially for teams growing fast enough that this quarter’s prefix-overlap ratio and QPS profile may look nothing like last quarter’s.

• Default to vLLM if you serve many different models, iterate on checkpoints frequently, need broad quantization/LoRA support, or don’t yet know your traffic shape well enough to commit to a specialist engine.
• Choose SGLang when you can confirm meaningful prefix overlap — multi-turn chat, agentic tool-calling with shared schemas, RAG with repeated context blocks — and when structured/constrained generation is a first-class requirement.
• Choose TensorRT-LLM when you have one (or a small, stable set of) production model(s), fixed precision and batch-size targets, dedicated NVIDIA hardware, and an engineering team that can own a compile/validate pipeline as ongoing infrastructure.
• Re-benchmark on your own traffic before committing capacity budget. Published numbers, including the ones in this article, are directional signals, not guarantees for your workload.
• Plan for hybrid deployments at scale. Many production fleets run vLLM or SGLang for the long tail of models and experimentation, and TensorRT-LLM for the one or two highest-QPS production paths where the throughput gain justifies the operational cost.
• Revisit the decision at every major GPU generation shift. TensorRT-LLM’s advantage grows on newer NVIDIA silicon (Blackwell FP4 versus Hopper FP8) faster than the other two engines’ relative position changes, since NVIDIA tunes the compiler for each architecture first.

Figure 4: A practical decision path. Start with whether the deployment is single-model and fixed (favor TensorRT-LLM if throughput is the binding constraint and you can own a compile pipeline), multi-tenant with structured or prefix-heavy traffic (favor SGLang), or broad/experimental model coverage with uncertain traffic shape (favor vLLM as the safe default). Reassess whenever traffic composition or hardware generation changes materially.

Frequently Asked Questions

Is TensorRT-LLM always faster than vLLM and SGLang?

Not universally — it’s faster on raw sustained throughput for a single, fixed, compiled model on NVIDIA hardware, with reported gains around 15-30% over vLLM in several 2026 H100 benchmarks. But that advantage assumes you’ve already paid the compile-and-validate cost and your traffic matches the batch-size profile the engine was built for. On workloads with heavy prefix reuse, SGLang can match or exceed TensorRT-LLM’s throughput despite not being compiled, because the win comes from skipping computation entirely rather than executing it faster.

Does SGLang’s RadixAttention advantage apply to every workload?

No. RadixAttention’s gains are proportional to how much of your traffic shares token prefixes. Multi-turn conversations, agentic loops with repeated tool schemas, and RAG pipelines reusing the same retrieved context see large gains — sometimes multiples, per SGLang’s own published benchmarks on engineered high-overlap traffic. Single-turn, unique-prompt workloads (batch classification, independent document summarization) see little to no benefit, and SGLang converges toward vLLM-level performance with a small scheduler overhead.

Can I switch quantization precision at runtime with these engines?

vLLM and SGLang both support loading pre-quantized checkpoints (GPTQ, AWQ, FP8, increasingly FP4) without a separate compilation step, so switching precision is closer to a config change plus a checkpoint swap. TensorRT-LLM requires rebuilding the compiled engine for a new precision target using NVIDIA’s Model Optimizer toolchain, which can take from minutes to over an hour depending on model size — this is the single biggest operational difference between the compiled and non-compiled approaches.

Which engine has the best support for new open-weight model releases?

vLLM typically has the fastest and broadest coverage, often supporting a new architecture within days of release due to its large contributor base and HuggingFace-native design. SGLang has closed much of that gap but still generally trails by days to weeks for brand-new architectures. TensorRT-LLM maintains a curated, NVIDIA-validated model list; support for a new architecture may lag by longer, or require a custom conversion effort outside the officially supported set.

Do I need separate engines for prefill and decode?

Not by default — all three engines can run prefill and decode on the same GPU pool with continuous batching. But at high scale, disaggregating prefill (compute-bound) from decode (memory-bandwidth-bound) onto separate GPU pools is an increasingly common optimization across all three engines, since it lets each stage be scaled and hardware-matched independently. This is a deployment-architecture decision layered on top of engine choice, not a replacement for it — see our dedicated piece on prefill-decode disaggregation for the trade-offs.

What happens if my traffic pattern changes after I’ve committed to TensorRT-LLM?

You’ll likely need to rebuild the compiled engine to match the new traffic profile — different batch-size ranges, sequence-length distributions, or precision needs can all require a recompile to stay optimal, and in some cases a compiled engine built for one profile will simply underperform (not fail) on a substantially different one. This is the core operational risk of committing to a compiled serving path: it optimizes hardest for the traffic shape you declared at build time, and drifts in production traffic erode that advantage silently until someone notices the throughput numbers have slipped.

Can I run vLLM, SGLang, and TensorRT-LLM side by side in the same production fleet?

Yes, and it’s an increasingly common pattern rather than an edge case. A typical setup routes the highest-QPS, most stable production model to TensorRT-LLM for the raw throughput and cost-per-token advantage, while everything else — experimental models, frequently-updated fine-tunes, agentic and multi-turn workloads — runs on vLLM or SGLang behind the same gateway or router layer. The main engineering cost is maintaining a consistent API surface and observability stack across engines that don’t share internals, which usually means standardizing on an OpenAI-compatible API layer (all three support this) and a shared Prometheus/Grafana metrics pipeline so on-call engineers aren’t context-switching between different dashboards depending on which engine served a given request.

Further Reading

• Expert-Parallel MoE Inference Serving Architecture (2026)
• Continuous Batching in LLM Inference Architecture (2026)
• Prefill-Decode Disaggregation in LLM Serving Architecture (2026)
• vLLM GitHub Repository and Documentation
• SGLang RadixAttention Documentation

By Riju — about