NVIDIA Dynamo and LMCache: Disaggregated LLM Serving and KV Cache Offload (2026)

A single GPU serving a chat model does two jobs that want opposite hardware. It chews through a long prompt in one compute-saturated burst, then spends hundreds of steps emitting one token at a time while barely touching the arithmetic units. Run both on the same replica and each phase degrades the other: a 30,000-token prompt arriving mid-batch stalls every user who is waiting on their next token.

Disaggregated LLM serving splits those phases onto separate GPU pools and moves the key-value (KV) cache between them. NVIDIA Dynamo and LMCache are the two projects most teams reach for in 2026, one as a full serving framework and the other as a KV cache layer that plugs into engines. This post explains what each does, how the pieces connect, the transfer math, and the cases where you should not bother.

What this covers: the prefill/decode resource profile, the research behind splitting phases, Dynamo’s frontend, router, planner, KVBM and NIXL, LMCache’s tiers and CacheBlend, worked bandwidth and hit-rate numbers (labelled illustrative), launch commands, failure modes, and a decision framework.

Context and Background

Every autoregressive transformer request has two phases. Prefill processes all prompt tokens in parallel, producing the KV tensors for every layer plus the first output token. Decode then generates one token per step, reading the entire accumulated KV cache each time. Prefill is dominated by matrix multiplications over many tokens, so it is compute-bound. Decode does a tiny amount of arithmetic per byte of weights and cache it reads, so it is memory-bandwidth-bound.

Users feel the two phases through two latency metrics. Time to first token (TTFT) is essentially prefill latency plus queueing. Time per output token (TPOT, also called inter-token latency or ITL) is the decode step time. Colocated engines mitigate interference with continuous batching and chunked prefill, which slices a long prompt into pieces that ride along with decode steps. That helps, but the prefill chunks still steal cycles from decode, and both phases are forced to share one parallelism and memory configuration.

Two research papers made the case for physical separation. DistServe (arXiv 2401.09670) argues that colocation causes strong prefill-decode interference and couples resource allocation for both phases; it assigns each phase its own GPUs, tunes parallelism per phase, and places them according to cluster bandwidth. Its abstract reports serving up to 7.4x more requests or meeting up to 12.6x tighter SLOs than the state-of-the-art systems the authors compared against, with over 90 percent of requests inside latency constraints. Splitwise (arXiv 2311.18677) makes a hardware argument: token generation underuses compute and can run on cheaper, lower-power machines. It reports 1.4x higher throughput at 20 percent lower cost, or 2.35x more throughput under the same cost and power budget. Those are the authors’ results on their workloads and testbeds, not a promise for yours.

The cost of separation is that the KV cache computed on a prefill GPU must reach a decode GPU. That transfer is why a purpose-built data mover, NVIDIA’s NIXL, sits at the heart of Dynamo. The same KV cache is also an asset worth keeping: if a second request shares a prefix with an earlier one, recomputing it is waste. That observation drives both KV-aware routing and tiered KV cache offload, which is LMCache’s core job.

If you run Kubernetes-native inference, our earlier piece on llm-d and prefill/decode disaggregation on Kubernetes covers the scheduler-centric approach. This post takes the other angle: a serving framework with its own router, planner and cache manager (Dynamo), and a cache layer that works under or beside engines (LMCache). For the engine itself, see the vLLM 0.30 Model Runner V2 migration. NVIDIA describes the framework as open source and the repository is Apache 2.0 licensed; supported engines are vLLM, SGLang and TensorRT-LLM, and the docs I checked list v1.5.1 as the stable release.

Reference Architecture: How Dynamo Composes a Disaggregated Stack

Dynamo is a set of cooperating components rather than one engine: a frontend that speaks an OpenAI-compatible HTTP API, a KV-aware router, prefill and decode workers that wrap an existing engine (vLLM, SGLang or TensorRT-LLM), a planner that resizes the pools, a KV Block Manager (KVBM) for tiered offload, and NIXL for moving bytes. The design bet is that the engine stays a black box for compute while routing, scaling, and cache placement become cluster-level concerns.

Figure 1: Dynamo’s control and data paths. The router consumes KV events from workers, the planner resizes the two pools, and KV blocks travel prefill-to-decode over NIXL.

The figure separates three flows. Requests flow top to bottom through the frontend and router into a prefill worker, then a decode worker. KV blocks flow sideways between workers via NIXL and, optionally, down into KVBM tiers. Telemetry flows back up: workers publish KV cache events to the router, and the planner watches load to decide how many prefill and decode replicas to run.

The KV-aware router: scoring workers by cache overlap

A round-robin balancer treats all replicas as equivalent, which is wrong once caches differ. Dynamo’s router keeps a prefix tree (the docs call the component a KvIndexer) built from KV events that workers emit, so it knows which blocks each worker holds. For every candidate worker it computes a logit and picks the lowest. Per the router docs, logit = kv_overlap_score_weight x potential_prefill_blocks + potential_active_blocks, where potential prefill blocks are the uncached input tokens divided by block size and active blocks are the decode blocks the worker would hold including the new request.

The intuition is a two-term cost: how much prefill work would this request cost here, and how loaded is this worker already. The default weight is 1.0. Raising it favors cache hits and lower TTFT, which suits prefill-heavy workloads; lowering it spreads decode load and protects inter-token latency, which suits long generations. --router-temperature at 0.0 chooses deterministically; above 0 it samples through a softmax to avoid piling every request onto one hot worker. Routing is enabled with --router-mode kv on the frontend, and --kv-cache-block-size should match the backend. If you cannot afford the event stream, --no-kv-events switches to an approximate indexer that assumes recently routed requests have their blocks cached; cheaper, less accurate.

Why does the weight matter so much? Imagine two workers. Worker A holds 90 percent of a 4,000-token prompt (block size 16 gives 250 blocks, 25 uncached) but already has 800 active blocks. Worker B holds nothing, has 100 active blocks. With weight 1.0, A scores 25 + 800 + new-request blocks, B scores 250 + 100 + the same. The new-request term is common to both here, so A wins only if the load gap is smaller than the uncached prefill gap. This is a toy calculation to show the trade, not a measured result; the point is that one scalar encodes your TTFT-versus-ITL preference.

The planner: scaling prefill and decode separately

Autoscaling on CPU or request count misses what matters for LLMs, because latency depends on request content and on which phase is the bottleneck. Dynamo’s Planner scales replicas from latency and load signals. According to its docs, the sla optimization target uses TTFT and ITL goals (defaults 500 ms and 50 ms) with regression-based performance models, the throughput target scales on queue depth and KV cache utilization, and latency scales earlier to keep queues short.

It runs two loops. Throughput-based scaling uses pre-deployment engine performance data and traffic prediction (predictors include arima, prophet, kalman and constant) on a default 180 second interval; load-based scaling uses forward-pass metrics and online linear regression on a 5 second interval to react to bursts. When both are enabled, the slow loop sets a capacity floor and the fast loop handles spikes above it. An advisory mode computes recommendations without acting, which is the sensible way to start. The practical lesson: the prefill-to-decode ratio is not a constant. A summarization workload wants many prefill GPUs per decode GPU; an agent that emits long reasoning traces wants the reverse.

NIXL and KVBM: the data plane

NIXL (NVIDIA Inference Xfer Library) is described in its repository as a library for fast point-to-point transfers in inference frameworks, with a plugin architecture over memory and storage types. Named backends include UCX, GDS (GPUDirect Storage), LIBFABRIC, POSIX, object-store and Mooncake plugins. It is Linux-only and Apache 2.0 licensed. For the prefill-to-decode handoff, the practical meaning is that the decode worker pulls registered GPU memory regions from the prefill worker directly, over RDMA when the fabric supports it, without staging through host memory.

KVBM, per the Dynamo README, offloads KV cache across GPU, CPU, SSD and remote storage. I could not retrieve the KVBM configuration page during research, so I will not quote its environment variables here; check the KVBM configuration reference in the Dynamo docs for current names and defaults before relying on any setting you find in older blog posts.

Deeper Analysis: The Handoff, the Math, and the Launch Commands

The handoff is where disaggregation either pays off or quietly loses. Sequence matters: the prefill worker computes KV, registers the blocks with NIXL, the router tells the decode worker where they are, and the decode worker pulls them before its first step. The first token is typically produced by prefill, so TTFT includes prompt compute plus (depending on the engine’s flow) at least part of the transfer before decoding continues.

Figure 2: The prefill-to-decode handoff. The decode worker pulls registered KV blocks, so a slow fabric shows up directly as added latency before token two.

The diagram shows why the pull model is convenient: the decode side decides when it has memory for the blocks, which gives natural backpressure. It also shows the failure surface. If the pull stalls, the request waits with prefill compute already spent.

Worked transfer math (illustrative)

KV bytes per token equal 2 (K and V) x layers x KV heads x head dimension x bytes per element. Take a hypothetical dense model with 80 layers, 8 KV heads (grouped-query attention), head dimension 128, stored in 16-bit (2 bytes). That gives 2 x 80 x 8 x 128 x 2 = 327,680 bytes, about 320 KiB per token. These are illustrative architecture numbers, not a specific product’s published figures; substitute your model’s config. Models using multi-head latent attention shrink this sharply, as covered in our comparison of MLA versus GQA and MQA KV cache compression.

A 8,000-token prompt then carries about 2.6 GB of KV (8,000 x 327,680 = 2.62 GB). Over a link delivering an effective 40 GB/s (an illustrative figure well below a datacenter RDMA NIC’s headline rate once protocol and contention are included), that is roughly 65 ms. Over a 100 Gb/s Ethernet path delivering about 10 GB/s effective, it is about 260 ms. Over a single congested 25 Gb/s path at roughly 2.5 GB/s, it is over a second. Meanwhile, prefilling 8,000 tokens on a modern accelerator for a large model might take on the order of hundreds of milliseconds to a couple of seconds depending on model and parallelism. If transfer time is a meaningful fraction of prefill time and sits on the critical path, you have eaten the benefit.

Three mitigations exist. First, layer-wise or block-wise pipelining overlaps transfer with the remaining prefill compute. Second, compression or quantization of KV (FP8 KV cache halves the bytes versus 16-bit, at some accuracy risk you must measure). Third, placement: DistServe explicitly places phases by cluster bandwidth, keeping the pair on the same node or NVLink domain when the fabric is thin. A useful rule of thumb to test, not a law: the transfer should be well under the TTFT slack you have, and ideally under 10 percent of prefill time.

Launching Dynamo locally

The repository quickstart for the vLLM backend, run inside the project container, starts a frontend and a worker:

python3 -m dynamo.frontend --http-port 8000 --discovery-backend file
python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B --discovery-backend file

The --discovery-backend file option lets you try it without etcd or NATS. The README notes that for local vLLM use you may also need --kv-events-config '{"enable_kv_cache_events": false}'. SGLang uses the same frontend with python3 -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --discovery-backend file. These are aggregated deployments; they show the plumbing, not disaggregation.

For disaggregation, the vLLM configuration reference documents --disaggregation-mode with values agg (default), prefill, decode, and encode for multimodal. It also states that a worker started with --disaggregation-mode prefill must be given --kv-transfer-config explicitly, and that both halves of a NIXL pair must use a matching connector. The old --connector flag now raises a ValueError. A shape of the launch, which I have not run and which you should adapt from the official example scripts under examples/backends/vllm/launch, is:

# frontend with KV routing (flag names from router docs)
python3 -m dynamo.frontend --router-mode kv --kv-cache-block-size 16

# prefill worker (connector JSON must match the decode side)
python3 -m dynamo.vllm --model <model> --disaggregation-mode prefill \
  --kv-transfer-config '<NIXL connector JSON from the Dynamo example>'

# decode worker
python3 -m dynamo.vllm --model <model> --disaggregation-mode decode \
  --kv-transfer-config '<same connector JSON>'

I deliberately leave the connector JSON as a placeholder: the docs I could fetch did not print it, and an incorrect literal would be worse than none. In production the docs point to Kubernetes deployments with a DynamoGraphDeployment resource, and mention a DynamoGraphDeploymentRequest that profiles the model automatically for SLA-based planning.

Why the router and disaggregation interact

KV-aware routing and disaggregation compound each other. In a disaggregated setup, only prefill workers need prefix cache locality, because they are the ones paying for recompute. Decode workers hold the live cache for in-flight requests. A sensible routing policy therefore scores prefill workers by overlap and decode workers by free memory and active blocks. That is exactly the structure of the logit above, applied per pool, and it is why a router that understands the topology beats a generic load balancer in front of two pools.

LMCache: Treating the KV Cache as a Storage Hierarchy

LMCache describes itself as a KV cache management layer for scalable LLM inference. Where Dynamo is a whole serving stack, LMCache is a component: it intercepts KV blocks as an engine produces them, stores them in tiers beyond GPU memory, and retrieves them when a later request shares content. The README lists CPU RAM, local disk, Redis or Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL and GPUDirect Storage as storage or transport backends, and the project is Apache 2.0 licensed.

Figure 3: A KV cache offload hierarchy. Blocks evicted from GPU memory fall to CPU RAM, then SSD, then a remote store, and are pulled back on a hit instead of being recomputed.

The economics follow from the hierarchy. GPU HBM is the scarcest and fastest tier, so a busy replica evicts old prefixes quickly. Host RAM is orders of magnitude larger per dollar, and a PCIe or NVLink-C2C transfer of cached blocks is usually cheaper than recomputing them through the full model. Disk and remote tiers trade latency for capacity and, in the remote case, sharing across replicas.

Chunking, multi-process mode, and the vLLM connector

According to the LMCache quickstart, the default chunk size is 256 tokens (the doc’s demo uses 16). Chunking defines the granularity of storage and lookup: smaller chunks raise hit rates for near-matching prefixes but add metadata and transfer overhead. Two deployment modes exist. The recommended MP (multi-process) mode runs a separate server:

lmcache server --host localhost --port 5555 \
    --l1-size-gb 20 --eviction-policy LRU --chunk-size 16

vllm serve Qwen/Qwen3-8B --port 8000 --kv-transfer-config \
  '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.host":"localhost","lmcache.mp.port":5555}}'

In-process mode embeds LMCache in the engine and is configured through LMCACHE_* environment variables (for example LMCACHE_CHUNK_SIZE) or a YAML file named by LMCACHE_CONFIG_FILE, with "kv_connector":"LMCacheConnectorV1". A simpler path in recent vLLM is --kv-offloading-backend lmcache --kv-offloading-size <GB>, which per the docs requires --disable-hybrid-kv-cache-manager. On vLLM 0.20.0 or newer, the docs recommend the connector module shipped by LMCache via kv_connector_module_path. The MP server also supports remote hosts (tcp:// addresses), a gRPC transport, and a CPU-only transfer mode. To test, send two completions whose prompts share a prefix and look for retrieve and store log lines in the server process.

One caveat worth reading twice: the version-compatibility page exists for a reason. Connector APIs inside vLLM have changed across releases, which is why our vLLM 0.30 migration guide matters if you pin LMCache. Pin both versions together and test after every engine upgrade.

CacheBlend and CacheGen: beyond prefix reuse

Plain prefix caching only helps when the new prompt starts with identical tokens. Retrieval-augmented generation breaks that: retrieved documents appear in different orders, so the second document’s KV was computed with a different preceding context. CacheBlend (EuroSys 2025, cited in the LMCache README) enables reuse of cached KV at any prompt position and selectively recomputes a subset of tokens to recover the cross-document attention that the stale cache lacks. The project positions this as non-prefix KV reuse. The trade is quality risk against speed: you are approximating attention across chunks, so validate answer quality on your own RAG evaluation set before enabling it broadly.

CacheGen (SIGCOMM 2024), also cited by the project, is about compressing and streaming KV cache for fast loading. The README lists it among related papers without detailing how the current LMCache release uses it, so I will not claim a specific compression ratio or integration state.

LMCache inside a disaggregated pair

LMCache can also act as the transport for prefill/decode separation. Its disaggregated prefill documentation shows a producer vLLM and a consumer vLLM, each with its own LMCache server (they must not share one), a MultiConnector combining a NixlConnector (kv_producer on the prefill side, kv_consumer on the decode side) with an LMCacheMPConnector set to kv_both, and a router in P/D mode: vllm-router --policy round_robin --vllm-pd-disaggregation --prefill http://<PREFILL_IP>:8001 --decode http://<DECODE_IP>:8002. NIXL handles the handoff with UCX/RDMA, with a handshake side channel configured through VLLM_NIXL_SIDE_CHANNEL_HOST and VLLM_NIXL_SIDE_CHANNEL_PORT (default 5600, unique per instance on a host).

The docs make several details explicit that bite in practice. kv_load_failure_policy set to fail makes failed transfers raise errors rather than silently recompute. NIXL over localhost uses loopback or TCP rather than RDMA, so a single-node test validates function, not latency. NIXL must be installed via the lmcache[nixl] extra, and offload under MultiConnector required a specific vLLM pull request (#46865 as named in the docs) or offload silently never triggers. That last item is the kind of silent failure you only catch by watching hit-rate metrics, a theme we return to below.

Deciding When It Pays: Hit-Rate Economics and a Decision Matrix

Disaggregation and KV offload solve different problems, and conflating them leads to expensive clusters that underperform. Disaggregation removes phase interference and lets you size pools independently. Offload and KV-aware routing remove redundant prefill. You can have either without the other, and the second is often the cheaper win.

Figure 4: A decision flow. Short prompts or small clusters stay aggregated; repeated prefixes justify offload and KV-aware routing; tight dual SLOs on a fast fabric justify disaggregation.

Prefix-cache hit-rate economics (illustrative)

Suppose a support assistant prepends a 3,000-token system prompt and tool schema to every request, followed by about 500 tokens of user-specific content. Without prefix reuse, every request prefills 3,500 tokens. With a perfect shared-prefix cache, it prefills 500, so prefill compute falls by about 86 percent. At a 70 percent hit rate on the shared part, the expected prefilled tokens are 3,500 minus 0.7 x 3,000 = 1,400, a 60 percent cut. These are arithmetic on assumed numbers, not benchmarks.

Now the offload argument. If each replica’s GPU cache holds only a few thousand distinct prefixes and you serve millions of sessions, multi-turn conversations return after minutes or hours, long after HBM eviction. At roughly 320 KiB per token from our earlier hypothetical, a 10,000-token conversation occupies about 3.2 GB. A 1 TB host-RAM tier holds on the order of 300 such conversations per node; an SSD tier holds far more. If a returning user’s 10,000-token history can be reloaded from RAM in tens of milliseconds instead of recomputed in seconds, TTFT for those turns collapses. The break-even is simple: reload time must be well below recompute time, and the hit probability must be high enough to justify the memory you dedicate.

KV-aware routing is what makes per-replica caches useful. Without it, a returning session lands on a random replica and misses even though another replica holds its prefix. With a shared remote tier, any replica can hit, at the cost of network transfer. Dynamo’s router attacks the problem by steering, LMCache’s remote backends by sharing; they are complementary, and mixing them is a design choice rather than a requirement.

When NOT to disaggregate

Skip disaggregation when prompts are short. If typical prompts are a few hundred tokens, prefill is cheap, interference is small, and the transfer overhead plus extra moving parts exceed the gain. Skip it on small clusters: with, say, four GPUs, splitting into a 2:2 or 1:3 prefill:decode ratio leaves no slack, and one failed worker is a quarter of a pool. Skip it without a fast fabric: if you cannot get RDMA-class bandwidth between pools, the transfer math above works against you. Skip it when your latency targets are loose enough that chunked prefill meets them. The papers’ headline gains were reported under SLO constraints on specific testbeds; if you have no tight TTFT and TPOT objectives, you are optimizing a constraint you do not have.

Comparison with llm-d and engine-native options

llm-d approaches the same problem from Kubernetes: it uses vLLM as the engine, an inference scheduler that is cache- and load-aware, and prefill/decode separation expressed as Kubernetes workloads. Dynamo ships its own frontend, router, planner and KV manager and can run on Kubernetes, Slurm or locally, with multiple engines behind it. If your platform team standardizes on Gateway API and Kubernetes primitives, llm-d fits naturally; if you want a single framework that spans engines and includes planner-driven autoscaling and a KV block manager, Dynamo is closer. See our llm-d walkthrough for that side. SGLang has its own prefill/decode and cache features; the practical effects of version changes are covered in the SGLang 0.5.18 versus 0.5.15 migration notes.

Need	Aggregated + chunked prefill	KV-aware routing + LMCache offload	Full prefill/decode disaggregation (Dynamo or llm-d)
Short prompts, small cluster	Best fit	Unnecessary	Overkill
Heavy shared prefixes, multi-turn	Wasteful recompute	Best fit	Helps, but offload is the main win
Tight TTFT and ITL together, long prompts	Struggles at load	Partial	Best fit with fast fabric
Operational simplicity	Highest	Medium	Lowest
Independent pool scaling	No	No	Yes
Needs RDMA-class interconnect	No	Optional	Strongly recommended

The matrix is my synthesis of the mechanisms above, not a benchmark result. Treat it as a way to ask the right question: which cost are you actually paying, interference or recompute?

Trade-offs, Gotchas, and What Goes Wrong

Silent no-ops. The LMCache disaggregated docs warn that offload under MultiConnector silently never triggers without a particular vLLM change. Hit-rate metrics are the only defense: alert when the retrieve count is zero on a workload you know has shared prefixes. A cache that never hits looks exactly like a healthy system with slow requests.

Version coupling. Connector interfaces live inside the engine. Dynamo, vLLM, LMCache and NIXL each version independently, and the LMCache docs gate recommendations on vLLM release numbers (for example, 0.20.0 or newer for the shipped connector module). Upgrading one component can break the others, so pin a tested set and roll it as a unit.

Pool imbalance and cold starts. A prefill-heavy burst saturates the prefill pool while decode GPUs idle, and the reverse happens during long generations. The planner’s 5 second load loop helps, but scaling a pool means loading model weights onto new GPUs, which takes far longer than a burst lasts. Keep headroom, and use the planner’s advisory mode to see what it would do before you let it act. Dynamo’s README also mentions streaming weights GPU-to-GPU via NIXL to speed this up; measure it on your model.

Transfer failures mid-request. If a decode worker cannot pull blocks, the request has already burned prefill compute. LMCache’s kv_load_failure_policy: fail surfaces the error, which is good for observability but means you need a retry or fallback path that re-routes to an aggregated worker. Decide the policy deliberately instead of inheriting a default.

Fabric contention. KV transfers share NICs with tensor-parallel all-reduce, expert-parallel traffic, and storage. Bandwidth that looked ample in a single-request test collapses under concurrent handoffs. Test with realistic concurrency and prompt-length distributions, because transfer volume scales with prompt length while decode load does not.

Cache correctness and privacy. Reusing KV across requests is safe only when the cached prefix is byte-identical in tokens and the same model, adapter, and tokenizer produced it. Mixing LoRA adapters or tokenizer changes without keying them into the cache hash yields wrong outputs. In multi-tenant systems, a shared prefix cache is also a timing side channel, because a fast response reveals that someone else already sent that prefix. Partition caches per tenant where that matters. CacheBlend adds approximation on top; evaluate quality, not just latency.

Operational surface area. Discovery needs etcd and NATS in the non-file-based setup the vLLM backend page describes. That is two more stateful systems to run, monitor and upgrade. For a team without platform engineers, this alone can outweigh the performance gain.

Honesty about evidence. I did not run these stacks for this article, and I am not citing vendor throughput numbers because I could not verify methodology for them. The DistServe and Splitwise figures come from peer-reviewed-style academic evaluations on their own testbeds. Your gains depend on prompt length distribution, hit rate, fabric and SLOs.

Practical Recommendations

Start by measuring, not by deploying. Capture your real distribution of prompt and output lengths, the share of tokens that repeat across requests, and your TTFT and ITL percentiles under load. If prefix repetition is high, adopt prefix caching and then KV offload first; it is the smaller change and often the larger win. If interference is the problem, with p99 ITL spiking whenever long prompts arrive and chunked prefill tuning exhausted, then evaluate disaggregation.

Prototype in this order. Run the single-node aggregated Dynamo or LMCache quickstart to learn the plumbing. Enable KV events and routing, and watch overlap scores and hit rates. Add a CPU tier sized from the per-token math for your model. Only then split pools, on hardware with RDMA, with the planner in advisory mode.

Checklist:

• Compute KV bytes per token from your model config and size tiers from it, not from guesses.
• Confirm the fabric delivers the effective bandwidth your transfer math assumes, under concurrent load.
• Pin engine, LMCache, NIXL and Dynamo versions as a tested bundle.
• Alert on cache hit rate, transfer failures, and per-pool queue depth.
• Define a fallback to aggregated serving when transfer fails.
• Key caches by model, adapter and tenant where isolation matters.
• Evaluate output quality when using non-prefix reuse such as CacheBlend.
• Revisit the prefill:decode ratio whenever traffic mix changes.

Frequently Asked Questions

What is disaggregated LLM serving?

It is a serving architecture that runs the prompt-processing (prefill) phase and the token-generation (decode) phase on separate GPU pools, transferring the KV cache between them. Separation removes interference between the two phases and lets you size and tune each pool independently, at the price of a KV transfer on every request and more components to operate.

What is NVIDIA Dynamo used for?

Dynamo is an open-source inference framework from NVIDIA that sits above engines such as vLLM, SGLang and TensorRT-LLM. It provides a frontend, a KV-aware router, a planner that autoscales prefill and decode pools, a KV block manager for tiered offload, and NIXL for fast transfers. It targets multi-GPU and multi-node deployments on Kubernetes, Slurm or a single machine.

What does LMCache do?

LMCache is a KV cache management layer that stores computed KV blocks in tiers such as CPU memory, local disk and remote backends, then reuses them for later requests. It integrates with vLLM through connectors, supports prefill/decode transfer via NIXL, and offers CacheBlend for reusing cached KV at non-prefix positions in workloads like retrieval-augmented generation.

Is Dynamo the same as llm-d?

No. Both support prefill/decode disaggregation and cache-aware routing, but llm-d is organized around Kubernetes and vLLM, while Dynamo is a framework with its own router, planner and KV manager that supports multiple engines and several deployment targets. They overlap in goals, not in packaging. Many teams will pick based on platform standards rather than features.

When should I not use prefill/decode disaggregation?

Avoid it when prompts are short, the cluster is small, you lack a fast interconnect, or your latency targets are loose enough for chunked prefill to meet. In those cases the transfer cost and operational complexity outweigh the benefit, and prefix caching with KV offload usually delivers more value for less effort.

How much KV cache does a request need?

Bytes per token equal 2 x layers x KV heads x head dimension x bytes per element. For a hypothetical 80-layer model with 8 KV heads, head dimension 128 and 16-bit values, that is 327,680 bytes per token, about 2.6 GB for an 8,000-token prompt. Check your own model’s configuration, since attention variants such as MLA change the figure sharply.

Further Reading

• llm-d on Kubernetes: prefill/decode disaggregation
• vLLM 0.30 Model Runner V2 migration
• MLA vs GQA vs MQA: KV cache compression
• SGLang 0.5.18 vs 0.5.15 migration
• NVIDIA Dynamo documentation and the Dynamo repository
• LMCache documentation, DistServe paper and Splitwise paper

By Riju — about