LLM Prefix Caching: RadixAttention vs vLLM Automatic Prefix Caching

Most production LLM traffic is not unique text. A chat assistant resends the same system prompt and the entire conversation history on every turn; a RAG service stuffs the same policy document in front of thousands of different questions; an agent loop re-reads a growing scratchpad and an unchanged tool schema at every step. If your engine recomputes the key-value (KV) tensors for all of that on every request, you are paying prefill compute for work you already did a second ago. LLM prefix caching removes that waste by keeping the KV cache of previously seen token prefixes and reusing it whenever a new request starts with the same tokens.

It matters now because prefill is the part of inference that scales with input length, and inputs have grown from hundreds of tokens to tens of thousands. Reuse has become the cheapest lever on time-to-first-token (TTFT) and GPU cost. But the two dominant open-source implementations, SGLang’s RadixAttention and vLLM’s automatic prefix caching, make different structural choices, and those choices decide your hit rate, your memory pressure, and even your security posture.

You will leave with a precise mental model of both designs, the arithmetic for estimating what caching will save you, and a checklist of the routing, offload, and isolation decisions that determine whether the savings actually show up.

What this covers: how each system finds and stores shared prefixes, how eviction behaves, how cache-aware routing and offload tiers (LMCache, HiCache) extend reuse beyond one GPU, how it all interacts with chunked prefill, and the pitfalls: salting, timing side channels, and tokenization boundaries.

Context and Background

Autoregressive transformers produce, for every token in the context, a key vector and a value vector in every layer. Attention for the next token reads all of them. The KV cache stores these tensors so each decode step only computes the new token’s projections. Prefill is the phase that builds the KV cache for the whole prompt in one parallel pass, and it is compute-bound; decode is memory-bandwidth-bound. Prefix caching targets prefill only.

The size of what is being cached is easy to derive. Per token, the KV cache holds two tensors (K and V) per layer, each with kv_heads x head_dim elements. For a Llama-3-8B-style configuration (32 layers, 8 KV heads via grouped-query attention, head dimension 128) in 16-bit precision, that is 2 x 32 x 8 x 128 x 2 bytes = 131,072 bytes, or 128 KiB per token. For a 70B-class configuration (80 layers, same 8 KV heads and head dimension) it is 320 KiB per token. These are derived from the published model configs, not measured values. A 2,000-token shared prefix therefore occupies roughly 250 MiB on the 8B shape and 625 MiB on the 70B shape, which is why capacity planning matters as much as hit detection.

Before paged memory management, serving engines reserved contiguous, worst-case KV buffers per request, and sharing between requests was nearly impossible. PagedAttention, introduced with vLLM, split the KV cache into fixed-size blocks addressed through a block table, much like virtual memory pages. Once KV storage is paged, sharing becomes a bookkeeping problem: two requests can point their block tables at the same physical blocks. For the kernel-level details of that paging, see our deep dive on FlashAttention-3 and PagedAttention I/O-aware kernels.

Two designs grew out of that observation. SGLang’s RadixAttention, described in the paper “SGLang: Efficient Execution of Structured Language Model Programs” (arXiv:2312.07104, Zheng et al.), manages the KV cache as a radix tree over token sequences and reports up to 6.4x higher throughput than prior systems across workloads such as agent control, JSON decoding, and multi-turn chat. vLLM’s automatic prefix caching (APC) keeps the paged block pool and adds a hash table keyed by chained block hashes. Both are exact-match prefix caches: they reuse KV only when the token sequence from position zero matches. Neither approximates attention, so outputs are the same as without caching (modulo floating-point non-determinism from different kernel batch shapes).

Commercial APIs expose the same idea as “prompt caching” with pricing discounts; our prompt caching architecture and economics guide covers the billing side. This post stays on the engine side: how the open-source engines implement it, and how to operate it well.

Two Data Structures for One Idea

Both systems answer the same question: given a new token sequence, what is the longest prefix whose KV we already hold, and where is it in GPU memory? The difference is whether the answer comes from walking a tree or from hashing blocks.

Figure 1: Both engines find the longest cached prefix, then prefill only the uncached suffix. vLLM does it by looking up chained block hashes; SGLang does it by walking a radix tree.

The diagram shows the shared control flow. The request is tokenized, the engine finds the longest reusable prefix by one of two mechanisms, the matching KV blocks are pinned and referenced, and only the remaining suffix goes through prefill. Decode proceeds identically afterward, which is why prefix caching never changes per-token decode cost.

vLLM automatic prefix caching: chained block hashes

In vLLM’s design, every KV block holds a fixed number of tokens (the block size). Only full blocks are cached. Each block’s hash is computed over three components: the hash of the parent (previous) block, the exact tokens in this block, and “extra hashes” that make a block unique even if the tokens match, such as LoRA adapter identifiers, multimodal input hashes, and a per-request cache salt. Because each hash includes its parent’s hash, a block hash identifies the entire prefix up to and including that block, not merely the block’s own tokens. That is what makes a flat hash table sufficient: a lookup of block k is a lookup of the whole prefix of length k x block_size.

The hash function is configurable. According to vLLM’s design documentation, the --prefix-caching-hash-algo option accepts sha256 (the default as of v0.11, using pickle serialization, so hashes may not reproduce across Python or vLLM versions), sha256_cbor (CBOR serialization, reproducible and cross-language), xxhash (a faster, non-cryptographic 128-bit hash with higher collision risk), and xxhash_cbor. The security-relevant point: a non-cryptographic hash is fine for a single-tenant deployment you control, but in a multi-tenant setting a cryptographic hash is the conservative choice, because a deliberate collision would let one request read another’s KV blocks.

# vLLM: prefix caching with a reproducible cryptographic hash
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-prefix-caching \
  --prefix-caching-hash-algo sha256_cbor \
  --block-size 16

The match algorithm is a loop. Starting at block 0 it computes the hash of each full block of the incoming prompt in sequence and looks it up; it stops at the first miss. Everything before the miss is a hit. The granularity cost is real: a prompt that shares 1,000 tokens with a cached one, at block size 16, reuses floor(1000/16) x 16 = 992 tokens, and the partial block is recomputed. At small block sizes that loss is negligible; at large block sizes (some attention backends or hybrid models force larger blocks) it can be material for short shared prefixes.

SGLang RadixAttention: a radix tree over tokens

SGLang represents all cached sequences as a radix tree (a compressed trie). Each edge is labeled with a run of tokens, and each node owns the KV tensors for the tokens on its incoming edge. When a request arrives, the scheduler walks the tree from the root following the request’s tokens to find the longest match. If the request diverges in the middle of an edge, the edge is split at the divergence point into two nodes so the shared part can be referenced separately from the new part. After the request is processed, its new suffix is inserted as a fresh child.

Figure 2: A radix tree naturally captures branching workloads: one system prompt fans out to many users, and one tool schema fans out to many agent steps.

Two properties follow from the tree structure. First, matching is at token granularity in the tree, not block granularity, so there is no partial-block loss in the abstract data structure (the underlying page size, where greater than one, reintroduces page-granular matching, and SGLang’s documentation notes matching uses page granularity when page_size > 1). Second, the tree gives you structural information a flat hash table lacks: you know which nodes are leaves, which have many children, and how deep each branch goes. That information drives eviction and, as we will see, routing.

What this means in practice: same hit rate, different ergonomics

For a pure “does the prefix match” question, the two are functionally equivalent at equal granularity. A chained hash and a trie path both uniquely identify a prefix. The practical differences are elsewhere:

Granularity. Hash lookup is inherently per-block; the tree can split at any token boundary (subject to page size).
Lookup cost. Hashing costs one hash per block per request (and the hashes can be computed incrementally as blocks fill, including during chunked prefill). A tree walk costs a traversal proportional to path depth, with edge comparisons over token runs.
Structural awareness. The tree knows about branching; the hash table only knows about individual blocks and their reference counts.
Extensibility. Extra keys (LoRA, multimodal, salt) slot naturally into the hash. In a tree they must be encoded into the key, or the tree must be partitioned per key.
Distribution. A hash is a compact, shareable identifier, which makes it a natural key for external stores and for cache events published to a router. A tree is a natural structure for a router to mirror approximately.

I would not pick an engine on the basis of this alone. The data structure matters less than the policies layered on top: eviction, routing, and tiering.

Eviction, Memory Pressure, and What the Cache Actually Costs

A prefix cache is not extra memory; it is the free memory you did not need yet. In both engines the KV cache pool is a fixed budget carved out of GPU memory after model weights and activations. Cached-but-idle blocks live in that same pool, and they are reclaimed the moment an active request needs space. This is the single most important operational fact: prefix caching never causes out-of-memory failures by itself, but it does mean that under load your cache shrinks toward zero.

LRU in vLLM: the free queue

vLLM tracks a reference count on every block. A block with a nonzero count is in use by a running request and cannot be evicted. When the count reaches zero, the block is appended to a free queue but keeps its hash and contents, so it can still produce a cache hit until its memory is reallocated. The free queue is a doubly linked list threaded through the block objects themselves, giving O(1) removal from the middle (needed when a cache hit “touches” a block that is sitting in the queue) and O(1) append.

Eviction order is LRU with a deliberate tie-break. The design document states that when a request finishes, its blocks are pushed to the tail in reverse order, so the last block of the sequence ends up closest to the head and is evicted first. The reasoning is that the final block covers the most tokens’ worth of unique prefix and is the least likely to be shared by a future request. Eviction pops the head block, removes it from the hash table, and hands its memory to the new allocation. Cache-hit blocks are touched (pulled out of the free queue) before new allocations happen, so a request cannot accidentally evict the prefix it just matched.

The effect is that older conversations’ tails disappear first, while their shared heads (system prompts, document preambles) survive longest because every new request refreshes them. That behavior matches the shape of most real workloads.

Leaf-first LRU in RadixAttention

SGLang’s tree makes eviction a structural decision. A node can only be evicted if no running request references it, and removing an interior node would orphan its descendants, which depend on the interior node’s KV. So eviction in the original design works on leaves: the engine evicts least-recently-used leaf nodes, and a parent becomes a leaf (and thus evictable) once all its children are gone. The paper describes an LRU eviction policy with reference counting for running requests; I am describing the leaf-first constraint as a consequence of the tree invariant rather than quoting an implementation detail. Mechanically it produces the same outcome as vLLM’s reverse-order trick: tails go first, shared trunks go last.

Worked memory-pressure example (illustrative)

Suppose an 80 GB GPU serves an 8B model in 16-bit weights (about 16 GB). After reserving activation workspace and a safety margin, say 50 GB remains for KV. At 128 KiB per token that holds roughly 50 x 1024 x 1024 / 128 = 409,600 tokens. These numbers are illustrative arithmetic, not a benchmark. If you run 64 concurrent requests at 4,000 tokens of live context each, you need 256,000 tokens, leaving about 153,600 tokens for idle cached prefixes. Double the concurrency or the context length and the cached share collapses to nothing. When engineers say “the prefix cache stopped helping at peak,” this is usually the mechanism: capacity was consumed by live sequences and the working set of hot prefixes no longer fit.

Three levers change that arithmetic:

Smaller KV per token. Grouped-query attention, multi-query attention, or multi-head latent attention reduce bytes per token. KV cache quantization (for example to 8-bit) roughly halves it, at some quality risk you must validate.
More tokens of budget. Tensor parallelism spreads weights and KV across GPUs, freeing more per-GPU KV budget. Larger memory GPUs help directly.
A second tier. Offloading evicted blocks to host RAM or storage turns eviction from “gone” into “slower to get back.” That is the subject of a later section.

Hit-rate math: what caching is worth

Define the average prompt length as L tokens and the average number of reused (cached) tokens per request as R. The token-level hit rate is h = R / L. Prefill compute for the uncached part is proportional to L – R for the linear layers (and, for attention, to the cross term between new and cached tokens). Ignoring the second-order attention term, prefill cost scales as (1 – h).

A worked example with illustrative numbers: a RAG service with a 6,000-token prompt where a 4,500-token document preamble repeats. If the cache holds that preamble, h = 0.75 and prefill work falls to about 25 percent of baseline. If prefill took 400 ms on your hardware (a made-up number for the arithmetic), the cached prefill is about 100 ms, plus a small lookup overhead. But end-to-end latency also includes decode. If the answer is 300 tokens at 20 ms each, decode is 6 seconds, and the 300 ms saved is only about 5 percent of total request latency. TTFT improves dramatically; end-to-end latency barely moves. vLLM’s own documentation makes the same caveat: caching reduces prefill time only, and gives no benefit when generating long answers dominates runtime or when queries share no prefix.

Throughput effects are a separate story. Prefill competes with decode for GPU time. Removing 75 percent of prefill work frees capacity that the scheduler can spend on decode batches or additional requests, so cluster throughput can improve even when per-request latency is dominated by decode. The honest way to estimate savings is a two-term model:

def estimate_savings(prompt_len, cached_len, block=16,
                     prefill_tps=20000, decode_tps=50, out_len=300):
    """Illustrative model. prefill_tps and decode_tps are placeholders,
    measure your own. Block granularity floors the reusable prefix."""
    reusable = (cached_len // block) * block
    new_tokens = prompt_len - reusable
    ttft_base = prompt_len / prefill_tps
    ttft_cached = new_tokens / prefill_tps
    decode = out_len / decode_tps
    return {
        "hit_rate": reusable / prompt_len,
        "ttft_base_s": ttft_base,
        "ttft_cached_s": ttft_cached,
        "e2e_base_s": ttft_base + decode,
        "e2e_cached_s": ttft_cached + decode,
    }

print(estimate_savings(6000, 4500))

Plug in measured prefill and decode rates for your model and GPU; the shape of the answer (large TTFT gain, small end-to-end gain for long outputs) is robust, the specific numbers are not.

The hit rate you achieve in production is almost always lower than the hit rate you could achieve in theory, for four reasons: eviction under memory pressure, routing that scatters related requests across replicas, prompts that embed volatile content (timestamps, request IDs, user names) early in the prefix, and cache cold starts after a deploy. The rest of this post is largely about those four leaks.

Workload Patterns: Where Prefix Caching Wins and Where It Does Not

Prefix caching rewards structure in how prompts are assembled. The same engine can deliver a 90 percent hit rate on one service and 5 percent on another, purely because of prompt layout.

Multi-turn chat

In a conversation, turn n sends the system prompt, all previous user and assistant messages, and the new user message. The previous turn’s full context is, token for token, a prefix of the new one. In principle the hit rate for turn n approaches (L – new_message_tokens) / L, which rises toward 1 as the conversation grows. Two details break the ideal. First, the assistant’s reply was generated token by token and cached as part of the previous request’s blocks, but chat templates may re-render it with different whitespace or special tokens on the next turn, which changes the token sequence and invalidates the cache from that point on. Second, the last partial block of the previous turn is not cached by vLLM (only full blocks are), so each turn recomputes up to one block. Keep the rendering of history byte-stable and you retain nearly everything.

RAG

Retrieval-augmented generation looks cache-friendly because the system prompt and instructions repeat, but the retrieved chunks usually differ per query and often appear in different orders. Exact-prefix caching only helps up to the first divergent token. Practical consequences:

Put the static instruction block and any pinned reference material first, retrieved chunks after.
If the same document set is queried repeatedly (a policy manual, a codebase, a long contract), place it as a stable, ordered prefix. Sort retrieved chunks by a deterministic key (document ID) rather than relevance score if answer quality allows, so overlapping retrievals share more prefix.
For chunk-level reuse regardless of position, exact-prefix caching is the wrong tool. LMCache’s CacheBlend (EuroSys 2025, per the LMCache project’s paper list) targets this case by reusing cached chunk KV at non-prefix positions and recomputing a subset of tokens to repair cross-attention between chunks. It trades exactness for reuse, so treat it as an approximation you must evaluate on your quality metrics.

Agents and tool-using loops

An agent step typically appends a tool call and a tool result to a growing transcript, then calls the model again. Every step shares the whole previous transcript as a prefix, and many agents share the same tool schema across all sessions. This is the workload RadixAttention was built around: the tree holds the shared tool schema as a trunk, each session as a branch, and each step as a deeper node. The paper’s reported gains of up to 6.4x are across a mix of workloads including agent control and structured decoding; do not read that as a number you will reproduce on your own traffic.

Agents expose one more failure mode: context management that mutates history. Frameworks that summarize or truncate old messages, reorder tool results, or inject a fresh timestamp into the system prompt each call invalidate the prefix cache at the mutation point. A one-token change at position 50 invalidates everything after it. When you cannot avoid mutation, append instead of rewriting: place volatile data at the end of the prompt.

Few-shot and batch evaluation

Offline evaluation jobs that run the same few-shot header across thousands of test items are the easiest win. The header is cached once; each item pays only for its own tokens. Sorting the batch so items sharing longer prefixes run adjacently also improves locality under limited capacity.

Where it does not help

Short prompts (the cached portion is smaller than a block or smaller than the lookup overhead), unique long prompts such as one-off document summarization with no repetition, and decode-dominated workloads like long-form generation see little benefit. Verify before assuming: both engines expose hit-rate metrics, and measuring on real traffic is cheap.

Cache-Aware Routing: Making the Hit Rate Survive Replication

A prefix cache lives on one GPU worker. The moment you run more than one replica behind a load balancer, a request that would have hit worker A’s cache can be sent to worker B, which has never seen the prefix. Round-robin routing divides your hit rate by roughly the replica count for workloads without strong session stickiness. This is the quiet reason many scaled deployments see far lower hit rates than single-node tests.

Figure 3: A cache-aware router keeps an approximate radix tree per worker, sends requests to the longest prefix match, and falls back to load-based routing when workers are imbalanced. Host RAM and shared stores extend each worker’s capacity.

SGLang router

SGLang’s router documentation lists four policies selectable with --policy: random, round_robin, cache_aware (the default), and power_of_two. The cache-aware policy works as follows. The router maintains an approximate radix tree for each worker, built from the requests it has routed there (it does not query the workers’ actual caches, hence “approximate”). It first checks load balance: if the gap between the most and least loaded workers exceeds --balance-abs-threshold (default 32) and the busiest worker’s load exceeds --balance-rel-threshold (default 1.0001) times the least loaded one’s, the system is considered imbalanced and the router uses shortest-queue routing. Otherwise it routes to the worker with the longest prefix match, provided the match ratio exceeds --cache-threshold (default 0.5); if no worker matches well enough, it picks the worker with the most available cache capacity. The router periodically evicts least-recently-used entries from its trees (--eviction-interval-secs, default 60) and bounds tree size with --max-tree-size (default 16,777,216 nodes). These defaults are from the SGLang router documentation at the time of writing and may change between releases.

# Illustrative: launch the SGLang router in cache-aware mode
python -m sglang_router.launch_router \
  --worker-urls http://w1:30000 http://w2:30000 http://w3:30000 \
  --policy cache_aware \
  --cache-threshold 0.5

The design embodies an explicit tension. Pure cache affinity concentrates a hot prefix on one worker, which overloads it; pure load balancing destroys locality. The threshold pair is the dial between them. Tuning it requires knowing your prefix popularity distribution: if one system prompt covers 80 percent of traffic, strict affinity sends 80 percent of requests to one worker, and the imbalance fallback will constantly override it. In that case the right answer is replicating the hot prefix on every worker (every worker will naturally cache it after a few misses) and using affinity only for the long-tail prefixes, such as per-session history.

Session affinity as the cheap alternative

For chat and agents, a simpler technique often captures most of the gain: sticky sessions. Route by a session or conversation ID hash, so every turn of one conversation lands on the same worker. You lose cross-session sharing of long prefixes unless the prefix is globally hot, but you avoid maintaining a router-side tree. Consistent hashing over the session ID also limits cache loss when replicas are added or removed. Combine it with a load-aware escape hatch for overloaded workers, mirroring the router’s imbalance fallback.

Engine-reported state versus router-inferred state

An approximate tree drifts: the router thinks worker B holds a prefix that B already evicted, or it does not know about blocks B cached from direct traffic. A more precise design has engines publish cache events (blocks stored, blocks evicted) to the router, and the router indexes them by block hash. This is where vLLM-style hashes are convenient, since a block hash is a compact, canonical identifier. Reproducibility matters here: the sha256_cbor and xxhash_cbor options exist precisely because default pickle-based hashes may not reproduce across Python or vLLM versions, which would break any external index keyed on them. If you plan to build or adopt hash-keyed routing, standardize the hash algorithm and pin versions.

Interaction with Chunked Prefill, Preemption, and Scheduling

Chunked prefill splits a long prompt’s prefill into fixed-size token chunks that are interleaved with decode steps in the same batches, bounding how long any one prefill can stall decoding. Prefix caching and chunked prefill compose cleanly because they act on different parts of the prompt: the cache hit determines the starting offset, and chunking determines how the remaining suffix is processed.

Mechanically, a request with a 10,000-token prompt and 6,000 cached tokens begins with 6,000 tokens already “computed.” The scheduler then chunks the remaining 4,000 tokens across steps. Blocks become eligible for caching as they fill with computed tokens, so a chunked prefill progressively publishes newly completed blocks to the cache. A second request arriving mid-way can already hit the first request’s completed blocks. This has a subtle benefit under bursty arrival: when many requests share a long uncached prefix and arrive within the same second, the first one’s chunks become visible to later arrivals before the first request finishes. Whether two simultaneous requests dedupe the in-flight prefix, or both compute it, depends on the engine’s scheduling order and release; it is worth testing with a synthetic burst rather than assuming.

There is one structural constraint worth knowing. To sample the first output token, the model needs logits for the last prompt position, so at least the final prompt token must go through the forward pass even when the entire prompt is cached. Engines handle this by capping the usable hit length below the full prompt length. So a 100 percent cache hit still has a small non-zero prefill.

Preemption interacts with the cache in a helpful way. When memory runs out and the scheduler preempts a running request by freeing its blocks, those blocks enter the free queue with their hashes intact. If they have not been reallocated by the time the request resumes, the resumed request matches its own earlier blocks and recomputes much less than a naive restart would. Under heavy pressure, though, the freed blocks are the first things overwritten, and resumption degrades to a full recompute. This is another face of the capacity cliff from the previous section.

Speculative decoding and prefix caching coexist, but note the hybrid-model caveat in vLLM’s documentation: for hybrid Mamba/attention models, recurrent state is stored only on the Mamba block grid, so cache hits resume at block boundaries, and a shared-prefix checkpoint flag (off by default) exists to store state at the junction between a shared system prompt and divergent content. Hybrid architectures do not get the same free granularity as pure attention, so check the documentation for your model family before relying on hit rates measured elsewhere.

Beyond One GPU: KV Offload, HiCache, and LMCache

GPU memory is the scarcest tier. Host RAM is an order of magnitude larger and storage far larger still, and a cache hit from host memory is usually much cheaper than recomputation for long prefixes, because moving KV bytes over PCIe or NVLink is cheaper than recomputing attention and MLP activations for thousands of tokens. The break-even depends on prefix length, interconnect bandwidth, and model size: for a 128 KiB-per-token cache, loading 10,000 tokens is about 1.25 GiB, which at a hypothetical effective 20 GB/s takes roughly 60 ms (illustrative arithmetic, not a measured figure), while recomputing 10,000 tokens through an 8B model is a much larger compute bill. Short prefixes flip the calculus because transfer latency and scheduling overhead dominate.

SGLang HiCache: a three-tier radix tree

SGLang’s hierarchical cache (HiCache) extends RadixAttention with tiers. According to the SGLang documentation, L1 is GPU memory private to one instance, L2 is host memory private to one instance on its node, and L3 is a shared storage backend whose reach depends on configuration, with named backends including file, mooncake, hf3fs, nixl, and aibrix. The HiRadixTree extends the original tree so each node records where its KV lives (GPU, CPU, L3, or several). Local tiers have exact addresses; L3 metadata is not synchronized and is queried from the backend on demand.

The workflow is a local match across L1 and L2 with no data copying, then, if the L3 hit exceeds a threshold (default 256 tokens), a prefetch of the missing portion into L2, then a write-back of newly computed KV after prefill. Three prefetch policies trade hit rate for latency: best_effort (stop as soon as prefill can proceed), wait_complete (wait for everything, maximizing hit rate), and timeout (stop at a deadline, which the docs recommend for production SLOs). Write policies are write_through, write_through_selective (only blocks whose access count passes a threshold), and write_back (write to the lower tier only on eviction from the upper). Under tensor parallelism, ranks agree on L3 hit length with an all_reduce using the minimum, so every rank loads the same prefix length.

LMCache: an engine-external KV layer

LMCache takes a different position: KV management as a layer outside the inference engine. Its project page describes a KV cache management layer that stores caches outside the engine so they can be reused across requests, sessions, and engine instances, with tiers spanning CPU memory, local disk, and remote backends including Redis or Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, and GDS. It also supports non-prefix reuse via CacheBlend, prefill-decode disaggregation transfers over NVLink, RDMA, or TCP, and a pluggable serialization interface for compression. The associated papers are the LMCache systems paper (arXiv:2510.09665), CacheGen (SIGCOMM 2024) for KV compression and streaming, and CacheBlend (EuroSys 2025). The project page makes qualitative claims about lower TTFT and higher throughput for long-context, multi-turn, and RAG workloads and does not give headline numbers, so I will not either; benchmark on your own traffic.

The architectural question is where to put the cache boundary:

Concern	Engine-internal tiers (HiCache)	Engine-external layer (LMCache)
Integration depth	Tight, tree-aware, tuned to one engine	Connector per engine; engine-agnostic by design
Cross-instance sharing	Via L3 backend	Via shared remote backends
Survives engine restart	Only data in L3	Yes if stored outside the engine process, per the project description
Prefill-decode disaggregation	Possible via backends	First-class transfer path
Non-prefix reuse	Not the core model	CacheBlend support
Operational surface	One system	Extra service to deploy and monitor

Offload is not free: write-through policies consume PCIe and network bandwidth that the model’s tensor-parallel all-reduce may also need, and prefetching can delay a request that would have run sooner by simply recomputing. That is why the timeout prefetch policy and selective write-through exist. Treat offload as a tuned subsystem with its own SLO, not a switch.

Head-to-Head: RadixAttention vs vLLM Automatic Prefix Caching

Dimension	SGLang RadixAttention	vLLM automatic prefix caching
Lookup structure	Radix tree over token runs	Hash table of chained block hashes
Match granularity	Token level in the tree; page level when `page_size > 1`	Full blocks only
Eviction	LRU over unreferenced nodes, leaf-first by tree invariant	LRU free queue with reverse-order tail insertion
Multi-tenant isolation	Prefix-keyed; see salting discussion below	Per-request `cache_salt` mixed into the first block hash
Router integration	Cache-aware router with approximate per-worker trees	Hash-keyed external indexes and KV events (deployment dependent)
Tiering	HiCache L1/L2/L3	Via connectors such as LMCache
Hash or key control	N/A (token keys)	Selectable: sha256, sha256_cbor, xxhash, xxhash_cbor
Primary strength	Branching workloads, structured programs, agents	General serving, large ecosystem, simple mental model

Feature parity has been converging for a while, and both projects ship fast, so check release notes for your version. My opinion, labeled as such: for new deployments with heavy agent or branching workloads, the tree-plus-router combination is the more cohesive story; for teams already standardized on vLLM, enabling APC and fixing prompt layout recovers most of the available gain without migrating. For an independent look at throughput under realistic mixes, see our SGLang vs vLLM vs TensorRT-LLM benchmark analysis.

Trade-offs, Gotchas, and What Goes Wrong

Prefix caching is easy to turn on and surprisingly easy to defeat. The failure modes below account for most of the “it doesn’t help” reports.

Cache salting and tenant isolation

A shared prefix cache is shared state, and shared state across trust boundaries leaks information. vLLM’s design documentation describes an optional per-request cache_salt injected into the first block’s hash: only requests with the same salt can reuse cached blocks, which limits cross-tenant timing inference. The cost is explicit: salting partitions the cache, so identical system prompts across tenants no longer share KV and your hit rate and memory efficiency drop. Decide the isolation unit deliberately. A per-organization salt keeps intra-organization sharing while isolating customers; a per-user salt is stricter and costlier; no salt maximizes sharing and is appropriate only for single-trust-domain deployments.

Timing side channels

A cache hit makes the first token arrive sooner. If an attacker can send requests to a server that shares a cache with a victim, then guessing a candidate prefix and measuring TTFT reveals whether someone sent it before. The paper “Auditing Prompt Caching in Language Model APIs” (Gu, Li, Kuditipudi, Liang, Hashimoto, arXiv:2502.07776) audited seven API providers and reported global cache sharing in all seven, with timing differences enabling detection of cached prompts; it also reports that the timing effects can leak architecture details. Exploitability on a self-hosted deployment depends on whether untrusted parties can submit requests to the same engine and measure latency precisely.

Figure 4: An attacker times the first token for guessed prefixes. A hit is fast; a miss is slow. Under a different salt the victim’s blocks cannot match, so the signal disappears.

Mitigations, in order of strength: isolate tenants with salts or separate deployments; avoid exposing fine-grained timing (batching, jitter and rate limits raise attacker cost but do not eliminate the channel); and never put secrets (API keys, personal data) into a shared prefix that other principals could probe. Hash choice matters here too: with a non-cryptographic hash such as xxhash, a crafted collision becomes conceivable, so use a cryptographic option in multi-tenant systems.

Tokenization boundaries

The cache keys on tokens, not characters. Two prompts that share a long character prefix may tokenize differently near the boundary, because byte-pair encoding merges can cross the join point: a trailing space, a changed final word, or a newline convention can alter token IDs for a few tokens before the divergence. In practice, the last token or two of a shared string may differ depending on what follows it, which truncates the match. Conversely, templating that inserts special tokens (role headers, end-of-turn markers) can make semantically identical histories differ. The defensive pattern is to have a fixed, token-stable delimiter between the static prefix and the variable tail, and to compute hit rates on token IDs in testing, not on strings.

Other common failure modes

Volatile prefix content. Timestamps, request IDs, random few-shot ordering, or per-user greetings near the start of the prompt zero out the hit rate. Move them to the end.
Block granularity loss. Short shared prefixes just over a block boundary waste a partial block; larger blocks make this worse.
Non-determinism. Cache hits change batch composition and kernel shapes, so floating-point results can differ slightly from an uncached run. Do not require bitwise reproducibility across cache states.
Cold starts. After a deploy or autoscale event, caches are empty and the first requests pay full prefill. Warm hot prefixes deliberately with a startup replay, or keep them in an external store.
Metric blindness. Teams enable caching and never check hit rate. Track token-level hit rate, evictions per minute, and TTFT split by hit and miss.
Over-trusting averages. A 60 percent average hit rate can hide a bimodal distribution: sessions with 95 percent and sessions with zero. Segment by route, tenant, and prompt template.

Practical Recommendations

Start from the prompt, not the engine. Enable prefix caching (it is a single flag in vLLM; SGLang’s RadixAttention is part of its default runtime) and then spend your effort on the factors that determine real hit rates.

First, restructure prompts as stable-first, volatile-last: system instructions, tool schemas, and pinned documents at the top in a fixed order; retrieved chunks, user messages, and timestamps below. Make history rendering deterministic and append-only. Second, size the cache honestly: compute bytes per token from your model config, multiply by your hot-prefix working set, and compare it to the KV budget left after peak concurrency. If the working set does not fit, add a host-RAM or shared tier before buying GPUs. Third, route for locality: use session-sticky routing at minimum, and a cache-aware router when you have many long-tail shared prefixes, tuning the balance thresholds against your real prefix popularity. Fourth, decide isolation up front, because retrofitting salts later resets your cache and changes your economics.

A short checklist:

[ ] Prefix caching enabled; hit-rate and eviction metrics exported and dashboarded.
[ ] Static content first; no timestamps, IDs, or per-user text in the first block.
[ ] Chat template rendering is deterministic and verified at the token-ID level.
[ ] KV bytes per token computed; hot-prefix working set compared with free KV budget at peak.
[ ] Routing preserves locality (sticky sessions or cache-aware policy) with a load-aware fallback.
[ ] Tenant isolation decision documented: salt scope, hash algorithm (sha256_cbor for reproducibility), and timing exposure reviewed.
[ ] Offload tier, if used, has a prefetch timeout and a bandwidth budget.
[ ] Warm-up procedure for deploys and autoscaling.
[ ] Hit-rate target and TTFT SLO tracked per route, not just globally.

Frequently Asked Questions

What is LLM prefix caching?

LLM prefix caching stores the key-value tensors computed for a prompt’s tokens and reuses them when a later request begins with the same token sequence. The engine skips prefill for the matching prefix and computes only the new suffix. It lowers time-to-first-token and prefill GPU cost, does not change decode speed, and produces the same outputs as uncached execution apart from minor floating-point differences from changed batch shapes.

How is RadixAttention different from vLLM automatic prefix caching?

RadixAttention keeps cached sequences in a radix tree and finds the longest match by walking it, splitting nodes where requests diverge. vLLM’s automatic prefix caching hashes each full block together with its parent block’s hash and looks blocks up in a hash table. Both reuse exact prefixes. The tree adds structural awareness useful for eviction and routing; the hash scheme adds compact identifiers and extra keys such as LoRA IDs and cache salts.

Does prefix caching reduce decode latency?

No. Prefix caching only removes prefill work, so it improves time-to-first-token and frees compute for other requests. If your responses are long, decode dominates end-to-end latency and the visible gain per request is small. Cluster throughput can still improve because the scheduler spends less GPU time on prefill. Measure TTFT separately from total latency when evaluating the feature.

Can prefix caching leak data between users?

It can leak through timing. When tenants share a cache, a fast first token on a guessed prefix reveals that someone submitted it before; research auditing commercial APIs found global cache sharing in every provider tested. In vLLM you can supply a per-request cache salt so only same-salt requests share blocks. Use cryptographic hashing, isolate tenants, and keep secrets out of shared prefixes.

Why is my prefix cache hit rate low?

Usual causes are volatile content early in the prompt (timestamps, IDs), chat templates that re-render history differently, tokenization differences at the join point, routers that scatter related requests across replicas, and eviction under memory pressure at high concurrency. Check token-level hit rate by route, confirm that the prompt prefix is byte-stable, compute free KV capacity at peak load, and add session-sticky or cache-aware routing.

Should I offload KV cache to CPU memory or LMCache?

Offload when your hot-prefix working set exceeds GPU KV budget and prefixes are long enough that loading KV bytes beats recomputing them, as with long documents and agent transcripts. Skip it for short prompts, where transfer overhead dominates. Use a prefetch timeout so a slow tier cannot hurt TTFT, budget PCIe and network bandwidth, and benchmark against your traffic before committing.