LLM Prompt Caching: Architecture and Economics (2026)

Most production LLM applications resend the same long prefix on every call. A 4,000-token system prompt, a tool schema, a retrieved policy document, a block of few-shot examples: all of it is recomputed from scratch, request after request, while only the final user question changes. LLM prompt caching architecture and economics in 2026 come down to one idea: store the attention state the model already computed for that shared prefix, and pay a fraction of the price to reuse it.

It matters now because caching has stopped being an optimization and become a pricing structure. Every major provider bills cache reads at a steep discount, several bill cache writes at a premium, and the multipliers changed again this year. A team that lays out its prompt carelessly can pay the write premium on every request and never collect a single read. A team that designs for the cache routinely cuts the input side of the bill by more than half.

You will leave with a precise mental model of how prefix reuse works inside the transformer, a comparison of provider-side and self-hosted caching, layout rules that raise hit rate, a break-even formula you can apply to any vendor’s multipliers, and a failure-mode catalogue.

What this covers: the KV cache mechanics, provider versus self-hosted architectures, cache-aware prompt design, a worked cost model with November 2026 verified multipliers, failure modes, and a practical checklist.

Context and Background: Why Prompt Caching Exists

To see why caching works, look at what a model does with your prompt. Inference has two phases with very different cost profiles. In prefill, the model reads every input token in parallel and computes a key and value vector for each token at every attention layer. In decode, it generates output tokens one at a time, each attending back over all stored keys and values. Prefill is compute-bound and parallel. Decode is memory-bandwidth-bound and sequential.

The stored keys and values are the KV cache, and they are the most expensive artifact the pipeline produces. For a long prompt, prefill dominates time-to-first-token and a large share of input cost. Work grows with input length, so an 8,000-token system prompt costs far more to process than the 300-token question after it. When that 8,000-token block is identical on every request, you pay the heaviest part of the bill repeatedly for zero new information.

The key structural fact is causality. In a decoder-only transformer, the KV pair for token i depends only on tokens 0 through i. It never depends on anything later. So two requests that begin with the same token sequence produce byte-identical KV state for that shared prefix. There is no reason to compute it twice.

That is why caching keys on the prefix, not the whole prompt. If request A is [system][docs][question_A] and request B is [system][docs][question_B], the [system][docs] portion yields identical state in both. The second request skips prefill up to the point where it diverges. The catch is that the match must be contiguous from token zero. Change one token near the front and every downstream KV pair is invalid, because each depended on the thing you changed. Almost every design decision in this article follows from that one constraint. For the underlying memory mechanics, see our guide to KV cache optimization for LLM inference.

Two workload shapes benefit most, and together they cover a large fraction of real applications:

• Repeated queries over a long document. Many users, or one user, ask different questions about the same annual report, manual or codebase. The document is prefilled once and every later question reuses the state.
• Multi-turn conversation and agents. Turn N shares its entire history with turn N+1. Without caching, a long chat re-prefills the full transcript on every message, so cumulative input grows quadratically with turn count. With caching, each turn prefills only its new tokens.

Agents are the extreme case. A coding or research agent loops through tool calls, and each loop iteration resends the system prompt, the tool definitions, and the entire growing trace. The marginal new content per step is a few hundred tokens; the resent content is tens of thousands. That ratio, a large stable prefix to a small new tail, is the single best predictor of caching value.

One more piece of context is the vocabulary. Providers use different words for the same machinery: “prompt caching” (Anthropic, OpenAI), “context caching” (Google), “prefix caching” (vLLM, SGLang), and “KV reuse” in research. This post uses them interchangeably unless noted, and distinguishes provider-side caching, where you rent the behavior behind a billing meter, from self-hosted KV reuse, where you own the memory and the eviction policy. The mechanics rhyme. The economics and failure modes do not.

The Prompt Caching Architecture: One Control Flow, Two Worlds

Prompt caching stores the KV state of a shared prompt prefix under a hash key, then reuses it on the next request with an identical prefix, skipping prefill for that portion. A hit lowers latency and input cost. A miss pays full prefill and, on most providers, writes the prefix to the cache for later requests.

Figure 1: The universal prompt caching control flow. A request is hashed block by block from token zero. A hit loads stored KV state and prefills only the new suffix; a miss prefills everything and writes the prefix back.

Figure 1 is the control flow shared by every implementation. The request is normalized into an ordered sequence. On Anthropic’s API that order is fixed as tools, then system, then messages. The system hashes the sequence in blocks, looks for the longest stored match, and loads that state. Only the tokens past the match go through prefill. Then decode runs as usual, and the usage record tells you how many tokens were read, written, or processed uncached.

Provider-side caching: three vendors, one shape

Hosted APIs from Anthropic, OpenAI and Google share a common shape with different surface details. All store prefix state in a fast tier, look it up by hash, and bill reads at a discount. Three properties define the model: a lifetime, a scope rule, and a read/write price asymmetry.

Lifetime. A cached prefix expires after an idle window. Anthropic’s default is 5 minutes, refreshed at no cost each time the entry is used, with a 1-hour option at additional cost. Per the Anthropic prompt caching documentation, the lifetime is measured from the start of the request that wrote or read the entry, so a slow generation eats into it. OpenAI’s newer models (GPT-5.6 and later) take prompt_cache_options.ttl with "30m" as default and "12h" as the long option. Earlier OpenAI models use prompt_cache_retention with in_memory or 24h semantics, and GPT-5.5 supports only 24h, per the OpenAI prompt caching guide. Google’s Gemini caching documentation does not state a default lifetime for implicit caching on the page I checked, so I make no claim about it.

Scope. Providers differ in how much control you get. Anthropic offers explicit breakpoints (up to four cache_control markers per request) and, new in this generation, an automatic mode: add one top-level cache_control field and the system places the breakpoint on the last cacheable block, moving it forward as the conversation grows. It consumes one of the four slots, so it combines with explicit markers. Specifying a different TTL on the last block than the automatic one returns a 400 error. OpenAI applies implicit breakpoints at the end of the latest eligible message on GPT-5.6 and later, with explicit prompt_cache_breakpoint supported and up to four cache writes per request; older models cache implicitly at model-dependent intervals. Gemini 2.5 and newer cache implicitly by default, and explicit caching remains for supported Gemini 3 and 2.5 models through generateContent, but not for Gemini 4 Argon and later or the Interactions API.

Price asymmetry. Writing a prefix costs at least as much as normal input because the provider performs the prefill and persists the state. Reading costs a small fraction. This asymmetry is the heart of the economics, modelled in detail below.

Self-hosted KV reuse: you own the cache

When you run open-weight models yourself, prefix caching belongs to the serving engine. Two designs dominate.

vLLM automatic prefix caching reuses KV blocks across requests with no change to your prompts. The vLLM design page describes each block hashed from three inputs: the parent block’s hash, the block’s own tokens, and “extra hashes” such as LoRA IDs, multimodal input hashes and cache salts. Only full blocks are cached, so a prefix shorter than a block never hits, and the tail of a prefix that ends mid-block is recomputed. Identical prefixes hash to identical physical blocks, which are shared automatically. Eviction is LRU over a free queue, with a detail worth knowing: when a request finishes, its blocks are appended in reverse order so the last blocks, which cover the most tokens and are least likely to be shared, are evicted first.

A minimal setup looks like this:

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    enable_prefix_caching=True,
    gpu_memory_utilization=0.90,
)

shared_prefix = SYSTEM_PROMPT + RETRIEVED_DOCS   # stable across requests
for question in questions:
    prompt = shared_prefix + question            # only the tail changes
    out = llm.generate(prompt, SamplingParams(max_tokens=256))

I did not verify in the design page whether prefix caching is on by default in the current release, so set the flag explicitly rather than assuming. The docs are also clear that the feature accelerates prefill only. If your workload is decode-heavy, with short prompts and long outputs, the prefix barely matters.

SGLang RadixAttention takes a structural approach. It stores the KV state of in-flight and recent requests in a radix tree, a prefix tree whose edges are token sequences, and evicts with LRU. The SGLang paper (arXiv 2312.07104, December 2023) reports “up to 6.4x higher throughput compared to state-of-the-art inference systems” across its workloads, a number that depends heavily on how much prefix sharing the benchmark workloads contain. Treat it as an upper bound, not an expectation. The radix tree excels when prompts form a branching hierarchy: one system prompt, many documents under it, many questions under each document.

Figure 3: A radix tree shares ancestor nodes across requests; HiCache extends the same state across GPU memory, host DRAM and shared storage.

The tree in Figure 3 is the mental model. A shared system prompt sits near the root and branches into document contexts, which branch into individual questions. Every node is KV state computed once and reused by all descendants. Two requests that share the system prompt and the document but ask different questions share the whole path down to the divergence point.

Where the cache lives: tiers beyond GPU memory

GPU memory is fast and scarce. The warm set you can keep is bounded by VRAM left after in-flight requests claim theirs. SGLang’s HiCache design addresses this with three tiers, per its design page: L1 in GPU memory, L2 in host DRAM, and L3 in a storage backend. L1 and L2 are private to one instance. L3 is shared across instances only if the backend is configured to span them, which makes cross-replica reuse a storage decision rather than an engine default.

Listed L3 backends include a demonstration-grade file backend, Mooncake (RDMA and multi-NIC), DeepSeek’s 3FS, NIXL (a unified layer over plugins such as 3FS, GPUDirect Storage and S3-compatible stores), and AIBrix KVCache, with LMCache as an alternative via --enable-lmcache. Three write policies govern movement: write_through pushes every access down immediately, write_through_selective pushes only data past an access-frequency threshold, and write_back writes only on eviction from the upper tier.

Prefetching from L3 to L2 has three policies. best_effort stops as soon as GPU prefill can proceed, wait_complete waits for all prefetches and favors hit rate, and timeout stops at a time limit, which the docs recommend for production latency objectives. The documented default trigger is 256 tokens of L3 hit length, and the timeout formula is min(max, base + per_ki_token x tokens / 1024) with defaults of 2 seconds base, 0.1 seconds per 1,024 tokens, and a 30-second cap. Those defaults mean a 100,000-token L3 hit would be capped by its own computed timeout of roughly 12 seconds, a reminder that cold-tier recall is not free. For the disaggregated-serving angle on the same idea, see our article on NVIDIA Dynamo, LMCache and KV cache offload.

The practical choice depends on traffic shape. A handful of hot system prompts fits comfortably in VRAM. A long tail of per-customer contexts, each recurring hours apart, is where a tiered design earns its complexity, because it converts a would-be miss into a slower hit.

Provider-side versus self-hosted: what actually differs

Dimension	Provider-side caching	Self-hosted (vLLM, SGLang)
Who pays for memory	Embedded in per-token price	You, as GPU and host RAM
Lifetime control	Fixed menu (minutes to hours)	Eviction policy and tier sizes
Visibility	Usage fields per response	Engine metrics, block-level
Multi-tenant isolation	Per workspace or organization	Your job: salts, key design
Cost of a miss	Write premium plus full prefill	Prefill compute only
Cross-replica reuse	Provider routing, best effort	Requires shared L3 or router

The last row deserves emphasis. Hosted caches live on specific machines. OpenAI’s guide notes that traffic above roughly 15 requests per minute for one prefix can overflow to other machines, reducing reuse. Self-hosted fleets have the same problem unless a cache-aware router sends requests with a shared prefix to the replica that holds it.

Cache-Aware Prompt Design: Hit Rate Is a Design Choice

Hit rate is not luck. Because the match must be contiguous from token zero, the first rule is simple: everything static goes at the front and everything variable goes at the back.

Figure 2: The bad layout opens with a timestamp and request ID, invalidating everything after it. The good layout front-loads stable content and places the variable parts after the cache breakpoint.

Figure 2 is the whole discipline in one picture. The bad layout starts with values that change on every call, so the first block hashes differently each time and nothing downstream can ever match. The good layout orders content by volatility: system instructions, tool definitions, pinned reference documents, a breakpoint, then the user’s question, then per-request metadata.

Rules that follow from the prefix constraint

• Never lead with a timestamp, UUID or session ID. Move it after the question, or pass it out of band if the model does not strictly need it inline.
• Keep the system prompt and tool schema byte-stable. Reordering JSON keys, changing whitespace on serialization, or running an A/B test that injects variant wording mints a new prefix. On Anthropic’s API, any change to tool definitions (names, descriptions, parameters) invalidates the tools, system and messages caches, because tools come first in the order.
• Order RAG context from most stable to least stable. A pinned policy document belongs above freshly retrieved chunks. If retrieval order is nondeterministic, sort chunks by document ID before assembly so identical retrievals produce identical token sequences.
• Put the breakpoint after the largest stable block. On providers with explicit control, mark the end of reusable content so you write it once and read it many times.
• Respect the lookback window. Anthropic’s documentation says the system checks at most 20 block positions per breakpoint. Consecutive tool_use blocks, and consecutive tool_result blocks, each count as one position. A long agent trace with many alternating blocks can push an earlier cached entry out of reach unless you add a second breakpoint nearer the end.

Anthropic’s invalidation table adds nuance you can exploit. Toggling web search, citations or the speed setting leaves the tools cache intact but invalidates system and messages. Changing tool_choice or adding and removing images keeps tools and system but invalidates messages. On the newest model generations, you can append a {"role": "system"} message mid-conversation without invalidating the system or message caches, which is a clean way to inject late-breaking instructions without breaking the prefix. The docs note this is not available on every model, so check your model’s row first.

A concrete request, annotated

Here is a minimal request using the automatic mode on Anthropic’s API, reproduced from the pattern in the documentation:

import anthropic
client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},      # automatic breakpoint, 5 minute TTL
    system=[{"type": "text", "text": LONG_POLICY_TEXT}],
    messages=[{"role": "user", "content": question}],
)

u = response.usage
print(u.cache_creation_input_tokens,   # written this request
      u.cache_read_input_tokens,       # read this request
      u.input_tokens)                  # uncached tail after the breakpoint

The model identifier above follows the docs’ naming for the Sonnet 5.5 tier; confirm the exact string for your account. The line that matters is the usage print. Total input equals cache_read_input_tokens + cache_creation_input_tokens + input_tokens. If both cache fields are zero, the prompt was not cached, most often because it fell under the model’s minimum cacheable length. Anthropic documents 512 tokens for the 5.5 generation and 5 generation, 1,024 for several Sonnet and older Opus models, 2,048 for Opus 4.7 and Haiku 3.5, and 4,096 for Opus 4.6, Opus 4.5 and Haiku 4.5. Short prompts are processed uncached without any error, so a missing hit looks like a silent no-op.

OpenAI reports the same facts in different fields: usage.input_tokens_details.cached_tokens for billed reads and cache_write_tokens for writes, with a per-lifetime split. On earlier models cached_tokens is rounded down to a multiple of 128 and excludes hidden system tokens. Gemini reports hits in usage.total_cached_tokens.

Version the prefix like an artifact

Teams that sustain their savings treat the cached prefix as a versioned artifact. Every tweak to the system prompt, every added tool, every change to the document set mints a new prefix and triggers a fresh wave of writes as traffic migrates. That is fine when intentional. It is a quiet tax when accidental: a config refactor that reorders fields, a library upgrade that changes JSON formatting, an experiment that injects variant text.

Make prefix changes explicit and rare. Keep the cacheable region behind a single reviewed code path. Log a prefix version identifier on every request so a hit-rate dip correlates with a deploy. When you must change it, schedule the change and expect a transient cost bump while caches warm. A useful trick: hash your assembled prefix in CI and fail the build when it changes without a version bump.

Measure token-level hit rate

Cache-aware design is a hypothesis until production traffic confirms it. The metric that matters is the share of cacheable tokens served as reads, not the share of requests with any hit, because a request can be a partial hit. On hosted APIs, sum cache-read tokens against total cacheable tokens from the usage fields. On vLLM or SGLang, scrape the engine’s cache metrics.

Watch the rate over a full traffic cycle. A hit rate that looks healthy at midday can collapse overnight when gaps between requests exceed the TTL, and the monthly bill reflects the average, not the best hour. Alert when the measured rate falls below the break-even threshold your cost model implies, which is the subject of the next section.

The Economics: A Worked Cost Model

This is where caching earns its keep or quietly betrays you. Let me build the model from the verified multipliers, then work a concrete example.

What the vendors charge, as of 2 November 2026

I checked these against each vendor’s public documentation on 2 November 2026. Rates change often; re-verify before budgeting.

Vendor and tier	Write cost	Read cost	Source note
Anthropic, 5-minute TTL	1.25x base input	0.1x on most models	Docs table
Anthropic, 1-hour TTL	2x base input	same as above	Docs table
Anthropic Opus 5.5 and Sonnet 5.5	as above	0.05x	Docs exception
Anthropic Fable 5.1 and Mythos 5.1	as above	0.025x	Docs exception
OpenAI GPT-5.6 and later, 30m	1.25x	0.1x	Guide; 0.05x for GPT-6.1 Sol and Terra
OpenAI GPT-5.6 and later, 12h	2.5x	0.1x	Guide
OpenAI GPT-5.5 and earlier	no extra write charge	discounted, up to 95%	Guide; rate per pricing page
Google Gemini	storage billed per hour on some models	listed per model	Pricing page

For concrete dollars, Anthropic publishes a per-model table. Claude Sonnet 5.5 is listed at $2 per million input tokens, $10 output, $2.50 for a 5-minute write, $4 for a 1-hour write, and $0.10 for a cache hit. Claude Opus 5.5 is $4 input and $0.20 per million cache hit. Claude Haiku 5.5 on prompts of 100,000 tokens or fewer is $0.10 input and $0.01 per million cache hit. Older entries show the previous shape: Claude Sonnet 4.6 lists $3 input and $0.30 hit, a plain 0.1x.

Google’s pricing differs in kind. Gemini 3.8 Flash and 3.6 Flash list $0.075 per million cached tokens through 31 December 2026, rising to $0.15 from 1 January 2027, with storage at $0.50 rising to $1.00 per million tokens per hour. Gemini 4 Argon lists $0.10 through 30 November 2026, rising to $0.20 from 1 December, with no storage price shown, and supports implicit caching only. Those scheduled increases mean a budget built on today’s cached rate is already stale for 2027. Model the post-change figure.

I did not verify OpenAI’s per-model dollar rates this run, because the guide defers to the pricing page. Use the multipliers above and the live price list.

The break-even formula

Let w be the write multiplier, r the read multiplier, and h the token-level hit rate on the cacheable prefix. Misses on a caching provider pay the write rate, since the miss also populates the cache. Relative cost of the prefix per request is:

cost = h x r + (1 – h) x w

Uncached cost is 1. Setting cost equal to 1 and solving gives the break-even hit rate:

h = (w – 1) / (w – r)

Plugging in the verified multipliers:

Scenario	w	r	Break-even hit rate
Anthropic 5m write, 0.05x read	1.25	0.05	20.8%
Anthropic 5m write, 0.1x read	1.25	0.10	21.7%
Anthropic 1h write, 0.05x read	2.00	0.05	51.3%
Anthropic 1h write, 0.1x read	2.00	0.10	52.6%
OpenAI 12h write, 0.1x read	2.50	0.10	62.5%

Two lessons fall out. First, the default short-lifetime tiers break even at roughly one hit in five, which is forgiving. Second, longer lifetimes demand much higher hit rates before they pay, because the write premium is larger. Buying the 1-hour tier is a bet that your gaps exceed 5 minutes often enough to matter. If your traffic arrives every 90 seconds, the 5-minute tier refreshes continuously at no extra cost and the 1-hour premium is wasted.

Worked example: a support assistant on Sonnet 5.5

All of the following uses the verified Sonnet 5.5 rates ($2 input, $10 output, 1.25x write, 0.05x read). The workload assumptions are mine and illustrative: an 8,000-token stable prefix (policy plus tool schema), a 300-token question, a 400-token answer, and one million requests per month.

Uncached, each request costs 8,000 x $2/M = $0.016 for the prefix, $0.0006 for the question, and $0.004 for the output: $0.0206, or $20,600 per month.

With caching, only the prefix term changes. At hit rate h, the prefix costs 8,000 x $2/M x (h x 0.05 + (1 – h) x 1.25):

Token-level hit rate	Monthly cost	Change vs uncached
0% (write every time)	$24,600	+19.4% worse
20%	$20,760	+0.8% worse
50%	$15,000	27.2% cheaper
80%	$9,240	55.1% cheaper
90%	$7,320	64.5% cheaper
95%	$6,360	69.1% cheaper

The 0% row is the cautionary tale: enabling caching on a prompt that never hits, with a changing prefix, costs 19.4% more than not enabling it. The 20% row confirms the 20.8% break-even from the formula. Past 80%, returns diminish because the output tokens and the question, which are never cached, dominate what remains.

If the same workload ran at the older 0.1x read rate, 90% hit would cost $8,040 instead of $7,320. The read multiplier matters less than the hit rate: moving from 50% to 90% hit saves $7,680 per month here, while halving the read multiplier at 90% saves only $720.

The multi-turn case: why chat is the best fit

Consider a 40-turn conversation where each turn adds 500 tokens. Without caching, turn k resends k x 500 tokens, so cumulative input is 500 x (1 + 2 + … + 40) = 410,000 token-reads. With caching and no TTL expiry, each turn writes only its 500 new tokens (20,000 token-writes in total at 1.25x) and reads the prior history (500 x 0 + 500 x 1 + … + 500 x 39 = 390,000 token-reads at 0.1x). In multiplier-weighted units, that is 20,000 x 1.25 + 390,000 x 0.1 = 64,000 versus 410,000, about 84% less input cost.

This ignores the system prompt, output tokens, and TTL gaps, so it overstates the real saving. It shows the shape: uncached cost grows quadratically with turns, and cached cost grows nearly linearly. The longer the conversation, the more the cache is worth.

Figure 4: Sequence of a cached workload. The first request pays the write premium, requests inside the TTL pay the read rate and refresh the timer, and an idle gap past the TTL forces a new write.

Figure 4 shows why traffic cadence is as important as prompt layout. The TTL refreshes on use, so steady traffic keeps the entry alive for free. Bursty traffic with long quiet periods pays the write premium again after every gap. A cheap mitigation on explicit-control providers is a keep-warm ping, but do the arithmetic first: a ping that reads a 20,000-token prefix at 0.05x costs the equivalent of 1,000 input tokens, while a cold rewrite at 1.25x costs 25,000. Pinging every four minutes against a five-minute TTL is cheap insurance only if real traffic would otherwise leave gaps.

Trade-offs, Gotchas, and What Goes Wrong

Caching is not free, and the failures are quiet. They show up as a bill that does not fall, not as an error.

All-unique prompts. If every request is a fresh one-shot with no shared prefix, such as a batch classifier that embeds each document inline, there is nothing to reuse. The write premium is pure waste, as the 0% row above shows. Leave caching off.

Decode-dominated workloads. Caching saves prefill only. If prompts are short and outputs long, as in long-form generation, prefill is a rounding error and the effect on cost and latency is negligible.

Mutating prefixes. A prompt that opens with live data, a rotating banner or a per-user greeting misses on every call no matter how the cache is configured. The fix is architectural: move the volatile content to the tail or to a tool call result.

Sub-minimum prefixes. A prefix under the model’s minimum length is silently uncached. Moving between models can flip a working setup into a non-working one, since minimums range from 512 to 4,096 tokens across Anthropic’s lineup and are 1,024 for OpenAI GPT-5.6 and later, 2,048 for Gemini 2.5, and 4,096 for the Gemini 3.x entries and Gemini 4 Argon that I checked. Always read the usage fields after a model swap.

TTL mismatch. Paying for a 1-hour or 12-hour lifetime when requests arrive every minute wastes the premium. Paying for the short tier when requests arrive every ten minutes means a write on nearly every call. Plot your inter-request gap distribution before choosing.

Routing and overflow. Hosted caches live on specific machines. OpenAI’s guide describes overflow routing above roughly 15 requests per minute per prefix, which can lower reuse. For earlier OpenAI models it suggests partitioning heavy groups across multiple prompt_cache_key values with a deterministic mapping. On GPT-5.6 and later the key is optional and routing is automatic, but it is still useful for per-customer accounting.

Security and tenant isolation. A shared cache is a timing side channel. If a cached prefix returns faster, an observer can infer that someone else sent it. Providers isolate caches: Anthropic isolates per workspace on the Claude API, Claude Platform on AWS and Microsoft Foundry, per organization on Amazon Bedrock and Google Cloud, and never across organizations. Self-hosting shifts the burden to you. vLLM supports a per-request cache_salt injected into the first block’s hash so only requests with the same salt reuse blocks. Its docs also warn that the faster non-cryptographic xxhash options raise collision risk, which in a multi-tenant deployment could cause undefined behavior or leak private information. The default has been sha256 since v0.11, and sha256_cbor is recommended for reproducible hashes across languages. If you serve several customers from one engine, choose the salt policy deliberately.

Observability gaps. A binary hit/miss metric hides partial hits and write waste. Instrument tokens read, tokens written and tokens uncached, per prefix version, and compute cost per request, not just hit rate.

Anti-patterns worth naming.

• Enabling caching globally “because it is free”. It is not free on providers that charge for writes.
• Mixing volatile and stable content in one text block, so a single changed word invalidates a large static portion.
• Putting the breakpoint on the final user message in a one-shot pipeline, writing content that will never be read.
• Letting different microservices assemble the “same” prompt independently with subtly different whitespace.
• Forgetting that images, thinking configuration and tool-choice changes can alter which cache levels survive, per the provider’s invalidation table.

Semantic-level reuse is a complementary technique, not a substitute. It catches near-duplicate questions rather than shared prefixes, and the two compose well. We cover it in semantic caching for LLM applications.

Practical Recommendations

Start by establishing whether your workload qualifies. If the stable portion of your prompt is at least a few thousand tokens, repeats across requests, and requests for the same prefix arrive within the TTL more often than not, caching is almost certainly a win. If it fails any of the three tests, measure before you enable it.

Then work in this order. First, restructure the prompt so static content comes first and volatile content last. This costs nothing and is a precondition for everything else. Second, turn on the simplest caching mode the provider offers, which on Anthropic is the automatic top-level field, and log the usage fields on every response. Third, compute your break-even from the real multipliers and set an alert at that hit rate plus a safety margin. Fourth, only then tune lifetime and breakpoints, using your measured inter-request gaps.

For self-hosted fleets, add a cache-aware router so requests sharing a prefix land on the replica that holds it, size host DRAM and a shared L3 only if your measured working set exceeds VRAM, and set a salt policy before onboarding a second tenant.

Checklist

• [ ] Stable content first, volatile content last; no timestamps or IDs at the top.
• [ ] Prefix is byte-stable; JSON serialization is deterministic; retrieval order is sorted.
• [ ] Prefix exceeds the model’s minimum cacheable length with margin.
• [ ] Usage fields (read, write, uncached) are logged per request with a prefix version tag.
• [ ] Break-even hit rate computed from current multipliers; alert configured.
• [ ] TTL chosen from the measured inter-request gap distribution, not guesswork.
• [ ] Scheduled price changes (for example Gemini’s December 2026 and January 2027 changes) modelled in forecasts.
• [ ] Tenant isolation: workspace boundaries on hosted APIs, cache_salt and a cryptographic hash on self-hosted engines.
• [ ] Quarterly re-check of vendor pricing and lifetime pages.

Frequently Asked Questions

What is LLM prompt caching?

LLM prompt caching stores the attention key and value state computed for a shared prompt prefix so later requests with an identical prefix skip that computation. Hosted providers bill cache reads at a discount and often charge a premium to write. Self-hosted engines such as vLLM and SGLang do the same inside GPU memory, trading memory for lower prefill latency and compute cost.

How much does prompt caching save?

It depends on hit rate and the vendor’s multipliers. On Anthropic’s Sonnet 5.5 rates (1.25x write, 0.05x read), an 8,000-token prefix with a 90% hit rate cut a total per-request cost by about 64% in the worked example above. At a 20% hit rate the saving is roughly zero. The workload assumptions are illustrative, so rerun the math with your own token counts.

How long does a prompt cache last?

It varies by provider. Anthropic’s default is 5 minutes, refreshed free on each use, with a 1-hour option. OpenAI’s newer models offer 30 minutes by default or 12 hours; earlier models use in-memory or 24-hour retention. Google’s caching page did not state a default lifetime in my check. Because lifetime tiers cost more to write, choose by your measured inter-request gaps.

Why is my cache hit rate zero?

The usual causes are a prefix below the model’s minimum length, a changing token near the start such as a timestamp or request ID, non-deterministic tool or JSON serialization, a changed tool definition, or traffic gaps longer than the TTL. Check the usage fields: if both cache read and cache creation tokens are zero, the prompt was not cached at all, which points to the minimum length.

Is prompt caching the same as semantic caching?

No. Prompt caching reuses internal model state for an exact shared prefix, so the model still runs and produces a fresh answer. Semantic caching stores final responses and returns one when a new query is similar enough to an earlier one, skipping the model call entirely. They address different layers and combine well, but semantic caching risks returning a stale or slightly wrong answer.

Does prompt caching make data less private?

Hosted providers isolate caches by workspace or organization and never share them across organizations, according to Anthropic’s documentation. The residual risk is timing inference within a shared scope. On self-hosted engines, multi-tenant deployments should use per-request salts and a cryptographic hash. Cached KV state is derived from your prompt, so apply the same data-handling controls you would to the prompt itself.

Further Reading

• KV cache optimization for LLM inference: the memory and paging mechanics behind prefix reuse.
• Semantic caching for LLM applications: response-level reuse that complements prefix caching.
• NVIDIA Dynamo, LMCache and disaggregated KV cache offload: tiered and shared KV storage at fleet scale.
• Anthropic prompt caching documentation
• OpenAI prompt caching guide
• vLLM prefix caching design

By Riju — about