LLM Semantic Caching Architecture: Cut Inference Cost and Latency (2026)

Two users ask your support bot “how do I reset my password?” and “what’s the process to change my login credentials?” A string cache treats these as different requests and pays for two full generations. LLM semantic caching collapses them into one: it matches requests by meaning, not by exact bytes, so the second query returns a stored answer for a fraction of a cent instead of a full model call. That difference compounds fast at production traffic. But semantic matching also introduces a danger a string cache never has — a false cache hit, where two prompts look similar in embedding space but demand different answers, and the cache confidently serves the wrong one.

This post is a reference architecture for building that system safely. It is written for engineers who already run an LLM feature and now watch the bill climb linearly with traffic.

What this covers: the two-tier exact-plus-semantic design, similarity threshold tuning and the precision/recall trade-off, embedding model choice, invalidation and eviction, where this sits relative to a gateway and provider prompt caching, and a worked cost model you can adapt.

Context and Background

Caching is the oldest performance trick in computing, but LLMs break its core assumption. Traditional caches key on an exact identifier — a URL, a SQL string, a hash. Natural-language prompts almost never repeat verbatim, so a naive exact-match cache on LLM traffic returns hit rates near zero. The information is repetitive; the surface form is not.

Semantic caching closes that gap by embedding each prompt into a vector and treating “cache lookup” as a nearest-neighbour search in that vector space. If an incoming prompt lands close enough to a previously answered one, you return the stored response. The canonical open-source implementation is GPTCache, released by Zilliz in 2023, which wraps an embedder, a vector store, and a similarity evaluator behind an OpenAI-compatible interface. Its own paper reports the cache serving repeated query patterns at a fraction of the latency of a live call (GPTCache, NLPOSS 2023).

The economics changed the conversation. As LLM features moved from demo to production, teams discovered that a large share of traffic is near-duplicate: FAQ variants, retried prompts, templated agent steps, and popular questions asked a thousand different ways. Redis, Portkey, TrueFoundry, and other vendors now ship semantic cache layers, and independent write-ups report bill reductions in the 60–75% range on suitable workloads — VentureBeat, for example, documented a 73% cut on a support workload. Those numbers are real but workload-specific; your mileage depends entirely on how repetitive your traffic actually is, which is the first thing you should measure. For the broader cost picture beyond caching, see our guide to AI inference cost optimization.

It helps to place semantic caching in the wider caching taxonomy, because the three techniques are routinely confused. KV caching lives inside a single generation and reuses attention state across decode steps. Prompt (prefix) caching lives across calls and reuses the model’s computed state for a shared prompt prefix. Semantic caching lives above the model entirely and reuses a whole response. They operate at different layers, carry different risk profiles, and — importantly — stack. A well-run stack can use all three at once, and none of them substitutes for the others.

The GPTCache architecture is worth understanding because it named the components everyone now reuses. It decomposes the system into a pre-processor that normalizes the request, an embedding generator that turns the prompt into a vector, a cache manager that owns the exact store plus the vector store, a similarity evaluator that scores candidate matches, and a post-processor that decides what to return. That separation is the reason the pattern ports across stacks: you can swap the embedder, change the vector backend, or replace the evaluator without touching the request flow. Whether you adopt GPTCache directly or build your own, those five roles are the seams along which a maintainable semantic cache is cut.

Reference architecture

A production semantic cache is two caches, not one. The first tier is an exact-match cache keyed on a hash of the normalized prompt — microsecond lookups, zero false positives, no embedding cost. The second tier is the semantic cache: an embedding model plus a vector store that answers “is anything close enough?” You always try the cheap, safe tier first and only pay for embedding and vector search on a miss. Everything else is threshold tuning, key design, and invalidation hung off this spine.

Figure 1: Request flow through the two-tier cache. An exact-hash hit returns immediately; a miss triggers embedding, an approximate-nearest-neighbour search, and a threshold check before deciding to serve the cached response or call the model.

Figure 1 shows the full path. A prompt is normalized and hashed for the exact tier. On an exact miss, it is embedded and the vector is used for an approximate-nearest-neighbour (ANN) query against the store. The single best neighbour’s similarity score is compared to a threshold. Above it, the cached response is served; below it, the LLM generates, and the new (vector, response) pair is written back into both tiers so the next near-duplicate is cheap.

The exact tier earns its keep

Do not skip the hash tier because “semantic covers everything.” It does not. Exact matches are common in real traffic — retries, idempotent agent loops, button-driven prompts, and identical FAQ clicks — and serving them from a hash lookup costs nothing and cannot be wrong. Every request routed to the exact tier is a request you did not have to embed, which directly lowers the added latency and embedding spend the semantic tier imposes.

Normalize before hashing: lowercase, collapse whitespace, strip trailing punctuation, and canonicalize any templated fields. Aggressive normalization raises the exact-tier hit rate but must stay lossless with respect to meaning — never normalize away a negation or a number. The exact key must also fold in the same context dimensions the semantic key uses (model, system prompt, RAG context), or you will serve a right-text answer generated under the wrong conditions.

There is a subtle ordering point here. The exact tier is not merely an optimization on top of the semantic tier; it is a correctness backstop. Because a hash match is deterministic and exact, it can never produce a false positive, so any traffic it absorbs is traffic the risk-bearing semantic tier never touches. In workloads dominated by button-driven or programmatic prompts — agent loops that re-issue the same step, health checks, canned suggestions — the exact tier alone can carry a double-digit hit rate at zero risk and zero embedding spend. Treat the semantic tier as the layer that catches the linguistic variation the hash tier structurally cannot.

The semantic tier and the vector store

The semantic tier needs an embedding model and a vector index supporting ANN search. The index choice is a latency/recall trade-off: HNSW graphs give high recall at low latency but cost memory; IVF is lighter but tunes recall through probe counts. For cache-scale corpora — often tens of thousands to a few million entries — an in-memory HNSW index in Redis, Qdrant, Milvus, or pgvector answers in single-digit milliseconds. Because the cache is itself ephemeral, you can favour speed over durability; a lost cache entry is a cache miss, not data loss.

Critically, you retrieve only the top-1 neighbour for a cache decision, not the top-k you would fetch for RAG. You are asking “is there one answer good enough to reuse?” not “what context is relevant?” That keeps the search cheap and the decision crisp.

Two index parameters deserve attention because they set the recall/latency floor. For HNSW, ef_search controls how many candidates the graph traversal considers at query time — higher values find the true nearest neighbour more reliably but cost latency, and for a cache you want recall high enough that a genuine near-duplicate is not missed by the index itself. For IVF, the nprobe count plays the analogous role. A cache that misses a stored answer because of low index recall is silently leaking money: the entry exists but never gets served. Because cache corpora are modest and read-heavy, most teams can afford generous recall settings; the vector search is rarely the bottleneck next to the embedding round-trip.

Namespacing matters too. Multi-tenant systems must partition the vector space per tenant (or per key dimension) so tenant A’s answer can never surface for tenant B, even at high similarity. Most stores support this via a metadata filter or a separate collection; do it structurally rather than relying on the similarity score to keep tenants apart.

How the write-back path behaves under load

The store step in Figure 1 is easy to underrate. When a miss generates a fresh answer, you write the (vector, response) pair back so the next near-duplicate is cheap — but under bursty traffic, many identical-in-meaning prompts can miss simultaneously before any of them has written back, causing a thundering herd of redundant generations. A short-lived in-flight lock keyed on the normalized prompt lets the first miss generate while near-simultaneous duplicates wait for its result, converting a burst of full generations into one generation plus several cache hits. This single-flight pattern is the difference between a cache that smooths spikes and one that merely records them after the damage is done.

Write-back also needs a quality gate. Do not blindly cache every generation: skip responses the model flagged as low-confidence, responses that hit a content filter, and error fallbacks, or you will poison the cache with answers you would not want reused. Caching an apology or a refusal and then serving it to the next similar question is a classic self-inflicted regression.

Where the LLM gateway fits

The cache belongs behind a gateway, not bolted onto each service. A gateway already terminates auth, routing, rate limits, and logging for model traffic; the cache is one more middleware in that chain, so every service inherits it without its own integration. This placement also gives you one consistent place to measure hit rate, false-positive rate, and saved cost. If you are still designing that layer, our LLM gateway architecture guide covers the surrounding concerns the cache depends on.

Deeper walk-through

With the spine in place, the two decisions that make or break the system are the similarity threshold and the cache key. Figure 2 traces a real request through the components; Figure 3 shows how the key is composed and invalidated.

Figure 2: A cache-miss path. The gateway checks the exact cache, embeds on miss, runs an ANN nearest-neighbour query, and only calls the LLM when the best score falls below threshold — then writes the result back.

Before the threshold decision, note one embedder detail that affects it directly: dimensionality and normalization. Models like text-embedding-3-small support Matryoshka-style dimension truncation, letting you trade a little accuracy for a smaller, faster index — useful when the cache holds millions of entries. Whatever dimension you pick, L2-normalize vectors so cosine similarity and dot-product agree, and keep the embedder version pinned. Silently upgrading the embedding model reshapes the entire vector space, which means your carefully tuned threshold no longer means what it did and old entries become incomparable to new ones — an embedder change is a cache-invalidating event, not a drop-in swap.

The threshold is the single most consequential number in the system. It is a cosine similarity cutoff, and it directly sets your operating point on the precision/recall curve. Set it too high (say 0.98) and you reject obvious paraphrases, driving hit rate — and savings — toward zero. Set it too low (say 0.80) and you start serving answers to questions the user did not ask. Published guidance and experiments cluster the useful range around 0.85–0.95 for English question-answering, with one arXiv study reporting a positive-hit-rate above 97% at a tuned operating point while another team accepted a 0.8% false-positive rate as the cost of a high hit rate (Portkey, arXiv 2411.05276). There is no universal number; it depends on your embedding model and how semantically close your distinct questions are.

Embedding model choice and its cost

The embedder governs both quality and the added cost floor. A stronger model separates near-duplicates from distinct-but-similar questions more cleanly, letting you run a safer (higher) threshold without losing recall. But every miss pays for one embedding call. OpenAI’s text-embedding-3-small is a common default at $0.02 per million tokens with 1,536 dimensions, versus text-embedding-3-large at a higher rate for better separation (OpenAI). For a typical 200-token prompt, that is roughly $0.000004 per embedding with the small model — negligible next to a generation, which is exactly why the math works. Self-hosted models like the BGE or E5 families remove even that per-call fee at the cost of running inference yourself.

A worked cost model

Make the savings concrete. Assume a chatbot on a mid-tier model where an average request is 1,500 input + 500 output tokens, priced at $3.00/1M input and $15.00/1M output. Full generation cost per request:

(1,500 x $3 + 500 x $15) / 1,000,000 = $0.0045 + $0.0075 = $0.0120

Now add the cache. Embedding a 1,500-token prompt on text-embedding-3-small costs 1,500 x $0.02 / 1,000,000 = $0.00003. Every request pays that (on the miss path); every hit avoids the $0.0120 generation. At 1,000,000 requests/month and a 40% semantic hit rate:

Metric	No cache	40% hit rate	60% hit rate
Requests served from cache	0	400,000	600,000
Full generations	1,000,000	600,000	400,000
Generation cost	$12,000	$7,200	$4,800
Embedding cost (all reqs)	$0	$30	$30
Cache infra (est.)	$0	~$300	~$300
Total monthly	$12,000	~$7,530	~$5,130
Savings	—	~37%	~57%

The dominant term is hit_rate x generation_cost_saved; embedding and infra are rounding errors by comparison. That asymmetry is the whole reason semantic caching pays. Note the cache-infra figure is illustrative — it stands in for a small managed Redis or Qdrant instance and will vary with your provider and corpus size.

One number the naive model hides is the cost of a false positive. A wrong cached answer is not free just because it skipped a generation — it can cost a support escalation, a lost conversion, or a compliance incident, none of which appear in the token math. If you assign even a modest expected cost to each false hit and multiply by your false-positive rate, the “optimal” threshold shifts higher than pure hit-rate maximization would suggest. High-stakes routes should therefore run more conservative thresholds than the cost model alone implies; the token savings are real, but so is the tail risk they trade against.

Eviction: LRU, LFU, and cost-aware

A cache is a bounded resource, so eviction policy decides what survives under pressure. LRU (least-recently-used) is the sane default and matches the temporal locality of trending questions — what was asked recently tends to be asked again. LFU (least-frequently-used) protects a stable core of perennial FAQs from being evicted by a burst of one-off queries, at the cost of adapting more slowly to shifting topics. The LLM-specific refinement is cost-aware eviction: weight each entry by the generation cost it saves per hit, so an entry that stands in for an expensive long-output answer is kept preferentially over one that replaces a cheap short answer. On a fixed memory budget, cost-aware eviction extracts more dollar-savings per byte than either pure-recency or pure-frequency policies, because it optimizes the quantity you actually care about rather than a proxy.

Semantic vs prompt vs KV caching

The most common architecture mistake is treating these three as alternatives and picking one. They operate at different layers and address different waste, so the right design usually runs more than one. The table below fixes the distinction that the topic spec calls out — provider prompt caching is not the same thing as semantic caching, and conflating them leads to leaving savings on the table.

Dimension	KV caching	Prompt (prefix) caching	Semantic caching
Scope	Within one generation	Across calls, shared prefix	Across calls, whole response
What it reuses	Attention state per token	Computed prefix state	The final answer text
Who owns it	Model runtime	Provider or gateway	Your application
Avoids generation?	No	No	Yes
False-positive risk	None	None	Real — needs tuning
Typical saving	Faster decode	~90% off cached input tokens	Full call avoided on hit

Prompt caching, offered by Anthropic, OpenAI, and Google, cheapens the input side of calls you still make: a long shared system prompt or few-shot block is billed at roughly 10% of input rate on a cache read, with providers differing on write fees and TTL (PromptHub). It never skips the generation. Semantic caching skips the generation entirely but only when a stored answer is genuinely reusable. The two compose cleanly: cache the prefix so the calls you make are cheap, and cache the response so you make fewer of them. Our companion piece on prompt caching architecture and economics goes deep on the provider-side mechanics this section only summarizes.

The latency argument

Cost is half the story; latency is the other half. A cache hit replaces a multi-second generation with an embedding call plus an ANN lookup — realistically tens of milliseconds for the embedding round-trip plus single-digit milliseconds for the vector search. Against a generation that streams over 2–10 seconds, a hit is a 50–100x latency improvement for that request. The added latency on a miss is just the embedding call, since vector search runs in parallel with nothing blocking — you pay a small fixed tax on misses to win big on hits. Whether that trade is positive depends on hit rate: below roughly 10–15% hits, the miss-path embedding tax can outweigh the wins, which is why measuring repetitiveness first matters.

Trade-offs, Gotchas, and What Goes Wrong

The central failure mode of semantic caching is the false cache hit: two prompts sit above the similarity threshold but require different answers, so the cache serves a confidently wrong response. “What’s the refund policy for orders over $50?” and “What’s the refund policy for orders under $50?” are cosine-close and semantically opposite. Negations, numbers, entities, and units are exactly where embeddings under-separate and where a false hit does the most damage.

Figure 4: A borderline match at 0.86 clears a 0.85 threshold but may still be the wrong answer. A re-rank or lightweight LLM-judge gate on borderline scores catches false positives before they reach the user.

Mitigations exist and stack. Raise the threshold for high-stakes routes; add a cheap re-ranker or a small LLM-judge to verify borderline matches before serving; and never semantically cache answers that depend on volatile or personal data — prices, inventory, account state, medical or financial specifics — where a stale-but-plausible answer is worse than a slow one. Reported false-positive rates in poorly configured systems reach alarming levels, which is why a naive “just add a cache” rollout is dangerous.

Staleness is the second trap. A cache entry generated last week may reflect a since-changed source document or a since-updated system prompt. Plain TTL expiry helps but is blunt; pair it with versioned keys — fold the model version, system-prompt hash, and RAG-context (document) hash into the cache key so a change invalidates the affected entries automatically. Figure 3 shows this composite key.

Multi-turn conversations are the third: semantic caching assumes self-contained queries, but a follow-up like “and the second one?” is meaningless without history, so cache at the resolved-query level or skip caching for context-dependent turns.

Cold start is the fourth, quieter trap. A fresh cache has zero hit rate by definition, so early traffic pays full price plus the embedding tax, and the savings only materialize as the corpus fills. Teams that judge the system in its first hours conclude it “doesn’t work”; the honest evaluation window is after the cache has warmed against representative traffic. A related mistake is caching non-deterministic answers generated at high temperature — if the same question legitimately warrants varied responses (creative generation, brainstorming), reusing one frozen answer degrades the product even when it is technically “correct.” Reserve semantic caching for factual, convergent queries where one good answer is the goal.

Finally, observability is not optional here. Without per-route dashboards for hit rate, false-positive rate, saved cost, and added latency, a semantic cache silently drifts — a prompt change, a topic shift, or an embedder upgrade can quietly collapse hit rate or spike false positives, and you will not notice until users do. Instrument it as a first-class production system, not a fire-and-forget optimization.

Practical Recommendations

Start by measuring, not building. Log a week of prompts, cluster them, and estimate the achievable hit rate before you commit — if traffic is genuinely diverse, semantic caching may not earn its complexity. If repetition is high, build the exact tier first; it delivers safe wins immediately and tells you how much pure-duplicate traffic exists.

Figure 3: The cache key folds prompt, model version, system-prompt hash, RAG-context hash, and tenant into one namespace so that a TTL expiry, a document version bump, or a prompt version bump cleanly invalidates the right entries.

Tune the threshold against a labelled set of paraphrase pairs and known-distinct pairs, not by feel. Track false-positive rate as a first-class metric alongside hit rate — a high hit rate with a high false-positive rate is a liability, not a win. For eviction, LRU is a sane default; cost-aware eviction (favour keeping entries that saved the most expensive generations) squeezes more value from a fixed cache size. Choose your store deliberately; our Valkey vs Redis vs Dragonfly caching ADR covers the engine trade-offs for the backing layer.

Checklist before you ship:

• [ ] Exact-hash tier in front of the semantic tier.
• [ ] Composite key: prompt + model version + system-prompt hash + RAG-context hash + tenant.
• [ ] Threshold tuned on labelled paraphrase/distinct pairs, not guessed.
• [ ] False-positive rate monitored, with a re-rank/judge gate on borderline scores.
• [ ] TTL plus version-based invalidation wired to source documents and prompts.
• [ ] No caching of volatile, personal, or high-stakes answers.
• [ ] Hit rate, saved cost, and added latency dashboarded per route.

Frequently Asked Questions

How is semantic caching different from provider prompt caching?

They solve different problems and compose well. Provider prompt caching (Anthropic, OpenAI, Google) caches the model’s internal KV state for a repeated prefix — a long system prompt or shared context — so the model skips recomputing it, billed at roughly 10% of input rate on a read (PromptHub). It still runs a full generation. Semantic caching skips the generation entirely by returning a stored response. Use prompt caching to cheapen the calls you do make, and semantic caching to avoid making them.

What similarity threshold should I use?

There is no universal value; it depends on your embedding model and how close your distinct questions sit in vector space. Practical operating points cluster around 0.85–0.95 cosine similarity for English Q&A. Tune it against a labelled set of paraphrase pairs (should hit) and distinct-but-related pairs (should miss), and pick the point that maximizes hit rate while keeping false positives under your route-specific tolerance.

Won’t the cache serve wrong answers?

It can, and this is the primary risk. A false cache hit happens when two prompts exceed the threshold but need different answers — negations and differing numbers are the classic traps. Mitigate with a conservative threshold, a re-rank or LLM-judge gate on borderline scores, and a hard rule against caching volatile or high-stakes content. Monitor false-positive rate continuously; do not treat hit rate alone as success.

How do I invalidate stale cache entries?

Combine two mechanisms. TTL expiry passively ages entries out and is the floor. On top, use versioned keys: fold model version, system-prompt hash, and source-document (RAG-context) hash into the cache key, so any change to those inputs makes old entries unreachable automatically. For RAG, document-hash invalidation flushes every entry derived from a document the moment it changes.

Which vector store should I use for the cache?

For cache-scale corpora, an in-memory HNSW index answers in single-digit milliseconds — Redis, Qdrant, Milvus, and pgvector are all reasonable. Because the cache is ephemeral, prioritize latency and operational simplicity over durability; a lost entry is just a miss. Match the store to what you already run so you are not adding a new system solely for the cache.

What hit rate makes semantic caching worth it?

Enough that hit_rate x generation_cost_saved clearly beats the embedding-plus-infra tax on every request. In practice, workloads with even 30–40% repetition see meaningful savings, because a saved generation is hundreds of times more expensive than an embedding. Below roughly 10–15% hits, the miss-path embedding latency and cost can outweigh the wins — which is why you measure repetitiveness before building.

Further Reading

• LLM gateway architecture — where the cache sits in the request path, alongside routing, auth, and rate limiting.
• LLM prompt caching architecture and economics — the provider KV-prefix caching that complements semantic caching.
• AI inference cost optimization — the full cost-reduction toolkit beyond caching.
• Valkey vs Redis vs Dragonfly caching ADR — choosing the backing store engine.
• GPTCache paper (NLPOSS 2023) — the reference open-source implementation.
• Redis: What is semantic caching — a vendor walkthrough of the pattern.

By Riju — about