Semantic Caching for LLM Applications: Embedding Similarity, Prompt Caching and Cost-Latency Trade-offs (2026)

Most teams that add a cache in front of a large language model (LLM) discover the same uncomfortable fact within a week: the cache works exactly as designed and still returns wrong answers. Semantic caching replaces the byte-for-byte key lookup of a classic cache with a nearest-neighbour search over embeddings, so “How do I reset my password?” and “I forgot my password, what now?” can share one stored answer. That is the entire appeal, and also the entire risk, because “close in embedding space” is not the same as “has the same correct answer.”

This matters now because token prices have fallen but request volumes, agent loops and context lengths have grown faster, so inference is still the largest line item in many AI products. Provider-side prompt caching has also matured, which changes where a semantic layer pays off and where it is pure liability.

You will leave with a decision framework: which of the four cache layers to deploy, how to compute your break-even hit rate, how to calibrate a similarity threshold with labelled data instead of folklore, and how to keep a shared cache from leaking data across tenants.

What this covers: the cache stack from exact match to KV reuse, the mechanics of embedding similarity, provider prompt caching rules, a working implementation, cost and latency models, failure modes, and a rollout checklist.

Context and Background

LLM calls are slow and expensive for structural reasons. Generation is autoregressive: each output token requires a forward pass that reads the model weights and the accumulated key-value (KV) state, so latency scales with output length, and providers bill input and output tokens separately, with output typically priced several times higher than input. A support bot answering the same dozen questions thousands of times a day is paying full price for near-identical work.

Caching is the oldest answer to repeated work, and LLM stacks now have caching at four distinct layers that are routinely confused:

1. Exact-match caching keys on a hash of the whole request. It is safe and nearly free but only hits on literally identical requests, which in chat products is a small fraction of traffic.
2. Semantic caching keys on an embedding of the request and returns a stored response when a new request is “close enough.” This is the layer this article is about.
3. Provider prompt caching (Anthropic, OpenAI, Google and others) lets the provider reuse the already-computed KV state of a repeated prefix of your prompt. You still get a freshly generated answer; you just pay less for, and wait less on, the unchanged front of the prompt.
4. Inference-engine KV and prefix caching (vLLM, SGLang, NVIDIA Dynamo with LMCache) does the same thing inside your own serving cluster. We covered the offload tiers in NVIDIA Dynamo and LMCache for disaggregated serving.

The open-source reference point for layer two is GPTCache, from Zilliz, which popularised the pattern: embed the query, search a vector store, apply a similarity evaluation, and return the stored answer on a hit. The project is described in an ACL 2023 NLP-OSS workshop paper, and its README lists support for vector stores such as Milvus, FAISS, Qdrant, Chroma, PGVector and Weaviate. Note its own caveat: the maintainers state that the project is under swift development with a changeable API, and that they no longer add support for new LLM APIs, steering users to the generic get/set interface. In 2026 many teams therefore treat it as a design template and build the thin layer themselves, or use a managed offering such as Redis LangCache or the semantic-cache feature of an AI gateway. Check each product’s current documentation, because this category changes quickly.

The distinction between layers three and two is the first thing to internalise. Prompt caching is lossless: the model still reads your full prompt and generates a new answer, so quality is unchanged. Semantic caching is lossy by construction: it can serve an answer that was generated for a different question. Every design decision below follows from that asymmetry. If you need background on how embeddings are produced and tuned for retrieval, see our piece on embedding model fine-tuning with contrastive learning and Matryoshka representations, since the quality of your similarity signal is the quality of your cache.

The Reference Architecture: Four Cache Layers and Where Each Belongs

A production LLM cache is a stack, not a feature: check an exact-match cache first, then a semantic cache, and only then call the model, where provider prompt caching and engine-level KV reuse reduce the cost of whatever you could not avoid. Each layer trades safety for hit rate, so order them from safest to riskiest.

Figure 1: The four cache layers. Layers 1 and 2 can skip the model entirely; layers 3 and 4 make a model call cheaper but never skip it.

The diagram shows the request path from top to bottom. A request is first hashed against the exact-match store. On a miss it is embedded and searched against the semantic store. Only after both miss does it reach the provider, where a repeated prefix may be billed at a discount, and below that the inference engine may reuse KV blocks. After generation, the answer is written back to the two answer caches so that the next similar request can skip the model.

Layer 1: exact match is your free baseline

Exact-match caching deserves more respect than it gets. Hash the canonical request (model, system prompt version, tool set, sampling parameters, and normalised user text) and store the response. The hit rate is low for free-form chat but surprisingly high for programmatic callers: batch jobs that retry, agents that re-ask the same sub-question, evaluation harnesses, and dashboards that fire the same summarisation prompt on every page load.

The important property is that an exact hit is provably the same question, so there is no correctness risk beyond staleness. It also costs microseconds rather than the tens of milliseconds an embedding call takes. Normalise whitespace and case, strip volatile fields such as request IDs and timestamps, and you often double the hit rate with no semantic machinery at all. Sampling parameters matter: caching a temperature 1.0 response and replaying it deterministically changes the product’s behaviour, which is acceptable for factual lookup and unacceptable for creative writing.

Layer 2: semantic caching and the embedding similarity contract

The semantic layer turns text into a vector with an embedding model, then asks the vector store for the nearest stored query. For L2-normalised vectors, cosine similarity equals the dot product, which is why most implementations normalise at write time and use inner-product indexes. The store returns a candidate and a score. If the score meets a threshold, the cached answer is served.

The contract you are signing is this: any two queries scoring above the threshold must be answerable by the same response. Embedding models are not trained for that contract. A general-purpose embedding model is optimised so that topically related text lands nearby, which is useful for retrieval and dangerous for caching. “Cancel my subscription” and “Don’t cancel my subscription” are about the same topic and score very high on many models, yet the correct responses are opposites. Negation, numbers, named entities, dates, units and polarity words are the classic failure cases, because they change the answer while barely moving the vector.

Three design responses follow. First, choose or fine-tune an embedding model on paraphrase and intent-equivalence data rather than topical relevance. Second, treat the score as a candidate generator and add a cheap verification step for the grey zone (covered in the deeper analysis). Third, never rely on a single global threshold: calibrate per intent class using labelled pairs from your own traffic.

Layer 3: provider prompt caching is lossless and should go first

Before building any semantic infrastructure, make sure you are exploiting provider prompt caching, because it carries no correctness risk and often delivers the larger saving for agentic and long-context workloads.

The providers differ in mechanics, so read the current documentation rather than trusting a summary, including this one. As documented at the time of writing, Anthropic lets you mark up to four explicit cache_control breakpoints, charges a premium to write a cache entry (1.25x the base input price for the default 5-minute lifetime, 2x for the 1-hour lifetime) and charges a fraction of the base input price to read it (0.1x for standard models, with some newer models listed at lower multipliers). Entries refresh on each use, and the lifetime is measured from the start of the request. Minimum cacheable prefix length varies by model, from 512 to 4,096 tokens on the current list, and shorter prompts are silently processed without caching.

OpenAI documents a similar prefix-matching model. Its guide describes cached input billed at a steep discount (stated as up to 95 percent off), configurable retention that differs by model generation, an optional prompt_cache_key for routing and accounting, and a requirement that the entire rendered prefix match. Both vendors warn that changes to tools, reasoning settings or anything before a breakpoint invalidate everything after it. Pricing and retention options have been revised repeatedly, so treat any specific multiplier above as a snapshot and confirm it on the vendor page before building a cost model.

Layer 4: engine-level KV reuse for self-hosted models

If you serve open-weight models, the equivalent of provider prompt caching is automatic prefix caching in your inference engine: vLLM hashes KV blocks, and SGLang’s RadixAttention keeps a radix tree of prefixes. These reduce time-to-first-token for shared system prompts and multi-turn chats on your own GPUs. They do not reduce the number of requests; they make each request cheaper to prefill. They compose cleanly with the layers above, and with KV compression techniques such as those in our analysis of multi-head latent attention versus GQA and MQA.

Deeper Analysis: Building and Calibrating a Semantic Cache

The mechanics are simple; the engineering is in scoping, thresholds and economics. This section walks through the request path, a minimal implementation, threshold calibration, and a cost-latency model you can adapt with your own numbers.

The request path in detail

Figure 2: A semantic cache lookup. The metadata filter on the vector search is what keeps tenants, models and prompt versions from contaminating each other.

The sequence has four steps worth defending. The scope step builds a metadata filter before any search: tenant or user group, model identifier, system prompt version, language, and the tool set. Two requests are only comparable if every one of those matches, so they belong in a filter rather than in the embedded text. The embedding step should receive normalised text, and ideally only the part of the prompt that defines intent (the last user turn plus a short summary of context), not a 20,000-token conversation. The search step asks for the top one or two candidates with the filter applied; filtered approximate nearest-neighbour search has its own recall pitfalls, which we examined in filtered vector search with pre-filter, post-filter and ACORN. The decision step compares the score to a threshold and either serves, verifies, or falls through to the model and writes back with a time-to-live (TTL).

A common design error is embedding the entire prompt. Long conversational context makes every pair of requests look similar (the shared history dominates the vector) or dissimilar (a different history prefix), neither of which reflects whether the new question is equivalent. Cache only at the granularity where the answer is context-independent: single-turn FAQ-style queries, stateless classification, extraction on a fixed document, or the first turn of a session. For multi-turn dialogue, rewrite the final turn into a standalone question first (the same query-rewriting step used in retrieval) and cache on that.

A minimal implementation

The code below is deliberately small. It shows the scoping, normalisation and threshold logic using an in-memory index; swap the store for pgvector, Qdrant, Redis or Milvus in production.

import hashlib, time
import numpy as np
from dataclasses import dataclass

@dataclass
class Entry:
    vec: np.ndarray
    answer: str
    scope: tuple          # (tenant, model, prompt_version, lang)
    expires_at: float

class SemanticCache:
    def __init__(self, embed, threshold=0.92, ttl_s=86_400):
        self.embed, self.threshold, self.ttl_s = embed, threshold, ttl_s
        self.exact: dict[str, Entry] = {}
        self.entries: list[Entry] = []

    @staticmethod
    def _norm(text: str) -> str:
        return " ".join(text.lower().split())

    def _key(self, scope, text) -> str:
        return hashlib.sha256((repr(scope) + self._norm(text)).encode()).hexdigest()

    def get(self, scope, text):
        now = time.time()
        hit = self.exact.get(self._key(scope, text))
        if hit and hit.expires_at > now:
            return hit.answer, 1.0, "exact"
        q = self.embed(self._norm(text))
        q = q / np.linalg.norm(q)
        best, best_score = None, -1.0
        for e in self.entries:                      # use an ANN index at scale
            if e.scope != scope or e.expires_at <= now:
                continue
            s = float(q @ e.vec)                    # cosine, vectors pre-normalised
            if s > best_score:
                best, best_score = e, s
        if best and best_score >= self.threshold:
            return best.answer, best_score, "semantic"
        return None, best_score, "miss"

    def put(self, scope, text, answer):
        v = self.embed(self._norm(text))
        v = v / np.linalg.norm(v)
        e = Entry(v, answer, scope, time.time() + self.ttl_s)
        self.exact[self._key(scope, text)] = e
        self.entries.append(e)

Three details matter in practice. The scope tuple is compared by equality, so a prompt-version bump automatically orphans old entries. The TTL is per entry, so volatile answers can use minutes while stable ones use days. And get returns the score and the reason, because you will need both in logs: you cannot tune a threshold you did not record.

Calibrating the threshold with labelled pairs

Published defaults for similarity thresholds circulate widely (values around 0.85 to 0.95 are common in tutorials), but they are meaningless without reference to a specific embedding model, because score distributions differ sharply between models. A cosine of 0.85 might be a near-paraphrase on one model and merely same-topic on another. The only defensible procedure is empirical:

1. Sample a few thousand real queries from production logs (with privacy controls applied).
2. Generate candidate pairs by embedding them and taking each query’s nearest neighbour, so the pairs are the ones your cache would actually encounter, including the hard near-misses.
3. Label each pair “equivalent” (the same stored answer is acceptable) or “not equivalent.” Use human reviewers for a few hundred, then a strong model as a judge for the rest, validated against the human subset. Our notes on LLM-as-a-judge bias and calibration apply directly, because judges have position and verbosity biases.
4. Sweep the threshold and compute precision (of served hits, what fraction were truly equivalent) and the hit rate at each point.
5. Pick the operating point from the precision requirement, not the hit rate. If a wrong answer is merely mildly annoying, 97 percent precision may be fine. If it touches billing, medical, legal or security answers, you may need 99.9 percent, and the achievable hit rate at that precision may be too small to justify the layer at all.

The precision-recall shape is nearly always the same: hit rate climbs gradually as you lower the threshold, then precision falls off a cliff. The right threshold sits just before the cliff, with a safety margin, and it differs by intent. “What are your opening hours” tolerates a loose threshold; “what is my refund status” should never be semantically cached at all because the answer depends on a user-specific record.

Figure 3: A three-band policy. The grey zone between the serve and call thresholds is where a cheap verifier earns its keep.

The grey-zone verifier

The three-band policy in Figure 3 acknowledges that a single cut-off forces a bad trade. Above the upper threshold you serve. Below the lower threshold you call the model. In between, a verifier decides. Options, in rising cost: a rule check that compares extracted numbers, dates, negations and entities in the two queries and rejects on any mismatch; a cross-encoder reranker or a pairwise-equivalence classifier, which sees both texts jointly and is far better at detecting polarity flips than a bi-encoder; or a small LLM prompted with “do these two questions require the same answer?”

The verifier costs real money and latency, so it only pays if it converts a meaningful share of grey-zone traffic into hits. A cross-encoder over short queries typically adds on the order of tens of milliseconds on CPU or single-digit milliseconds on GPU (illustrative; measure on your hardware), which is still two orders of magnitude cheaper than a long generation. A small-LLM verifier is more flexible but can approach the cost of simply answering, so reserve it for expensive downstream calls such as long-context or agentic requests.

The economics: a break-even hit rate

Let C be the average cost of an uncached model call, e the cost of the semantic lookup (embedding plus vector search, paid on every request that reaches this layer), and h the hit rate. Expected cost per request is e + (1 – h) x C, against C without the cache. The cache pays for itself only if h is greater than e / C. That ratio is the break-even hit rate.

A worked example with illustrative numbers (not benchmarks): suppose an average call costs 0.8 cents, and the lookup costs 0.01 cents including the embedding API call and a vector query. Break-even is 0.01 / 0.8, about 1.25 percent. At a 25 percent hit rate the expected cost falls to 0.01 + 0.75 x 0.8 = 0.61 cents, a 24 percent saving. The same arithmetic shows why the cache is worthless for cheap calls: if C is 0.05 cents, break-even is 20 percent and a 25 percent hit rate barely clears it.

Add the term most spreadsheets omit: the cost of false hits. If a fraction f of served hits are wrong and each wrong answer costs a (support escalation, refund, trust), the expected harm is h x f x a per request. The cache is net positive only when the saving, h x C, exceeds that harm plus lookup cost. With C at 0.8 cents and a at, say, 50 cents, even a 0.5 percent false-hit rate costs 0.25 cents per hit, roughly a third of the saving. This is why precision, not hit rate, is the number to optimise.

Latency follows the same structure. A hit costs the lookup time (embedding plus search, often tens of milliseconds, illustrative) instead of the model’s time-to-first-token and generation time (hundreds of milliseconds to many seconds). A miss pays the lookup and the full call, so p99 latency gets slightly worse on misses even as p50 improves dramatically. Run the embedding and the model call speculatively in parallel if p99 matters more than cost, and cancel the model call when a hit lands.

Prompt caching economics: when the prefix does the work

Prompt caching has its own break-even, and it is simpler. With a write premium w (for example 1.25x) and a read multiple r (for example 0.1x) relative to the base input price, a prefix of P tokens reused n times within its lifetime costs P x (w + (n – 1) x r) against P x n uncached. Using the Anthropic multipliers above, one write and one read costs 1.25 + 0.1 = 1.35 units against 2.0, so a prefix that is reused even once inside the 5-minute window is already cheaper. Two reads cost 1.45 against 3.0. For the 1-hour lifetime at 2x, one write plus one read costs 2.1 against 2.0, a slight loss, and it needs a second read to win. The arithmetic is why prompt caching suits agent loops and long fixed system prompts, and why a prefix reused only once per hour may not pay at the 5-minute tier.

Two practical rules fall out. Put stable content first and volatile content last: tool definitions, then system instructions, then reference documents, then the changing conversation. A timestamp or user name in the first paragraph destroys the hit rate for everything behind it. And keep the tool list stable: both major vendors document that changing tool definitions invalidates the cached prefix, so dynamically assembled tool menus are a common hidden cause of zero cache reads. Verify by reading the usage fields in the response (Anthropic reports cache_creation_input_tokens and cache_read_input_tokens) instead of assuming a hit.

Prompt caching also interacts with latency differently from semantic caching. It shortens prefill, so time-to-first-token falls for long prompts, but generation time is unchanged. OpenAI’s documentation explicitly cautions that cached lifetime does not guarantee every request reuses stored tensors or sees lower latency. A semantic hit, by contrast, skips generation entirely and returns the full answer at once, which is why it feels dramatically faster for long outputs.

Putting the layers together in a deployment

Figure 4: A deployable layout. Versioned entries and an audit loop make invalidation and false-hit monitoring routine operations rather than incidents.

The deployment in Figure 4 places the cache in a gateway or SDK wrapper so that every application inherits it. An in-process LRU absorbs repeated identical calls inside one worker; a shared Redis tier handles exact keys across workers with TTLs; and a vector index, partitioned per tenant, holds embeddings with pointers to an answer store that also records the metadata needed for invalidation. Releases of the underlying corpus or system prompt bump a version field that the scope filter uses, instantly orphaning stale entries. A delete path exists for privacy requests. Metrics feed back a hit rate by intent and a sampled audit of served hits.

Where you put this layer is a real architectural choice. A gateway-level cache sees all traffic and is language-agnostic, but it has the least context about what is safe to cache. An application-level cache knows which endpoints are stateless, but is duplicated per service. A common compromise is a gateway that implements the mechanism and receives per-route policy (threshold, TTL, allowed or denied) from the application teams. If you already run an LLM gateway for routing, read our analysis of LLM routing and model cascades, since a cache hit is the cheapest possible “model” in a cascade and should sit at the front of it.

Choosing a cache layer: a decision matrix

Scenario	Exact match	Semantic cache	Provider prompt cache	Engine prefix cache
FAQ and help-center bot, stateless	Weak hit rate	Strong fit, with a verifier	Moderate (shared system prompt)	Moderate
Agent loop with a 30k-token fixed context	Rare hits	Poor fit, queries are unique	Strong fit	Strong fit
Batch classification of repeated records	Strong fit	Useful for near-duplicates	Moderate	Moderate
Personalised or account-specific answers	Only with user in key	Unsafe unless user-scoped	Strong for static prefix	Moderate
Creative generation where variety is the product	Avoid	Avoid	Safe, prefix only	Safe
Regulated answers (medical, legal, financial)	Safe if keyed on version	Only with human-approved answers	Safe	Safe
Self-hosted open-weight serving	Safe	Possible	Not applicable	Strong fit

The pattern in the matrix is consistent: lossless layers (prefix caches) are safe nearly everywhere, and the lossy layer needs a stable, repetitive, context-independent query population to earn its place.

Choosing and operating the embedding model

The embedding model is the heart of the semantic layer, and three operational properties matter beyond benchmark rank. First, latency and cost per call: the embedding step happens on every request, including misses, so a hosted API adds a network round trip and a per-token fee, whereas a small local model (a few hundred million parameters or less, on CPU or a shared GPU) removes both at the price of operating it. Short queries are cheap to embed either way; the cost concern is high request volume rather than token count.

Second, dimension and index size. A 1,024-dimension float32 vector occupies 4 KB, so ten million cached entries need about 40 GB of raw vectors before index overhead. Matryoshka-style models let you truncate vectors to a smaller prefix with modest quality loss, and scalar or product quantization shrinks them further; the trade-offs are explained in our guide on HNSW, DiskANN and IVF-PQ vector indexes. For a cache, recall requirements are looser than for retrieval, since a missed hit only costs a model call, so aggressive compression is usually acceptable.

Third, version stability. If you change embedding models you must re-embed or discard every cached vector, because scores from different models are not comparable. Pin the embedding model identifier into the scope metadata, exactly as you pin the prompt version, so a model swap automatically starts a clean cache rather than silently mixing score distributions.

Trade-offs, Gotchas, and What Goes Wrong

Semantic caches fail in ways that are quiet. Nothing crashes; answers are just subtly wrong, and the dashboard shows a healthy hit rate. These are the failure modes that recur.

Cross-tenant leakage. A shared vector index with no scope filter will happily serve one customer’s cached answer, containing their data, to another customer who asked a similar question. This is the most serious failure because it is a confidentiality incident, not a quality bug. Partition by tenant (separate index, collection or mandatory filter), and never cache responses that were generated from user-specific retrieval or tool output unless the user identity is part of the key. Treat the cache like any other data store holding customer content: encryption at rest, retention limits, and a deletion path for privacy requests.

Cache poisoning. Whatever is written is later served to others. If an attacker can get a manipulated or prompt-injected answer stored, and then phrase a victim-style query that lands above threshold, the poisoned answer is distributed at scale. Mitigate by only caching responses that passed output guardrails, by caching only for authenticated and rate-limited callers, and by keeping a short TTL on anything generated from untrusted retrieved documents. Our overview of LLM red teaming and jailbreak defense covers the guardrail side.

Staleness and silent drift. A cached answer to “what is the current pricing” is correct until the price changes and wrong afterwards, with no signal. Tie entries to the version of the data they depend on, set TTLs by volatility, and add explicit invalidation hooks on content releases. Model upgrades are the other trigger: an answer produced by an older model version may no longer match your current style or safety behaviour, so the model identifier belongs in the scope.

Embedding blind spots. Negation, quantities, units, dates, and entity swaps (“flights from Paris to Rome” versus “from Rome to Paris”) barely move a bi-encoder vector. The rule-based extractor in the grey-zone verifier catches many of them cheaply. Also expect language effects: multilingual models can place a query and its translation very close, which is a hit if you want it and a bug if the response language must match the question.

The thundering herd at expiry. When a popular entry expires, many concurrent requests miss simultaneously and all call the model. The classic remedies apply unchanged: request coalescing, jittered TTLs and serving stale while revalidating. See our guide to cache stampedes and request coalescing.

Caching non-determinism. Replaying one stored answer to everyone turns a stochastic system into a deterministic one. For factual or support content this is usually a feature (consistency), but it removes variation that some products rely on, and it hides regressions: if the cache serves the old answer, an improved prompt appears to have no effect in A/B tests. Bypass the cache for evaluation traffic, or version it per experiment arm.

Measurement traps. A rising hit rate is not good news by itself. Lowering the threshold raises it and silently lowers precision. Track hit rate together with sampled precision from the audit loop, plus user-facing signals such as thumbs-down or re-ask rates on cached versus fresh answers. A cached answer that users immediately re-ask is a false hit you can detect without labelling.

Premature infrastructure. Standing up a vector database, an embedding service and an audit pipeline is a real operating burden. If your traffic is under a few thousand requests per day, or your calls are cheap, the break-even arithmetic above will often say no. Start with exact match plus provider prompt caching, measure the near-duplicate rate in logs, and add the semantic layer only when the measurement justifies it.

Practical Recommendations

Sequence the work by risk and by return. Lossless layers first, lossy layer last, and never skip the measurement step between them.

Start by instrumenting what you have. Log, per request, the prompt length, the cached-token counts the provider reports, the model, and a hash of the first N tokens. Within a week you will know how much of your input is a stable prefix and how many requests are exact or near duplicates. That data decides everything else.

Then restructure prompts for prefix stability: static instructions and tool definitions first, documents next, per-request content last, with no timestamps or identifiers near the top. Add explicit cache breakpoints where your provider supports them, and confirm hits through the usage fields. This often delivers most of the achievable saving for agentic and long-context applications with zero correctness risk.

Only then consider semantic caching, and scope it narrowly: a stateless, repetitive endpoint, a calibrated threshold from labelled pairs, a verifier for the grey zone, tenant-partitioned storage, and a kill switch per route. Roll it out in shadow mode first, where the cache computes what it would have served while the model still answers, and compare the two offline. Promote to live serving only after shadow precision meets your target.

Checklist before enabling semantic caching in production:

• [ ] Prefix caching verified working (non-zero cache-read tokens in usage data).
• [ ] Exact-match layer in front, with normalised keys and parameters included.
• [ ] Embedding model and prompt version pinned into the cache scope.
• [ ] Tenant or user isolation enforced by index partition or mandatory filter.
• [ ] Threshold chosen from a labelled sweep for each intent class, with a precision target.
• [ ] Grey-zone verifier in place for numbers, negation and entities.
• [ ] Denylist for personalised, regulated and time-sensitive intents.
• [ ] TTLs set by data volatility; invalidation hooks on content and model releases.
• [ ] Shadow-mode comparison completed and a sampled audit running after launch.
• [ ] Alerts on hit rate by intent, audit precision, and re-ask rate for cached answers.

Frequently Asked Questions

What is semantic caching in an LLM application?

Semantic caching stores previous prompts as embedding vectors alongside their model responses. When a new request arrives, the system embeds it, searches for the most similar stored prompt, and returns the saved response if the similarity score exceeds a threshold. This avoids a model call for paraphrased repeats. Unlike exact-match caching, it can hit on differently worded questions, but because it can return an answer generated for a different question, it needs calibrated thresholds and safeguards.

How is semantic caching different from prompt caching?

Prompt caching, offered by model providers, reuses the computed internal state of a repeated prompt prefix, so you pay less and wait less on the unchanged portion, but the model still generates a fresh answer. Semantic caching is a layer you build or buy that skips the model call altogether by returning a stored response. Prompt caching is lossless; semantic caching is lossy. Most production systems should use both, starting with prompt caching.

What similarity threshold should I use for semantic caching?

There is no universal value, because score distributions depend on the embedding model and your query mix. Tutorials often quote cosine thresholds between roughly 0.85 and 0.95, but treat those as starting points only. Label a few thousand nearest-neighbour pairs from your own traffic, sweep the threshold, and choose the point that meets your required precision. Use stricter thresholds, or no caching, for high-stakes or user-specific intents.

Is GPTCache still a good choice in 2026?

GPTCache remains a useful reference implementation: it demonstrates the embed, search, evaluate and return pipeline and supports many vector stores under an MIT license. However, its README says the API may change and that maintainers no longer add support for new LLM APIs, directing users to its get and set interface. Many teams use it for prototyping or build a thin custom layer, or use a managed gateway feature. Review repository activity before depending on it.

Does semantic caching reduce latency as well as cost?

Yes on a hit, no on a miss. A hit replaces model time-to-first-token and full generation with an embedding call and a vector search, typically tens of milliseconds, so long answers return dramatically faster. A miss pays the lookup and then the normal model call, so tail latency worsens slightly. If p99 matters, run the lookup and model call in parallel and cancel the model call on a hit.

Can a shared semantic cache leak private data?

It can, if it is not scoped. A global index may return one user’s cached answer, which contains their information, to another user with a similar question. Prevent this by partitioning the cache per tenant or enforcing mandatory metadata filters, by excluding responses built from user-specific retrieval or tools, and by applying retention limits and deletion on request. Treat the cache as a data store subject to your privacy controls.

Further Reading

• NVIDIA Dynamo and LMCache: disaggregated LLM serving and KV cache offload for the engine-level layer beneath prompt caching.
• Embedding model fine-tuning with contrastive hard negatives and Matryoshka for improving the similarity signal your cache depends on.
• LLM routing and model cascades for placing the cache as the first tier of a cost-quality cascade.
• RAG latency budget optimization for the retrieval-side latency budget that interacts with cache lookups.
• Anthropic prompt caching documentation and OpenAI prompt caching guide for current multipliers, minimum lengths and retention options.
• GPTCache on GitHub and the GPTCache paper in the ACL Anthology.

By Riju — about