RAG Latency: Optimizing Retrieval, Reranking and Streaming for Production (2026)

Most retrieval-augmented generation demos feel fast because they run against ten documents, one user and a warm laptop. Production feels slow because a single answer now crosses a query rewriter, an embedding model, an approximate nearest neighbour index, a lexical index, a reranker, a prompt assembler and a large language model, and each hop has its own tail. The user does not experience the average of those hops. They experience the sum of the slow ones, on the one request in twenty that hits every slow path at once.

RAG latency is therefore a budgeting problem before it is an optimization problem. If you cannot say how many milliseconds each stage is allowed to consume at p50, p95 and p99, you cannot tell whether a change helped, and you will tune the stage that is easiest to measure instead of the one that hurts. This guide decomposes the pipeline, shows where the milliseconds actually go, and walks through the levers that matter: index parameters, hybrid search, reranker design, parallelism, speculative retrieval, caching tiers, prompt compression and streaming.

What this covers: a stage-by-stage latency budget, ANN recall versus latency trade-offs, cross-encoder versus late-interaction reranking, parallel and speculative execution, four caching tiers, streaming UX, an instrumented Python pipeline, a worked (illustrative) budget table, and the failure modes that wreck tail latency.

Context and Background

A basic RAG request used to be two calls: embed the question, query a vector database, paste the results into a prompt. The 2026 production shape is longer. Teams add a query rewriting step to resolve pronouns in chat history, run hybrid retrieval that merges dense and lexical results, apply a reranker to clean up the candidate list, and often add guardrails before and after generation. Every addition improves answer quality on some slice of traffic and adds a stage that can stall.

The latency literature for interactive systems has been stable for decades. Jakob Nielsen’s widely cited response-time thresholds put roughly 100 ms as the limit for feeling instantaneous, about one second as the limit for keeping a user’s flow of thought, and about ten seconds as the limit for holding attention at all. A RAG answer will never be instantaneous because decoding a few hundred tokens takes seconds, so the design target shifts from total completion time to time to first token (TTFT) and to the perceived progress that follows. Streaming turns a six-second wait into a one-second wait followed by reading speed, which is a very different product.

Two components dominate most budgets. The first is the language model, whose cost splits into prefill (processing the prompt, which grows with retrieved context) and decode (generating tokens one at a time). The second is retrieval and reranking, which together can range from tens of milliseconds to over a second depending on index type, candidate count and whether the reranker is a hosted API. Embedding calls to a hosted provider typically add a network round trip of tens to a few hundred milliseconds, and are easy to forget because they hide inside an SDK call.

Storage choices matter more than benchmark tables suggest. If your vectors live in Postgres, the index implementation and memory fit decide whether vector search latency is single-digit or hundreds of milliseconds; our comparison of pgvectorscale and pgvector with StreamingDiskANN covers that layer in detail. If you are choosing between Postgres and a dedicated engine, the pgvector versus Qdrant ADR on hybrid search and reranking sets out the decision criteria. This article assumes the store is chosen and asks how to spend the milliseconds around it.

For deeper background on how percentile latency behaves in fan-out systems, Jeffrey Dean and Luiz André Barroso’s paper “The Tail at Scale” (Communications of the ACM, 2013) remains the canonical reference, and it applies directly: a RAG request fans out to several services, and the slowest of them sets your latency.

The Latency Budget: Where the Milliseconds Go

A RAG latency budget is a table that assigns every pipeline stage a target at p50 and p95, and that sums to a user-facing goal, typically a time to first token under one to two seconds at p95. You build it by measuring each stage separately, then deciding which stages can run in parallel, which can be cached and which can be shed under load.

The pipeline as a critical path

The request path in its naive form is a chain. Each stage waits for the previous one, so latencies add. Figure 1 shows the stages and where the lexical branch can run beside the dense branch.

Figure 1: The RAG pipeline as a critical path. Only the dense and lexical retrieval branches are naturally parallel; everything else waits on the stage before it.

The long description for readers who cannot see the figure: a user query enters a small-model query rewriter, which feeds both an embedding step (leading to ANN vector search) and a BM25 lexical search; the two result lists merge in a reciprocal rank fusion step, a reranker rescoring the top fifty candidates keeps five, the prompt builder assembles them, the LLM prefills and then decodes, and tokens stream to the client.

Two properties of this graph drive everything else. First, the critical path runs through the rewriter, the embedder, the ANN search, the reranker and the LLM prefill, so a saving anywhere else is invisible. Second, TTFT is the sum of everything up to and including the first decoded token, while the remaining decode time is hidden behind streaming. That is why stages before the LLM are budgeted tightly, while decode speed is budgeted against reading speed.

Decomposing the budget

The following table is a worked example. All numbers are illustrative, chosen to be plausible for a mid-sized deployment using a hosted embedding API, an in-memory HNSW index of a few million chunks, a hosted cross-encoder reranker and a hosted LLM. Replace them with your own traces.

Stage	p50 (ms)	p95 (ms)	Notes
Auth, routing, request parsing	5	15	Rarely the problem, always measured
Query rewrite (small LLM, 30 output tokens)	250	600	Often the largest pre-retrieval cost
Query embedding (hosted API)	60	180	Network dominated
ANN search, top 50	12	40	Depends on ef_search and filters
BM25 search, top 50 (parallel with ANN)	10	35	Hidden behind ANN when parallel
Fusion (RRF)	1	3	Negligible
Rerank 50 candidates (cross-encoder)	150	450	Scales with candidate count and chunk length
Prompt build, tokenization	5	12	CPU, in process
LLM prefill, about 3,000 tokens	350	900	Scales with context length and load
First token to client	10	30	Network and buffering
TTFT (sequential, naive)	~850	~2,200	Sum of the above, minus the parallel BM25

Notice what the table says. The rewrite, rerank and prefill stages together account for most of the median time, and the embedding call costs more than the vector search it feeds. A team that spends a quarter shaving the ANN search from 12 ms to 8 ms has improved the p50 by half a percent.

Percentiles, not averages

A mean hides the problem. Suppose each of six sequential stages independently exceeds its p95 on 5% of requests. The probability that at least one stage is slow on a given request is 1 – 0.95^6, about 26%. So roughly a quarter of requests contain a stage in its slow tail, even though every stage looks healthy in isolation. This is the arithmetic behind the tail-at-scale argument, and it is why you set budgets at p95 and p99 per stage and then check the end-to-end percentile, which will be worse than any single stage suggests.

Tail causes in RAG are mostly mundane. Cold caches after a deploy, garbage collection pauses, noisy neighbours on shared vector nodes, reranker batch queues that fill during bursts, and LLM provider rate limiting that returns after a retry. Each has a different fix, and none shows up in a median.

Budgeting rules that hold up

Three rules keep budgets honest. Allocate against p95, not p50, because user complaints track the tail. Give every external call a timeout and a fallback; a stage without a deadline has an unbounded budget. And reserve headroom: if the goal is a 1,500 ms p95 TTFT, plan the stage targets to sum to about 1,100 ms so that retries and load spikes do not immediately breach the goal.

A useful discipline is to attach a deadline to the request at the edge and pass the remaining time down the call chain. A reranker that receives 120 ms of remaining budget can truncate its candidate list; one that receives 20 ms can be skipped. Without deadline propagation, every stage assumes it has all the time in the world.

Retrieval Tuning: ANN Recall Versus Latency

Vector search latency is controlled by a small number of parameters, and each one trades recall for time. Understanding the shape of that trade is more useful than memorising any benchmark, because your corpus, dimensionality and filters will differ from the benchmark’s.

HNSW and the ef_search dial

Hierarchical Navigable Small World (HNSW) graphs, introduced by Malkov and Yashunin, search by greedy descent through layers of a proximity graph and then expand a candidate beam at the bottom layer. The beam width at query time is usually called ef_search (pgvector exposes it as hnsw.ef_search, with a default of 40 in its documentation; Qdrant exposes hnsw_ef as a search parameter). A larger beam visits more graph nodes, which raises recall and raises latency, roughly in proportion to the number of distance computations.

The practical consequence is that recall is concave in ef: the first increments buy a great deal, and the last few points of recall cost several times the latency. A typical curve on a well-built index might move recall@10 from the high eighties to the mid nineties while latency doubles, then take another doubling to reach the high nineties. Treat that as a shape, not a measurement. Measure your own curve with a held-out set of queries and exact brute-force ground truth, and pick the knee.

The build parameters matter too. M (graph degree) and ef_construction set the quality ceiling of the graph and the memory footprint; you cannot buy back a poorly built graph with a large ef_search. Graph memory grows with M, and the full-precision vectors dominate RAM, so a corpus that no longer fits in memory falls off a cliff as the index starts paging. That cliff, not the beam width, is the most common reason vector search latency jumps by an order of magnitude in production.

Filters change the geometry

Metadata filters are the quiet killer. Post-filtering (search, then discard non-matching hits) can return fewer than k results when the filter is selective, and the typical patch is to over-fetch, which costs latency. Pre-filtering inside the graph traversal keeps results full but can disconnect the reachable graph when the filter is very selective. Engines handle this differently: some switch to brute-force over the filtered subset when it is small, and pgvector added iterative index scans in version 0.8.0 to keep scanning until enough filtered rows are found. Whichever engine you use, test with the filters your tenants actually apply, at their real selectivity, because a filter that matches one percent of the corpus behaves nothing like an unfiltered query.

Disk-based indexes and quantization

When the index no longer fits in memory, there are two families of answer. Graph indexes designed for SSD, such as Microsoft’s DiskANN (Subramanya et al., NeurIPS 2019), keep a compressed representation in RAM for navigation and fetch full vectors from disk for the final distance computation, trading a few extra disk reads per query for a corpus that is many times larger than memory. Scalar and product quantization shrink vectors so more of them stay hot. Both reduce the memory bill and add a rescoring step, so they interact with latency in two ways: the compressed distance is cheaper, but the rescoring reads add a tail. Benchmark at your concurrency, not single-query.

Dimensionality is the other lever. Embeddings trained with Matryoshka representation learning let you truncate vectors, search a short prefix cheaply and rescore a few candidates at full length. The pattern is covered in our guide to Matryoshka embeddings and adaptive retrieval; the latency gain comes from fewer bytes per distance computation and a smaller working set, with a rescoring step to recover accuracy.

Hybrid search: lexical and dense in parallel

Dense retrieval misses exact identifiers, rare terms and product codes; BM25 misses paraphrase. Hybrid search runs both and merges. The common merge is reciprocal rank fusion (RRF), from Cormack, Clarke and Büttcher (SIGIR 2009), which scores each document as the sum over result lists of 1 / (k + rank) with k conventionally 60. RRF needs no score calibration between the two engines, which is why it is the default in most stacks.

For latency, the point is that the two branches are independent. Issue them concurrently and the cost is the slower branch plus fusion, not the sum. If both live in the same engine (Postgres with a full-text index and pgvector, or a single Qdrant query), you may be able to do the whole thing in one round trip, which also saves a network hop. If they live in different systems, put both calls on the async event loop and apply a hard deadline to the slower one, falling back to the other branch’s results when it times out. Returning dense-only results at 90% of the quality is better than returning nothing at all.

Reranking: The Largest Controllable Stage

A reranker rescoring the top candidates is usually the single stage where a team can trade a measurable amount of latency for a measurable amount of answer quality, and the one most often left at defaults. Reranker latency scales with candidate count times chunk length, so the two cheapest optimizations are to rerank fewer candidates and to feed it shorter chunks.

Cross-encoders: accurate, expensive

A cross-encoder concatenates the query and one candidate passage, runs the pair through a transformer, and outputs a relevance score. Because query and document attend to each other, accuracy is high, but nothing can be precomputed: scoring fifty candidates means fifty full forward passes over query-plus-passage token sequences. A passage of 300 tokens plus a 20-token query is a 320-token sequence; fifty of them is 16,000 tokens of encoder work per query.

On a GPU, batching makes this tolerable. A small cross-encoder (tens to a few hundred million parameters) can score fifty short pairs in a few tens of milliseconds on a modern accelerator, while the same work on CPU may take a few hundred. Hosted reranking APIs add network time and a queue, and their latency is typically less predictable than a self-hosted model on dedicated hardware, though they remove operational burden. Verify current figures from the provider you use rather than relying on published marketing numbers; they change with model versions.

Late interaction: precompute, then cheap scoring

ColBERT (Khattab and Zaharia, SIGIR 2020) takes a different approach. It encodes the query and each document separately into sets of token-level vectors, then scores with MaxSim: for each query token vector, take the maximum similarity over the document’s token vectors and sum those maxima. Document vectors are computed at index time, so query-time work is one query encoding plus vector operations.

The trade is storage and index complexity. A document of 200 tokens stores 200 vectors instead of one, which is why ColBERTv2 adds residual compression and PLAID adds a pruned candidate-generation scheme to keep memory and latency under control. Late interaction typically sits between bi-encoders and cross-encoders in accuracy and well below cross-encoders in query-time cost. It is attractive when you can afford the index and when you want to rerank hundreds of candidates cheaply, and less attractive when your corpus changes constantly and the multi-vector index is a burden to maintain.

Approach	Query-time cost	Precompute	Accuracy tendency	Best when
Bi-encoder only	Lowest	One vector per chunk	Baseline	Latency is the hard constraint
Cross-encoder rerank	High, linear in candidates	None	Highest per candidate	Small candidate sets, GPU available
Late interaction (ColBERT-style)	Low to moderate	Many vectors per chunk	Between the two	Large candidate sets, stable corpus
LLM-as-reranker	Highest	None	Variable	Offline or low-QPS use only

Controlling reranker cost

Four levers apply regardless of model choice. Cut the candidate count: rerank thirty instead of one hundred if your measured recall at thirty is within a point or two. Truncate chunks to the first few hundred tokens, since relevance is often evident early. Apply an early exit: if the top retrieval scores are far above the rest, skip the reranker. And set a deadline: if the rerank has not returned within its budget, serve the fused order. Measure the quality cost of each lever with a labelled evaluation set; a cheaper reranker that drops answer groundedness is not a latency win, it is a quality regression with a latency receipt.

Parallelism and Speculation

Once stages are individually tuned, the next gain comes from changing the dependency graph. Anything that does not depend on a previous result can start earlier, and anything likely to be needed can start before you are sure.

Run what is independent

The embedding call and the BM25 search both need only the query text. The access-control lookup, the user profile fetch and the cache check need only the request. Launching these together costs nothing but code structure. In an async framework, create the tasks at the top of the handler and await them at the join point rather than awaiting each in sequence.

Speculative retrieval

The query rewrite step is the awkward one: it is slow, and it gates retrieval. The speculative pattern starts retrieval on the raw user query immediately, while the rewrite runs in the background. When the rewrite returns, compare it with the raw query. If the rewrite is substantially the same, which in single-turn questions is common, use the speculative results and save the rewrite latency entirely. If it differs materially, issue a second retrieval with the rewritten query and merge or replace.

The cost is extra retrieval load on the fraction of requests where the rewrite matters. With cheap vector search and expensive rewriting, that is a good exchange. Figure 2 contrasts the sequential and speculative paths.

Figure 2: Sequential versus parallel and speculative execution. Speculation removes the rewrite from the critical path for requests where the rewrite does not change the query.

The long description: the top path chains rewrite, embed, ANN, rerank and prefill. The lower path embeds the raw query at once, runs ANN and BM25 in parallel, runs the rewrite in the background, optionally issues a second retrieval if the rewrite differs, reranks with an early cutoff, then prefills with a cached prefix.

A related trick is to skip the rewrite entirely for first-turn questions that contain no anaphora (pronouns or references to earlier turns), using a cheap heuristic or a tiny classifier. Rewriting is essential in multi-turn chat and wasteful in a fresh single-turn query. Gate it.

Speculative prefill

The LLM prefill is the next target. If the system prompt, tool definitions and stable instructions come first and the retrieved context comes last, a provider or serving engine with prefix caching can reuse the key-value cache for the stable part. Self-hosted engines such as vLLM support automatic prefix caching, and several hosted providers offer prompt caching with discounted pricing and lower latency for cached prefixes; the exact discount, minimum prefix length and time-to-live differ by provider and change, so check current documentation. Structure the prompt so that the bytes that rarely change come first and the per-request retrieved text comes last, because a single changed token invalidates everything after it.

Caching Tiers

Caches are the only optimization that can remove a stage entirely. A RAG system has four natural cache points, each with a different hit rate, a different correctness risk and a different cost. Figure 3 shows how they stack on the request path.

Figure 3: Four caching tiers. Each tier that hits skips the stages beneath it; the prefix cache never skips a stage but shortens prefill.

The long description: an incoming query is checked against an exact cache; on a miss, a semantic cache; on a miss, a retrieval cache keyed on the query embedding or normalized query; a hit skips ANN and reranking. After retrieval, a prefix-cache check decides between a short prefill and a full prefill, and either path returns the answer.

Tier one, exact match. Key on the normalized query plus tenant, permissions version and corpus version. Hit rates are modest for free-text questions and high for templated or suggested queries, but the cost is a hash lookup and the correctness risk is nearly zero provided the key includes everything that affects the answer.

Tier two, semantic cache. Store answers keyed by embedding similarity, and return a stored answer when a new query is close enough. This can eliminate the whole pipeline, including the LLM, which makes it the highest-value tier and the riskiest: a similarity threshold that is too loose returns the wrong answer confidently, and a stale cache returns outdated facts. The design questions of thresholds, invalidation and per-tenant isolation are worked through in our semantic caching architecture guide.

Tier three, retrieval cache. Cache the candidate list or final chunk set for a query embedding or normalized query, with a short time-to-live and invalidation on index updates. This skips ANN and reranking but still calls the LLM, so the generated answer reflects fresh prompt instructions. It is a good fit when retrieval is the expensive part and answers must be personalized or rephrased.

Tier four, prefix cache. Described above, this is a serving-layer optimization rather than an application cache. It shortens prefill for the stable portion of the prompt.

Cache effectiveness depends on the traffic distribution. Measure the fraction of queries that repeat or near-repeat before building anything; support and documentation assistants often show heavy repetition, while exploratory research tools show little. Build the cheapest tier whose measured hit rate pays for its complexity.

Prompt Size, Prefill and Streaming

Context length is a latency cost

Prefill time grows with the number of prompt tokens, and in standard attention the compute grows faster than linearly with sequence length. Stuffing twenty chunks into the prompt because the context window allows it is therefore a latency decision as well as a cost decision. Retrieval quality improvements that let you send five good chunks instead of fifteen mediocre ones pay back twice: shorter prefill and often better answers, since long-context models can underuse material buried in the middle of a prompt, a pattern reported in the “Lost in the Middle” study by Liu et al. (2023).

Prompt compression takes this further. Approaches include extractive trimming (keep only the sentences in each chunk that score highest against the query), summarization of chunks at index time, and learned token-pruning methods such as LLMLingua from Microsoft Research, which reports large compression ratios on some tasks with modest quality loss. Treat reported compression ratios as task-specific. The simplest durable wins are mundane: strip boilerplate, navigation text and duplicate passages at ingestion time, deduplicate near-identical chunks at retrieval time, and cap each chunk at a sensible length.

Streaming as the primary UX lever

If you can only do one thing for perceived speed, stream. Server-sent events (SSE) or WebSockets deliver tokens as they are generated, so the user sees the first words at TTFT and reads at human speed while the rest decodes. Perceived latency is then TTFT plus almost nothing, rather than TTFT plus the full decode.

Streaming also opens up a design space beyond tokens. Send structured progress events before the answer starts: a “searching” status immediately, a “sources” event as soon as the reranker returns (so the user can see which documents were found while the model is still prefilling), and then the token stream. These events cost nothing on the critical path and turn dead time into visible progress. Figure 4 shows the event sequence.

Figure 4: Streaming sequence. The client receives a status event at once, a sources event after reranking, and tokens from the first decoded token onward.

The long description: the client sends a question; the RAG service immediately emits a retrieving status event, queries the vector store with hybrid search, sends the candidates to the reranker, which returns the top five; the service emits a sources event, sends the prompt to the LLM, receives the first token, and streams tokens to the client.

Two implementation notes save debugging time. Disable proxy buffering on any reverse proxy or CDN in the path; a single buffering hop silently converts a stream back into a batch. And measure TTFT at the client, not at the service, because the gap between “service emitted first token” and “browser painted it” can be hundreds of milliseconds on a bad path.

Decode speed and the reading-speed target

Once streaming works, decode throughput only needs to beat reading speed. Typical adult silent reading is on the order of 200 to 300 words per minute, which is roughly 5 to 7 tokens per second at about 1.3 tokens per word. Any serving stack that sustains tens of tokens per second per request is already ahead of the reader, so squeezing decode further yields little perceived benefit for chat, while it matters for programmatic consumers that wait for the full response. Know which of your consumers is a human and which is a pipeline, and budget separately.

Output length is the remaining decode lever. A system prompt that asks for concise answers, a max-tokens cap and structured output with short fields all cut total completion time directly, because decode is sequential.

Instrumenting the Pipeline

You cannot manage a budget you cannot see. The minimum viable instrumentation is a span per stage, tagged with the request id, tenant, cache outcome and the stage’s own parameters, exported to a tracing backend such as one speaking OpenTelemetry. Histograms per stage give you p50, p95 and p99; a trace per slow request tells you which stage broke it.

The sketch below shows an async pipeline with parallel branches, a propagated deadline, per-stage timing and graceful degradation. It is illustrative: the embed, ann_search, bm25_search, rerank and stream_llm functions stand in for your own clients.

import asyncio, time
from contextlib import asynccontextmanager

class Deadline:
    def __init__(self, total_ms: float):
        self.end = time.monotonic() + total_ms / 1000
    def remaining_ms(self) -> float:
        return max(0.0, (self.end - time.monotonic()) * 1000)

timings: dict[str, float] = {}

@asynccontextmanager
async def stage(name: str):
    t0 = time.monotonic()
    try:
        yield
    finally:
        timings[name] = (time.monotonic() - t0) * 1000
        # in production: histogram.record(timings[name], {"stage": name})

async def with_timeout(coro, ms: float, fallback):
    try:
        return await asyncio.wait_for(coro, timeout=ms / 1000)
    except asyncio.TimeoutError:
        return fallback

def rrf(lists, k=60, top=50):
    scores = {}
    for lst in lists:
        for rank, doc_id in enumerate(lst, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)[:top]

async def answer(query: str, send_event):
    t_start = time.monotonic()
    dl = Deadline(total_ms=1500)          # TTFT goal
    await send_event({"type": "status", "text": "retrieving"})

    async def dense():
        async with stage("embed"):
            vec = await embed(query)
        async with stage("ann"):
            return await ann_search(vec, k=50, ef_search=80)

    async def lexical():
        async with stage("bm25"):
            return await bm25_search(query, k=50)

    d_task = asyncio.create_task(with_timeout(dense(), 250, []))
    l_task = asyncio.create_task(with_timeout(lexical(), 150, []))
    dense_ids, lex_ids = await asyncio.gather(d_task, l_task)

    async with stage("fuse"):
        cands = rrf([dense_ids, lex_ids], top=50)

    # shrink rerank work to what the remaining budget allows
    n = 50 if dl.remaining_ms() > 900 else 20 if dl.remaining_ms() > 500 else 0
    if n:
        async with stage("rerank"):
            top = await with_timeout(rerank(query, cands[:n]), 300, cands[:5])
    else:
        top = cands[:5]
    await send_event({"type": "sources", "ids": top[:5]})

    async with stage("prefill_to_first_token"):
        first = True
        async for tok in stream_llm(build_prompt(query, top[:5])):
            if first:
                timings["ttft_ms"] = (time.monotonic() - t_start) * 1000
                first = False
            await send_event({"type": "token", "text": tok})

Three details matter. The deadline is passed implicitly through dl.remaining_ms(), which the rerank step uses to shrink or skip itself. Each branch has a timeout and an empty-result fallback, so a stalled BM25 index degrades quality instead of failing the request. And every stage writes a named timing, which you export as a histogram so that the p95 of rerank is a dashboard panel rather than a guess.

Beyond latency, trace quality signals alongside timings: retrieval recall on a canary set, the fraction of answers with citations, and the rate at which fallbacks fire. A latency dashboard that is green because every request is silently skipping the reranker is worse than a red one.

Worked Budget: Before and After

Applying the levers to the earlier table produces the following. These numbers are again illustrative, derived from the assumptions stated; they show how the optimizations compose, not what your system will achieve.

Stage	Before p95 (ms)	Change applied	After p95 (ms)
Query rewrite	600	Gated off for single-turn queries; speculative otherwise	0 on critical path for most requests
Query embedding	180	Regional endpoint, connection reuse, small cache	90
ANN search	40	Tuned ef_search to the recall knee	30
BM25	35	Already parallel	hidden
Rerank	450	20 candidates, truncated chunks, deadline	180
Prefill	900	5 chunks instead of 8, prefix cache on stable prompt	450
Misc and network	57	Streaming without proxy buffering	40
TTFT	~2,200		~800 to 900

The sum of the “after” column is not the arithmetic total of its p95s, since p95s of independent stages do not add to the p95 of the sum, but it is a conservative planning figure. The point is the ordering of effort: the three biggest wins (skipping the rewrite, trimming the reranker and shortening prefill) came from changing what runs, not from making any one component faster. This is the typical shape of a RAG latency optimization effort, and the reason a budget table is worth the hour it takes to build.

Trade-offs, Gotchas, and What Goes Wrong

Every latency win has a quality price, so measure both. Reducing ef_search, cutting reranker candidates and compressing prompts each lowers recall or groundedness somewhere. Keep a labelled evaluation set and report quality next to latency in the same dashboard, or you will optimize your way to a fast, wrong system.

Speculation multiplies load. Speculative retrieval and parallel branches spend extra compute to save time. Under a traffic spike, the extra load can push shared vector nodes into queueing, which raises everyone’s latency. Put the speculative work behind an admission control or turn it off when the system is saturated.

Semantic caches can be confidently wrong. A threshold tuned on one embedding model breaks when you upgrade the model. A cache without tenant and permission keys leaks one customer’s answer to another. Treat cache keys as a security boundary and version them with the corpus and model.

Cold starts and index rebuilds produce tail spikes. After a deploy, caches are empty, the index may be paging from disk and connection pools are cold. Warm new instances with replayed traffic before routing real users, and avoid rebuilding or compacting indexes on nodes serving queries.

Retries amplify. A client that retries a slow request adds load to an already slow system. Use hedged requests sparingly (send a second copy after the p95 delay, as discussed in “The Tail at Scale”), cap retries and prefer deadlines to blind retries.

Provider variance is outside your control. Hosted embedding, reranking and LLM endpoints have their own tails and rate limits. Maintain a fallback path, such as a smaller model or a cached answer, for when a provider degrades, and track provider latency separately so you can tell your regression from theirs.

Streaming hides failure until it is too late. Once tokens have started flowing you cannot swap in a fallback. Do the cheap sanity checks (retrieval returned something, safety pre-checks passed) before the first token, and design the client to handle a stream that ends abruptly.

Practical Recommendations

Start by measuring, not tuning. Add a span per stage, plot p50, p95 and p99 for each, and write the budget table with your real numbers. Almost every team discovers that the stage they assumed was slow is not the biggest one. Then work the list in order of expected return.

Set a TTFT goal at p95 (for chat, one to two seconds is a reasonable starting target) and allocate it across stages with 20 to 30 percent headroom.
Propagate a deadline through the call chain and give every external call a timeout and a fallback.
Run independent stages concurrently: embedding, lexical search, permissions and cache lookups.
Gate or speculate the query rewrite; skip it for single-turn queries without references.
Measure your recall-versus-latency curve and pick ef_search at the knee, with your real filters.
Rerank fewer, shorter candidates; add an early exit and a deadline; compare cross-encoder and late-interaction options on your own data.
Put stable prompt text first to benefit from prefix caching, and send fewer, better chunks.
Stream tokens, emit status and sources events, and verify that no proxy buffers the stream.
Add caches in order of measured hit rate, keyed with tenant, permission and corpus version.
Track quality metrics beside latency so that a speedup cannot hide a regression.

Revisit the budget whenever you change the embedding model, the reranker, the LLM or the corpus size by an order of magnitude; each shifts where the milliseconds go.

Frequently Asked Questions

What is a good RAG latency target?

For chat, aim for a time to first token of one to two seconds at p95, then stream the remainder at reading speed. Total completion for a few hundred tokens will typically take several seconds, which is acceptable when streaming. Programmatic consumers that wait for the full response need a separate target. Set targets per percentile and per stage, and leave headroom of 20 to 30 percent for load spikes and retries.

Which stage usually dominates RAG latency?

In most production pipelines the LLM prefill and decode dominate total time, while among the pre-LLM stages the query rewrite, the reranker and the embedding network call usually outweigh the vector search itself. Index search is often tens of milliseconds when the index fits in memory. Measure your own traces; the answer changes with hosted versus self-hosted components and with prompt length.

How do I reduce reranker latency without hurting quality?

Rerank fewer candidates (test thirty against fifty or one hundred), truncate chunks, batch on a GPU, add an early exit when retrieval scores are decisive, and set a deadline with a fallback to the fused order. Consider late-interaction models if you need to score many candidates cheaply. Validate each change on a labelled evaluation set, since quality loss is sometimes concentrated in a slice of hard queries.

Does a higher ef_search always improve results?

It raises recall of the approximate search, but with sharply diminishing returns, and it raises latency roughly in proportion to the extra distance computations. Past the knee of your recall curve, extra beam width mostly adds latency. Also remember that ANN recall is only one input to answer quality; a reranker and good chunking often matter more than the last point of recall.

Should I cache RAG answers?

Often yes, but tier your caches. Exact-match caches are safe and cheap. Semantic caches give the biggest savings and the biggest correctness risk, so key them by tenant, permissions, model and corpus version, and tune thresholds on real traffic. Retrieval caches skip ANN and reranking while still generating a fresh answer. Measure repeat rates first; exploratory workloads gain little from answer caching.

Does streaming reduce actual latency?

No, total completion time is unchanged, but perceived latency drops sharply because the user reads while the model generates. Streaming also lets you emit status and source events during retrieval, converting dead time into visible progress. It does require that no proxy or CDN buffers the response, and that you perform sanity checks before the first token since you cannot retract a started stream.