RAG Retrieval Evaluation: Recall@k, nDCG, MRR, Golden Sets and RAGAS in a CI Pipeline (2026)

A team ships a retrieval-augmented generation (RAG) assistant, users complain that answers are “sometimes wrong”, and the first instinct is to swap the language model. Three weeks and one expensive model upgrade later, the complaints persist, because the answer was never the problem: the right passage was not in the prompt. When the generator is handed the wrong context, no model is clever enough to recover. The cheapest, fastest way to find that out is RAG retrieval evaluation run on the retriever alone, before any tokens are generated.

This matters in 2026 because retrieval stacks have grown many moving parts: chunkers, embedding models, hybrid lexical and dense search, rerankers, query rewriters. Every one of them can be changed in an afternoon, and every one can silently shift quality in either direction. Without a measurement harness you are tuning by anecdote.

You will leave with exact definitions of Recall@k, MRR and nDCG with a worked numeric example, runnable Python for all three plus a bootstrap confidence interval, a method for building golden sets and synthetic queries (and their biases), a clear view of what RAGAS can and cannot tell you, and a design for CI regression gates and production drift monitoring.

What this covers: separating retrieval from generation eval, metric mechanics, golden set construction, synthetic query bias, LLM-as-judge metrics, statistics, CI gates, drift monitoring and failure modes.

Context and Background

Information retrieval (IR) has measured search quality for decades. The Text REtrieval Conference (TREC) established the pooled-judgment methodology, and the BEIR benchmark (Thakur et al., 2021) popularised nDCG@10 as the headline number for zero-shot retrieval across heterogeneous datasets. RAG did not invent a new problem; it inherited an old one and added a new consumer: instead of a person scanning ten blue links, a language model reads the top few passages and has to ground its answer in them.

That changes which metrics matter. A human forgives a mediocre rank 1 if the answer sits at rank 3. A prompt with a fixed token budget behaves differently: you typically pass the top 3 to 10 chunks, order can still influence the model (long-context models are known to use information at the start and end of a context better than the middle, as shown in the “Lost in the Middle” study), and irrelevant chunks are not free because they dilute and can mislead. So both coverage (is the evidence present at all?) and ranking quality (is it near the top, and is the junk below it?) matter.

The tooling split into two camps. Classical IR metrics need labelled relevance judgments: for each query, which chunks are relevant. They are deterministic, cheap and reproducible. LLM-judged frameworks such as RAGAS (Es et al., 2023, docs) avoid hand labels by asking a language model to judge relevance, faithfulness and similar properties. They are convenient and flexible, and they are noisy, costly and biased in ways you have to understand before trusting a gate on them.

This post argues for a layered approach: deterministic label-based retrieval metrics as the hard CI gate, LLM-judged metrics as a diagnostic layer, and production sampling as the drift detector. If you have not yet settled how documents are split and indexed, read the companion post on RAG chunking strategies, hybrid search and reranking first, because chunk boundaries define what “relevant” even means. If your evaluation shows the embedding model is the bottleneck, see embedding model fine-tuning with contrastive hard negatives and Matryoshka.

Separate Retrieval Evaluation from Generation Evaluation

Retrieval evaluation asks one question: given a query, did the system return the passages a correct answer needs, and rank them well? Generation evaluation asks a different one: given those passages, did the model produce a faithful, relevant, complete answer? Measuring them separately lets you localise a failure to one stage instead of guessing.

Figure 1: Where each metric family sits. Retrieval metrics need the query and the ranked chunks; generation metrics also need the answer. Only the left side can be a cheap deterministic CI gate.

Figure 1 shows the structure. The pipeline emits three observable artefacts: the retrieved ranked list, the assembled prompt context, and the generated answer. Retrieval metrics (Recall@k, MRR, nDCG, precision@k) read only the first. Context-quality judgments (context precision and recall in RAGAS terms) read the query plus the retrieved context. Generation metrics (faithfulness, answer relevance) read the answer too. The further right you go, the more an LLM judge is involved and the less deterministic the number.

Why the separation pays for itself

Consider the four outcomes of a single question. If retrieval fails and generation is faithful, the model either refuses or hedges, which is an acceptable failure mode and a fixable retrieval bug. If retrieval fails and generation is unfaithful, you get a confident hallucination, the worst case, and the fix is still upstream. If retrieval succeeds but generation fails, the problem is prompting, context ordering, or model capability. Only if you log the retrieved list can you tell these apart. A single end-to-end “answer correctness” score collapses all four into one number and sends you to the wrong stage half the time.

There is also a cost argument. Scoring a retriever against a golden set of a few hundred queries costs embedding calls and index lookups: seconds to minutes and effectively free. Scoring end-to-end answers costs one generation plus several judge calls per query. A change to the chunk size can be evaluated on every pull request if you only run the retrieval layer, and nightly if you run the whole stack.

Retrieval has its own two sub-stages

Modern retrieval is itself a funnel: a candidate generator (BM25, dense, or both fused) returns perhaps 50 to 200 candidates, then a reranker narrows to the final 3 to 10. Evaluate both. Candidate-stage Recall@100 tells you the ceiling: if the right chunk is not in the candidate set, no reranker can find it. Final-stage nDCG@10 or MRR tells you how well the reranker orders what it was given. A common and expensive mistake is tuning the reranker when Recall@100 is already the bottleneck.

Metric Definitions with a Worked Numeric Example

Metrics here use binary relevance unless stated: each chunk is relevant (1) or not (0) for a query. Graded relevance (0 to 3) is possible with nDCG and is discussed later.

Recall@k, precision@k and hit rate

Recall@k is the fraction of the relevant chunks that appear in the top k results: |relevant ∩ top_k| / |relevant|. Precision@k is the fraction of the top k that are relevant: |relevant ∩ top_k| / k. Hit rate@k (sometimes called success@k) is 1 if at least one relevant chunk is in the top k, else 0, which equals Recall@k when each query has exactly one relevant chunk.

Which one you should gate on depends on how many chunks the answer needs. For single-fact lookups, hit rate is the honest metric. For multi-hop or synthesis questions that need several chunks, Recall@k with multiple relevant items is the right one, because retrieving one of three necessary chunks gives the model a plausible but incomplete picture, which is exactly how partial hallucination happens.

MRR

Mean reciprocal rank averages, over queries, the reciprocal of the rank of the first relevant result: 1/rank_first_relevant, and 0 when none is retrieved. It rewards getting a relevant chunk to the very top and ignores everything after the first hit. That makes it a good fit for assistants that show or prioritise a single best passage, and a poor fit when the answer needs several chunks.

nDCG

Discounted cumulative gain adds up relevance with a logarithmic discount by position: DCG@k = Σ rel_i / log2(i + 1) for i from 1 to k. Normalising by the ideal ordering gives nDCG@k = DCG@k / IDCG@k, bounded in [0, 1]. The discount encodes diminishing attention down the list. Some implementations use the alternative form (2^rel − 1) / log2(i + 1) that boosts high grades; with binary relevance the two are identical, but with graded labels they differ, so always state which one you compute. This is why numbers from two libraries can disagree.

Worked example

One query, ten retrieved chunks. The golden set says three chunks are relevant (call them A, B, C). The retriever returns A at rank 2 and B at rank 5; C is not in the top 10.

Metric	Computation	Value
Recall@5	2 relevant found / 3 relevant	0.667
Recall@10	2 / 3	0.667
Precision@5	2 / 5	0.400
MRR (this query)	1 / 2	0.500
DCG@10	1/log2(3) + 1/log2(6) = 0.6309 + 0.3869	1.0178
IDCG@10	1/log2(2) + 1/log2(3) + 1/log2(4) = 1 + 0.6309 + 0.5	2.1309
nDCG@10	1.0178 / 2.1309	0.478

Notice what each number says. Recall@10 of 0.667 says a third of the needed evidence never reached the model. MRR of 0.5 says the first useful chunk is second, which is fine. nDCG of 0.478 is penalised both for the missing chunk and for the two misplaced ones. No single metric tells the full story, which is why the CI report should show several.

Runnable Python: Metrics and a Bootstrap Confidence Interval

The reference implementation below has no dependencies beyond NumPy. It is short enough to audit, which matters because a metric bug silently corrupts every decision downstream. It reproduces the worked example above; run it before trusting it on your data (the code was written for this article, and its output on the example was checked by hand against the table).

import math
import numpy as np

def recall_at_k(ranked, relevant, k):
    """ranked: list of chunk ids, best first. relevant: set of ids."""
    if not relevant:
        return None  # undefined: exclude the query, never score as 0 or 1
    return len(set(ranked[:k]) & relevant) / len(relevant)

def hit_at_k(ranked, relevant, k):
    return 1.0 if set(ranked[:k]) & relevant else 0.0

def reciprocal_rank(ranked, relevant, k=None):
    for i, cid in enumerate(ranked[:k], start=1):
        if cid in relevant:
            return 1.0 / i
    return 0.0

def ndcg_at_k(ranked, gains, k):
    """gains: dict chunk_id -> graded relevance (binary = 0/1)."""
    dcg = sum(gains.get(c, 0) / math.log2(i + 1)
              for i, c in enumerate(ranked[:k], start=1))
    ideal = sorted(gains.values(), reverse=True)[:k]
    idcg = sum(g / math.log2(i + 1) for i, g in enumerate(ideal, start=1))
    return dcg / idcg if idcg > 0 else None

def bootstrap_ci(per_query_scores, n_boot=10000, alpha=0.05, seed=0):
    x = np.array([s for s in per_query_scores if s is not None])
    rng = np.random.default_rng(seed)
    means = rng.choice(x, size=(n_boot, len(x)), replace=True).mean(axis=1)
    return x.mean(), np.quantile(means, [alpha / 2, 1 - alpha / 2])

# Worked example from the article
ranked = ["x1", "A", "x2", "x3", "B", "x4", "x5", "x6", "x7", "x8"]
rel = {"A", "B", "C"}
gains = {c: 1 for c in rel}
print(recall_at_k(ranked, rel, 5))      # 0.666...
print(reciprocal_rank(ranked, rel))     # 0.5
print(ndcg_at_k(ranked, gains, 10))     # ~0.478

Two design decisions are worth defending. First, queries with no relevant chunks return None and are excluded, because averaging them as zero punishes the retriever for an unanswerable question, and averaging them as one rewards it for nothing. Those queries belong in a separate “should refuse” test set. Second, the bootstrap resamples queries, not individual chunks, since the query is the independent unit; resampling at the wrong level yields intervals that are far too tight.

Building a Golden Dataset for RAG

A golden dataset is a versioned collection of (query, relevant chunk ids, optional reference answer) triples that represents what real users ask and what the corpus can answer. It is the single most valuable evaluation asset you will build, and it is the thing teams most often skip because it feels like manual labour. The labour is real; the payoff is that every future change becomes measurable.

Figure 2: Golden set construction. Real logs supply realism, synthetic generation supplies coverage, human review supplies trust, and versioning supplies reproducibility.

Sourcing queries

Figure 2 shows three sources feeding one reviewed set. Production logs are the most realistic: sample queries from real traffic, stratified by intent, length, and frequency band, and strip personal data before anything is stored. Subject-matter-expert (SME) queries cover the high-stakes questions that rarely appear in logs, such as safety or compliance questions. Synthetic queries (next section) fill coverage gaps for documents nobody has asked about yet. A pragmatic starting mix is a majority of real or SME queries with synthetic as a supplement, because a set that is mostly synthetic measures your generator’s idea of a question.

Labelling relevance without going insane

Labelling every chunk against every query is infeasible. The IR community’s answer is pooling: take the union of the top results from several diverse systems (BM25, two embedding models, a reranked list), and have annotators judge only that pool. Documents outside the pool are assumed non-relevant, which is a known bias: a future retriever that finds a genuinely relevant chunk the pool missed will be penalised. Mitigate it by periodically re-pooling with the newest system and adding newly found relevant chunks to the set.

Label at the right granularity. If you re-chunk the corpus, chunk-id labels break. Two robust options exist: label at the document and section level and map to chunks by character-span overlap, or store the answer-bearing span (document id plus character offsets) and define a chunk as relevant when it overlaps the span by more than a threshold. The span approach lets one golden set survive chunk-size experiments, which is precisely when you need evaluation most.

Size, strata and versioning

How many queries? Enough that the confidence interval is narrower than the effect you care about. As an illustration (not a measurement), with a hit rate near 0.82 on 200 queries, the 95% Wilson interval is roughly 0.76 to 0.87, about plus or minus five points, so a 2-point improvement is invisible. Hundreds of queries are a floor for overall numbers; per-segment reporting needs per-segment sample sizes. Tag each query with strata (intent, document type, difficulty, language) so the report can show where a change helps or hurts, since averages hide segment regressions.

Version the set like code: store it in Git or a data-versioning tool, give it a semantic version, record the corpus snapshot hash it was built against, and never edit in place. When the corpus changes, queries whose evidence was deleted must be re-labelled or retired. Hold out a blind test split that nobody tunes against; otherwise your development set slowly turns into a training set and the headline number optimistic.

Synthetic Query Generation and Its Biases

Synthetic generation prompts an LLM to write questions a given chunk can answer, then labels that chunk as relevant. It is cheap, scales to the whole corpus, and bootstraps evaluation before you have any traffic. RAGAS includes a testset generation feature along these lines, and BEIR-style synthetic query tools such as GPL and doc2query follow the same idea. Used carelessly it produces a flattering and misleading benchmark.

The lexical-overlap bias

A question written while staring at a chunk tends to reuse the chunk’s wording. A retriever with strong lexical matching, BM25 above all, looks excellent on such queries, while real users paraphrase, abbreviate and misspell. If your synthetic set rewards BM25-like behaviour, it will undervalue the dense retriever that actually serves real users better. Countermeasures include instructing the generator to paraphrase and avoid copying noun phrases, generating from a summary rather than the raw text, and measuring the n-gram overlap between query and source chunk so you can bucket or filter high-overlap items.

The single-chunk and answerability biases

Generators naturally produce questions answerable from one chunk. Real questions are often multi-hop, comparative, or ambiguous, and some are unanswerable from the corpus. A purely synthetic set under-represents all three. Add multi-chunk questions by seeding the generator with two linked chunks, include deliberate unanswerable queries, and keep them in separate strata.

The self-preference and ambient-knowledge problems

If the same model family writes the questions and later judges the answers, it may favour its own phrasing. And a strong model may write a query whose answer it already knows from pretraining, so a run with retrieval disabled still looks correct. Test for this by checking how many synthetic questions are answered correctly without any context; those items do not measure retrieval at all and should be dropped from generation evaluation.

A practical quality gate for synthetic items

Filter synthetic queries with three checks: a round-trip check (does a strong retriever or the generator itself recover the source chunk from the query, to discard ill-posed questions), an overlap filter (drop or flag near-verbatim queries), and a human spot-check of a random sample, say 50 items per batch, with an accept rate you record. If acceptance falls below a threshold you set, regenerate rather than patch. The accepted-rate trend itself becomes a quality signal for your pipeline.

RAGAS and LLM-as-Judge Metrics: What They Measure

RAGAS (Retrieval Augmented Generation Assessment) is an open-source framework whose original paper proposed reference-free evaluation: judge a RAG pipeline using an LLM rather than human labels. Its metric names and exact definitions have evolved between releases, so check the documentation of the version you pin; the descriptions here follow the paper and the docs as I understand them and were not re-verified against a specific release this run.

Figure 3: Inputs each RAGAS-style metric consumes. Context precision and recall judge the retrieved context; faithfulness and answer relevancy judge the generated answer.

The core metrics

Context precision asks whether the relevant items in the retrieved context are ranked near the top: an LLM judges each retrieved chunk as useful or not for the question (against a reference answer or the generated answer, depending on the variant), and the score weights by rank, so it behaves like a precision-at-each-rank average. It is the LLM-judged cousin of nDCG: it penalises junk placed above good chunks.

Context recall asks whether the retrieved context contains the information needed to produce the reference answer. The usual mechanism splits the reference answer into claims and checks whether each is supported by the retrieved context. It is the LLM-judged cousin of Recall@k, and it requires a reference answer.

Faithfulness (groundedness) splits the generated answer into atomic claims and checks each against the retrieved context; the score is the fraction of supported claims. A faithful answer can still be wrong if the context was wrong, which is why faithfulness is a generation metric and not a retrieval one.

Answer (response) relevancy measures whether the answer addresses the question, in the original formulation by generating likely questions from the answer and comparing their embeddings with the original question. It does not check factual truth.

Where judge-based metrics earn their keep

They are valuable in three situations. When you have no labels yet, a judge gives a first signal. When relevance is semantic rather than identifier-based (the answer is in a paraphrased chunk you never labelled), a judge can credit retrievals a strict ID match would count as misses. And for generation-side properties such as faithfulness there is no practical deterministic alternative.

Where they mislead

Judge scores carry error from several sources. The judge model is itself non-deterministic unless temperature is zero, and even then the provider can change the model underneath you. Judges show known biases: length and verbosity preference, position bias when comparing, and self-preference for text from their own family, as documented in the “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” work. Claim-splitting is itself an LLM step that can drop or invent claims. Finally, scores are not calibrated: 0.82 from one judge model is not comparable to 0.82 from another.

The consequence is a rule I would apply to every pipeline: never gate a merge on a judge score alone, and never compare judge scores across judge-model versions. Pin the judge model and prompt version, store raw judge outputs for audit, and calibrate against a human-labelled sample. Compute agreement (Cohen’s kappa or simple percent agreement) between judge and humans on 100 to 200 items; if agreement is poor on your domain, the metric is noise in a nice-looking format.

A decision table

Question	Best tool	Why
Did the retriever return the labelled evidence?	Recall@k, hit rate	Deterministic, free, reproducible
Is evidence ranked near the top?	nDCG, MRR	Position-aware, deterministic
Is there junk above good chunks?	Context precision (judge) or precision@k	Judge handles unlabelled relevant chunks
Is the answer grounded in context?	Faithfulness (judge)	No cheap deterministic equivalent
Did a change help or hurt?	Paired test on per-query deltas	Controls query difficulty

Statistical Significance and Confidence Intervals

Retrieval evaluation numbers are sample means over a finite query set, so they come with uncertainty. A change from 0.81 to 0.83 nDCG on 150 queries is, in the vast majority of cases, indistinguishable from noise. Treating small deltas as wins is how teams accumulate “improvements” that cancel out in production.

Paired comparison beats comparing intervals

Two systems are evaluated on the same queries, so their scores are correlated: hard queries are hard for both. Comparing two separate confidence intervals discards that structure and is conservative. Instead compute the per-query difference d_i = score_B(i) − score_A(i) and test whether the mean difference is zero. The bootstrap on d_i (resample queries, recompute the mean) gives a confidence interval for the improvement; if it excludes zero, the change is likely real. A paired permutation test (randomly flipping the sign of each d_i) is a similarly assumption-light alternative and is a standard recommendation in IR evaluation literature (Smucker, Allan and Carterette compared such tests for IR).

For binary outcomes like hit@k, McNemar’s test on the discordant pairs is natural. Illustration with made-up numbers: out of 200 queries, B hits where A misses on 14 queries and A hits where B misses on 6. The statistic is (14 − 6)² / (14 + 6) = 3.2, giving p of about 0.07, not significant at the usual 0.05 level despite a visible 4-point gain. The lesson: 14 versus 6 sounds convincing and is not.

def paired_bootstrap(a_scores, b_scores, n_boot=10000, seed=0):
    a = np.array(a_scores, dtype=float)
    b = np.array(b_scores, dtype=float)
    d = b - a                      # same queries, same order
    rng = np.random.default_rng(seed)
    idx = rng.integers(0, len(d), size=(n_boot, len(d)))
    boots = d[idx].mean(axis=1)
    lo, hi = np.quantile(boots, [0.025, 0.975])
    return d.mean(), (lo, hi)      # significant if the interval excludes 0

Multiple comparisons and effect size

If you test ten metrics and twelve segments per change, some will cross p < 0.05 by chance. Pre-declare one primary metric (for example nDCG@10 overall) as the decision criterion and treat the rest as diagnostics. For large batches of experiments apply a correction such as Bonferroni or Benjamini-Hochberg. Also report effect size, not just p-values: with thousands of queries, a trivial 0.3-point gain can be “significant” and still not worth the added latency of a larger reranker.

Variance you forgot to count

Sampling error from the query set is not the only variance. Approximate nearest neighbour indexes (HNSW and similar) can return slightly different results depending on build order and parameters, so rebuilding an index on identical data can shift Recall@k by a small amount. LLM-based query rewriters and judges add run-to-run noise. Quantify this by running the same configuration several times with different seeds; if run-to-run spread is, say, a point wide, require improvements to clear that floor.

Regression Gates in CI

The goal of a CI evaluation pipeline is simple: no change that degrades retrieval quality merges unnoticed. The design question is what runs when, because cost and latency scale with the number of judge calls.

Figure 4: A tiered CI design. Deterministic retrieval metrics run on every pull request, judge-based metrics run nightly, and a full report gates releases.

Tier 1: pull-request gate (deterministic, minutes)

On every pull request that touches chunking, embedding, index parameters, retrieval code, or prompts for query rewriting, run the retriever against the golden set and compute Recall@k, MRR and nDCG@10 with confidence intervals. No LLM generation, no judge. Compare against a stored baseline (the metrics of the current main branch on the same golden set version). Fail the check when the paired bootstrap shows a statistically credible drop, or when overall nDCG falls below a floor, or when any critical stratum drops by more than a set tolerance. The tolerance values are yours to choose from your measured run-to-run noise; I deliberately give no universal number because a correct one does not exist.

Tier 2: nightly (judge-based, tens of minutes)

Run the full pipeline end to end on the golden set or a sampled subset: faithfulness, context precision and recall, answer relevancy, plus latency and token cost per query. These are tracked as trends with alert thresholds rather than hard merge blockers, for the noise reasons above. A sustained downward trend opens a ticket; a single dip triggers a rerun.

Tier 3: release gate (full, hours)

Before a production rollout, run the full set including the blind test split, the unanswerable-query stratum, and adversarial cases such as near-duplicates and prompt-injection text inside documents. Compare against the production baseline and require sign-off on any regression in a protected stratum.

Mechanics that keep CI trustworthy

Freeze inputs. Pin the golden set version, corpus snapshot, embedding model version, index parameters and random seeds, and write all of them to the run metadata so any result can be reproduced.
Store per-query results, not only aggregates. Diffing which queries flipped from hit to miss is the fastest debugging tool you will have.
Update baselines deliberately. A baseline update is a reviewed commit, not an automatic overwrite on merge, otherwise slow decay becomes the new normal.
Keep the gate fast. If the PR check takes an hour, people route around it. Cache embeddings of the golden queries and of unchanged chunks.
Mind the cost of changing the embedding model: a model change forces re-embedding the entire corpus, so evaluate on a representative sample first, then confirm on the full index.

Drift Monitoring in Production

A golden set is a snapshot; production is a moving target. Corpus content changes, users start asking new kinds of questions, and upstream components get silently updated. Offline scores can stay green while real quality decays, so you need a second loop that watches live traffic.

What to monitor without labels

Several signals need no human labelling. Track the distribution of top-1 and top-k similarity scores: a drop in the mean top-1 cosine similarity, or a rise in the share of queries below a similarity floor, suggests queries are drifting away from what the index covers. Track the empty or low-confidence retrieval rate and the refusal rate of the generator. Track query embedding drift by comparing the centroid or distribution of recent query embeddings against a reference window with a distance such as maximum mean discrepancy or a simple population stability index on clusters. None of these says quality is bad; they say something changed and deserves a look.

Sampling for labelled truth

Pair the unlabelled signals with a small sampled review. Each week, draw a stratified sample of live queries (oversampling low-similarity and thumbs-down ones), run the judge-based metrics on them, and send a subset to human reviewers. Feed confirmed failures back into the golden set, tagged with the date and failure type. This closes the loop: production teaches the test set what it was missing, which is how a golden set stays representative.

Triggers worth wiring up

Corpus updates (re-index events), embedding or reranker model changes, large ingestion batches, and provider-side model deprecations should each trigger an out-of-band evaluation run. A silent change in a hosted embedding endpoint is a real risk: if the vendor updates weights behind a stable name, your stored vectors and fresh query vectors may no longer live in the same space. Pin versions where the provider allows, and alert on the version string in API responses where it is exposed.

Failure Modes: Where Retrieval Evaluation Itself Goes Wrong

Chunk-boundary errors

When an answer spans two chunks, or a heading that gives a passage its meaning lands in the previous chunk, retrieval can fail even though the document is in the index. Label-by-chunk-id golden sets hide this because the “right” chunk was defined by the old chunker. Span-based labels (above) expose it: Recall@k drops when a chunker splits an answer span, which is exactly the signal you want. Overlap between chunks and parent-document retrieval are the usual fixes, and both should be judged by the same metrics.

Near-duplicates and versioned documents

Policies, manuals and wikis contain many near-identical chunks across versions. The retriever returns a v2 chunk, the label says v3, and strict ID matching scores a miss even though the content is equivalent; the reverse mistake rewards returning an outdated version. Resolve it with canonical-equivalence groups: cluster near-duplicates (for example by MinHash or embedding similarity above a threshold), treat any member as relevant if the labelled one is, and separately test that the latest version outranks stale ones when freshness matters. Duplicates also crowd the top k, lowering effective diversity, so consider maximal marginal relevance or deduplication and measure its effect.

Label leakage and tuning to the test

If you generate synthetic queries from chunks and also fine-tune an embedding model on those same pairs, the golden set is contaminated. Keep fine-tuning data and evaluation data disjoint by document, not just by query, and keep the blind split sealed.

Metric-goal mismatch

High nDCG@10 can coexist with poor answers when the model only sees the top 3. Pick k to match the number of chunks you actually place in the prompt, and report Recall@k at that value. Conversely, optimising only Recall@100 while the reranker is weak produces a good ceiling and a mediocre experience.

Averages hiding segments

A 1-point average gain can hide a 10-point loss on non-English queries, short queries, or one product line. Always report per-stratum results and gate on protected strata.

Judge circularity

Using the same LLM family for query generation, retrieval-time query rewriting, and judging creates a closed loop that rewards itself. Diversify models across roles, and keep a human-labelled calibration set that never goes through the loop.

Practical Recommendations

Start small and make it reproducible before making it fancy. A 150 to 300 query golden set with span-level labels, a 40-line metrics module, and a PR check will catch most of the regressions that matter. Add judge-based metrics once the deterministic layer is stable, and treat them as trend instruments. Keep the evaluation code in the same repository as the retrieval code so a pull request can change both together, and make the golden set a versioned artefact with an owner.

Prefer decisions that survive uncertainty. Gate on paired confidence intervals, not on a single point estimate. Declare one primary metric per experiment. Choose k from your real prompt budget, and measure both the candidate-stage ceiling and the final-stage ranking separately. When a number surprises you, look at the per-query diff before believing it.

Checklist

[ ] Retrieval evaluated separately from generation, with ranked lists logged per query
[ ] Golden set built from real logs, SME queries and filtered synthetic queries, with strata tags
[ ] Span-based relevance labels that survive re-chunking; near-duplicates grouped
[ ] Blind test split sealed; fine-tuning data disjoint from evaluation data
[ ] Recall@k at the real prompt k, MRR and nDCG@10 computed with query-level bootstrap intervals
[ ] Paired test for every A/B change; one primary metric declared up front
[ ] Judge model and prompt pinned; judge agreement with humans measured
[ ] Three CI tiers: PR (deterministic), nightly (judge), release (full)
[ ] Production drift signals plus weekly sampled review feeding the golden set
[ ] Run metadata stores corpus hash, model versions, index parameters and seeds

Frequently Asked Questions

What is a good Recall@k for RAG?

There is no universal threshold, because it depends on corpus difficulty, query mix and how many relevant chunks each question needs. Treat any quoted “good” number as meaningless without the dataset. Use your own baseline: set Recall@k at the prompt budget as the primary retrieval metric, measure the current system with a confidence interval, and demand improvements that clear that interval. Compare configurations against each other on the same golden set, not against numbers from public benchmarks.

Should I use nDCG or MRR for RAG?

Use MRR when the application hinges on a single best passage, and nDCG when several chunks contribute to the answer or when relevance is graded. Because RAG prompts usually include multiple chunks, nDCG@k paired with Recall@k at your prompt size is the more informative default. MRR remains a useful, easy-to-explain companion, particularly for question-answering over FAQ-style content where one chunk holds the answer.

Is RAGAS enough to evaluate a RAG system?

No. RAGAS-style metrics are LLM-judged, so they are noisy, model-dependent and not calibrated across judge versions. They are excellent for faithfulness and for bootstrapping evaluation without labels, and weaker as a sole merge gate. Pair them with deterministic label-based retrieval metrics, pin the judge model, and validate judge scores against a human-labelled sample before relying on them for decisions.

How many queries does a golden dataset need?

Enough that the confidence interval is narrower than the improvement you want to detect. Hundreds of queries are a practical floor for overall metrics, and each segment you want to report needs its own adequate sample. Start with 150 to 300 high-quality queries, check interval widths, and grow the set where intervals are too wide or where production shows gaps. Quality and representativeness matter more than raw count.

Can I use synthetic queries as my only test set?

It is risky. Synthetic queries over-represent lexical overlap with the source chunk, single-chunk answerability and the generating model’s style, which flatters lexical retrievers and misses multi-hop and unanswerable cases. Use synthetic data to bootstrap and to fill coverage gaps, filter it with round-trip, overlap and human spot checks, and anchor the final gate on real or expert-written queries.

How do I detect retrieval drift in production?

Monitor label-free signals such as top-k similarity distributions, low-confidence and empty retrieval rates, refusal rates and query-embedding distribution shift against a reference window. Then sample live traffic weekly for judge-based and human review. Trigger extra evaluation on re-indexing and any embedding or reranker change. Feed confirmed failures back into the golden set so offline tests track what users really ask.