Agentic RAG Architecture: Retrieval Inside the Agent Loop

Your support bot retrieves five chunks, stitches them into a prompt, and answers. The chunks were mediocre, the answer confidently wrong, and nothing in the pipeline noticed. That failure mode is structural, not a tuning bug. Agentic RAG fixes it by moving retrieval inside the agent loop, so the model can plan, decompose the question, retrieve, judge what came back, and retrieve again before it commits to an answer.

This matters now because 2026 retrieval workloads are harder: multi-hop questions, contradictory sources, and users who expect grounded citations rather than plausible prose. A single retrieve-then-generate pass cannot recover from a bad first retrieval. An agentic loop can. This post is a practitioner’s map of that architecture: what each component does, how the loop terminates, what it costs, how to evaluate it, and when a static pipeline is still the right call.

What this covers: the reference loop, planner and query decomposition, hybrid retrieval and reranking, reflection patterns (CRAG, Self-RAG, plan-and-execute), evaluation metrics, cost and latency budgets, failure modes, and clear guidance on when not to reach for agentic RAG at all.

Context and Background

Static retrieval-augmented generation is a two-step function. Embed the query, fetch the top-k nearest chunks from a vector store, concatenate them into the prompt, and generate. It works well when the question maps cleanly to one region of your corpus and the answer lives in a couple of passages.

The design assumes the first retrieval is good enough. That assumption breaks in predictable ways. A vague query embeds to a vague region and pulls loosely related chunks. A multi-hop question (“which vendor in our 2024 contracts also failed the 2025 audit?”) needs two retrievals chained together, but static RAG only runs one. Ambiguous terminology, acronyms, and synonyms all degrade recall silently.

The deeper problem is that static RAG has no feedback signal. It cannot tell whether the retrieved context actually answers the question. If retrieval misses, generation fabricates to fill the gap, and the system reports success. There is no step that asks, “is this enough to answer faithfully?”

Retrieval quality dominates end-to-end accuracy far more than model choice. If the right passage never enters the context window, no amount of prompt engineering recovers it. This is the lever agentic systems pull: they treat retrieval as an action the model can invoke, inspect, and repeat, not a fixed preamble. For the graph-structured variant of this problem see our GraphRAG knowledge-graph architecture guide, and for the general shape of these pipelines the AWS overview of RAG is a solid primer.

The industry framing crystallized around Self-RAG and FLARE, which showed that letting the model decide when and what to retrieve beats a fixed schedule. Agentic RAG generalizes that insight into a full control loop.

A worked example makes the gap concrete. Suppose recall on the first retrieval is 70 percent — a realistic number for a tuned dense index on a technical corpus. Static RAG answers with whatever those chunks contain, so 30 percent of answers are built on incomplete evidence and there is no signal that anything is missing. An agentic loop that re-retrieves once when the critique flags low coverage can lift effective recall well past 90 percent, because the second, rewritten query targets exactly the gap the first pass left. The cost is one extra retrieval and one extra grading call, paid only on the queries that need it.

That conditional cost is the whole economic argument. Static RAG pays a flat, low price and eats a fixed error rate. Agentic RAG pays more, but only on the hard queries, and converts that spend into recovered accuracy. Whether the trade is worth it depends entirely on your query mix, which is why measuring where static RAG fails comes before adding any loop.

The Agentic RAG Reference Architecture

Agentic RAG is a control loop where an LLM-driven planner treats retrieval as a callable tool rather than a fixed first step. The model decomposes the query into sub-queries, runs hybrid search, reranks candidates, drafts an answer, critiques it for faithfulness and coverage, then either re-queries with a rewritten search or stops when it is confident or the budget is exhausted.

Figure 1: The agentic RAG loop — a query flows through a planner into sub-queries, hybrid retrieval, reranking, generation, and a critique gate that either re-queries or emits the final answer.

Figure 1 shows the loop as a cycle rather than a pipeline. A user query enters the planner, which assesses intent and sets a retrieval budget. The planner decomposes the query into sub-queries, each of which drives a hybrid retrieval and cross-encoder rerank. The generator drafts an answer, a critique step scores it, and a decision gate routes control either back to the planner for another pass or forward to the final answer. The single arrow that distinguishes this from static RAG is the one that loops back.

The planner: intent, decomposition, and budget

The planner is the brain of the loop. Given the raw query, it classifies intent, decides whether retrieval is even needed, and allocates a budget of retrieval calls and tokens. Simple lookups get one hop; comparative or multi-entity questions get a plan with several.

Decomposition is where most quality is won. “Compare our EU and US data-retention policies for IoT telemetry” is unanswerable as a single embedding. The planner splits it into “EU data-retention policy IoT telemetry” and “US data-retention policy IoT telemetry,” retrieves each independently, then composes. Each sub-query is narrow enough to embed to a tight, relevant region.

Query rewriting is the planner’s other core job. Users write terse, underspecified queries. The planner expands acronyms, adds synonyms, and rephrases into the vocabulary of the corpus. A rewrite from “OEE for line 3” to “overall equipment effectiveness metrics production line 3” can swing recall by double-digit points on a technical corpus.

The budget matters because an unbounded loop is a cost and latency hazard. A typical policy caps total retrieval calls (three to five), sets a wall-clock ceiling, and defines a confidence threshold that lets the loop exit early. Budget is not a nice-to-have; it is the difference between a system that terminates and one that spirals.

Decomposition also needs a stopping rule of its own. Over-decomposition is a real failure: a planner that splits every query into six sub-queries multiplies retrieval cost and often produces redundant chunks that the reranker then has to fight. A good planner decomposes only when the query genuinely spans multiple entities, time ranges, or documents, and answers atomic questions in a single hop. Treat the number of sub-queries as a cost the planner must justify, not a default.

There is a build choice here worth naming. The planner can be a separate, cheaper model called explicitly, or the same generation model prompted to plan first. A separate small planner is cheaper per call and easier to evaluate in isolation, but adds a network hop and a second failure surface. A single model that plans and generates is simpler and shares context, at a higher per-call cost. Most 2026 production systems run a small dedicated planner for routing and decomposition and reserve the large model for generation and critique.

Retrieval and reranking as an inner stage

Each sub-query flows through a retrieval stack, not a single index lookup. The dominant 2026 pattern is hybrid search: run a sparse lexical retriever (BM25) and a dense embedding retriever in parallel, then fuse their results.

Lexical search nails exact terms — part numbers, error codes, proper nouns — that embeddings blur. Dense search captures paraphrase and semantic similarity that keywords miss. Reciprocal rank fusion (RRF) merges the two ranked lists into one without needing calibrated scores, and it is robust enough that most teams use it as the default fusion method.

The fused candidate set is deliberately over-fetched — say the top 50 — then narrowed by a cross-encoder reranker to the top 5 to 8 that actually enter the prompt. This two-stage retrieve-then-rerank shape is the workhorse of production agentic retrieval, and the next section takes it apart in detail.

The critique gate and termination

After generation, the critique step decides whether the draft is trustworthy. It grades the answer against the retrieved context for faithfulness (is every claim supported?) and coverage (did we retrieve enough to answer fully?). This is the reflection that names the whole pattern.

If the critique passes, the loop emits the answer. If it fails, control returns to the planner, which rewrites the failing sub-query and retrieves again. Termination is governed by three conditions in priority order: confidence threshold met, retry budget exhausted, or an explicit abstain when no adequate context can be found. A well-designed loop abstains gracefully rather than fabricating — a property static RAG cannot offer.

Deeper Analysis: Patterns and a Walk-through

The reference loop has several named implementations that differ in where and how they insert the critique. The three that dominate 2026 production systems are Corrective RAG (CRAG), Self-RAG, and plan-and-execute. Figure 2 contrasts the static and agentic control paths so the added machinery is visible at a glance.

Figure 2: Static RAG runs a straight line from query to answer; agentic RAG inserts a plan step and a critique gate that can loop back, turning a pipeline into a controller.

The long-description of Figure 2: on the left, static RAG proceeds query, retrieve top-k, generate, answer, with no branch. On the right, agentic RAG proceeds query, plan and decompose, retrieve and rerank, generate, critique — and the critique either loops back to the plan step when it finds a gap or proceeds to the answer when the draft is grounded.

CRAG inserts a lightweight retrieval evaluator between retrieval and generation. It grades each retrieved chunk as correct, ambiguous, or incorrect. Correct chunks pass through; ambiguous ones trigger a refined re-retrieval; incorrect ones trigger a fallback, often a web search or a broadened query. CRAG is retrieval-centric — it fixes bad context before generation ever sees it.

Self-RAG puts the decision inside the generator. The model is trained to emit reflection tokens that signal whether to retrieve, whether a passage is relevant, and whether its own output is supported. Self-RAG interleaves retrieval and generation token by token, which is powerful but requires a model tuned for those control tokens.

Plan-and-execute is the most general. A planner writes an explicit multi-step plan up front, an executor runs each step (including retrievals and other tools), and a replan step revises the plan when a step returns something unexpected. This is the natural fit for multi-hop and tool-heavy tasks, and it is what most LangGraph and LlamaIndex agentic-retrieval templates implement today.

Pattern	Where critique lives	Best for	Main cost
CRAG	After retrieval, before generation	Noisy corpora, single-hop with quality risk	Extra grader call per retrieval
Self-RAG	Inside the generator (reflection tokens)	Fine-grained faithfulness control	Requires a tuned model
Plan-and-execute	Explicit planner + replan step	Multi-hop, tool use, comparisons	Multiple LLM calls per query

The three are not mutually exclusive. A common production shape wraps CRAG-style chunk grading inside a plan-and-execute loop: the planner decomposes, each retrieval is graded and corrected CRAG-style, and a final critique decides termination. Pick the outer pattern by task shape and borrow the critique mechanism that fits your model and corpus.

Multi-agent retrieval and tool use

Retrieval is one tool among several. Once the model can call retrieval as an action, it can call other actions in the same loop: a SQL query against a metrics warehouse, a web search for fresh facts, a calculator, or a code sandbox. The critique gate then judges an answer assembled from heterogeneous evidence, not just vector-store chunks. This is where agentic RAG blurs into general agents.

Multi-agent retrieval splits the work across specialists. A router dispatches sub-queries to source-specific retrievers — one owns the product docs, another the ticket history, another an external API — each tuned to its own index and chunking. A synthesizer agent then fuses their returns and a critic grades the whole. This isolates retrieval quality per source and lets you evaluate and improve each retriever independently, at the cost of orchestration complexity and more inter-agent calls.

The trap with multi-agent designs is premature decomposition into agents. Every agent boundary adds a message hop, a serialization step, and a place for context to be lost. Start with one loop and multiple tools; graduate to multiple agents only when a single planner can no longer hold the routing logic or when teams need to own retrievers independently.

The retrieval stack, in detail

Figure 3 zooms into the inner retrieval stage that every pattern shares. Getting this stack right delivers more accuracy per dollar than any change to the outer loop.

Figure 3: The retrieval stack — a sub-query fans out to BM25 and dense search, the results fuse via reciprocal rank fusion, a cross-encoder reranks the candidates, and a dedup and packing step assembles the final context.

The cross-encoder is the highest-leverage component. Unlike the bi-encoder used for the dense index (which embeds query and document separately), a cross-encoder reads the query and candidate together and scores their joint relevance. It is too slow to run over a whole corpus but ideal for rescoring 50 candidates. Public benchmarks in 2026 show cross-encoder reranking adding roughly 33 to 40 percent accuracy improvement for about 120 ms of added latency, and on the Agentset leaderboard Zerank-2 leads at 1638 ELO with Cohere Rerank 4 Pro close behind at 1629.

Context assembly is the underrated final step. After reranking you still must pack chunks into a token budget, dedup near-identical passages, and order them so the most relevant sit where the model attends best. Sloppy packing wastes the gains the reranker just bought you.

Ordering deserves special care because of the lost-in-the-middle effect: models attend most reliably to the start and end of a long context and can skip evidence buried in the center. A practical rule is to place the single highest-ranked chunk first, the second-highest last, and fill the middle with the remainder. On long contexts this ordering alone can recover several points of faithfulness at zero extra retrieval cost.

Chunking upstream governs the ceiling on all of this. Chunks that are too small fragment a single idea across several passages, so no one chunk is self-contained and recall suffers. Chunks that are too large dilute the embedding and drag irrelevant text into the window. Overlapping windows of a few hundred tokens, split on semantic boundaries rather than fixed character counts, are the sane default; agentic loops amplify good chunking and cannot rescue bad chunking.

Evaluating retrieval quality

You cannot improve what you cannot measure, and agentic RAG has more places to fail than static RAG. Four metrics anchor a practical eval program.

Context precision asks whether the retrieved chunks are relevant and ranked above the irrelevant ones. Context recall asks whether retrieval found all the passages needed to answer. Faithfulness asks whether every claim in the answer is supported by the retrieved context. Answer relevancy asks whether the answer actually addresses the question.

Precision and recall diagnose the retriever; faithfulness and answer relevancy diagnose the generator. Splitting them tells you which half to fix. A faithful answer built on low-recall context is confidently incomplete; a high-recall context with low faithfulness means the generator is ignoring its evidence.

Here is the loop in compact pseudocode, showing where each eval hooks in:

def agentic_rag(query, max_hops=4, conf_target=0.8):
    plan = planner.decompose(query)          # sub-queries + budget
    context, hops = [], 0
    while hops < max_hops:
        for sub_q in plan.pending():
            hits = hybrid_search(sub_q, k=50)  # BM25 + dense + RRF
            ranked = cross_encoder.rerank(hits)[:8]
            context += assemble(ranked)        # dedup + token pack
        draft = generator.answer(query, context)
        review = critic.grade(draft, context)  # faithfulness + coverage
        if review.faithful and review.confidence >= conf_target:
            return draft
        plan = planner.replan(query, review)   # rewrite failing sub-queries
        hops += 1
    return abstain_or_escalate(query, context) # budget spent

Run these evals offline on a labelled set before shipping, then keep a sampled online eval with an LLM-as-judge grader calibrated against human labels. Without tracing which hop retrieved what, a production agentic system is a black box: you see p95 latency and an answer, but not which retrieval missed. Pair evals with per-hop traces from day one.

Trade-offs, Gotchas, and What Goes Wrong

The loop that buys you quality also buys you a new class of failures. Figure 4 shows the guardrail loop that keeps those failures bounded.

Figure 4: The guardrail loop — context-precision and faithfulness checks feed a decision gate that either emits the answer, retries within budget, or abstains and escalates when the budget is spent.

Cost and latency blowup is the first and worst. Each hop is one or more LLM calls plus retrieval plus reranking. A four-hop query can cost five to ten times a static answer and take several seconds. If you do not cap hops and set a wall-clock ceiling, a small fraction of pathological queries will dominate your bill and your tail latency.

Work the latency budget explicitly. A single hop might spend 200 ms on hybrid retrieval, 120 ms on reranking, and 800 ms to 2 seconds on generation and critique, so four hops can push past 8 seconds before any network overhead. That is fine for an async research task and unacceptable for an interactive chat. Decide your p95 target first, then let it dictate the hop cap — not the other way around. Streaming the first grounded draft while later hops refine it is a common way to hide the tail from the user.

Infinite loops are the failure mode of an unbounded critic. If the critique step is too strict it never accepts an answer, and the loop rewrites forever. Always enforce a hard hop cap and require the loop to abstain when the cap is hit — abstention is a feature, not a bug.

Retriever drift and over-retrieval are subtler. Aggressive query rewriting can wander away from the user’s actual intent across hops, so each retrieval is individually plausible but collectively off-topic. Over-retrieval floods the context window with marginally relevant chunks, diluting the signal and raising cost. Pin rewrites to the original query and keep the reranked context tight.

Eval gaming is the trap that bites teams late. If you optimize a single LLM-judge metric, the system learns to satisfy the judge rather than the user — hedged answers that score faithful because they claim little. Use a small basket of metrics plus periodic human review, and treat a suspiciously perfect faithfulness score as a smell, not a win. Model-routing choices interact here too; see our LLM model-routing architecture guide for how to route the cheap grader and the expensive generator separately.

Practical Recommendations

Start static, then earn the loop. Ship a hybrid-search static RAG with a reranker first and measure its context precision, recall, and faithfulness. Only add agentic steps for the query classes where the static baseline demonstrably fails — usually multi-hop, comparative, and ambiguous questions. Most corpora have a large head of simple queries that never need a loop.

Route by complexity. Use a cheap classifier or the planner itself to decide whether a query gets zero hops (answer directly), one retrieval (static path), or the full agentic loop. This keeps median cost near static while reserving the expensive path for the queries that justify it. Persisting intermediate findings across hops also helps; our agent memory architecture guide covers the memory layer this loop leans on.

Instrument before you optimize. You cannot tune what you cannot see. Capture per-hop traces that record each sub-query, its retrieved chunks, the rerank scores, and the critique verdict. When quality regresses, that trace tells you within seconds whether the fault is a bad rewrite, a weak retriever, or an over-strict critic — a diagnosis that is otherwise guesswork across three coupled components.

Finally, budget for abstention. A system that says “I do not have enough information to answer” on the queries it cannot ground is more valuable in production than one that always answers. Wire abstention into the loop and into your evals, and treat a rising abstain rate as a signal to improve retrieval, not as a defect to suppress.

A practical checklist:

Hybrid retrieval (BM25 + dense) with reciprocal rank fusion, not dense-only.
A cross-encoder reranker over an over-fetched candidate set (50 in, 5 to 8 out).
Explicit budgets: hop cap, token cap, wall-clock ceiling, confidence threshold.
A critique gate that can abstain, plus per-hop tracing.
An offline eval set (precision, recall, faithfulness, answer relevancy) gating every release.
Complexity-based routing so simple queries skip the loop.

Frequently Asked Questions

What is the difference between agentic RAG and normal RAG?

Normal (static) RAG runs a fixed sequence: retrieve top-k chunks once, then generate. Agentic RAG turns retrieval into a tool the model controls inside a loop. It plans, decomposes the query, retrieves, critiques whether the context supports a faithful answer, and re-retrieves if not. The practical difference is a feedback signal: agentic RAG can recover from a bad first retrieval, while static RAG cannot notice one.

When should I not use agentic RAG?

Skip it when queries are simple, single-hop, and latency-sensitive. If most questions map cleanly to one region of your corpus, a static hybrid-search-plus-reranker pipeline is cheaper, faster, and easier to debug, and it often matches agentic quality on that traffic. The loop adds real cost and tail latency, so reserve it for multi-hop, comparative, or ambiguous queries where a single retrieval provably fails.

How does CRAG differ from Self-RAG?

CRAG adds a retrieval evaluator between retrieval and generation that grades chunks as correct, ambiguous, or incorrect and triggers re-retrieval or a fallback before generating. Self-RAG moves the decision inside the generator, which emits reflection tokens deciding when to retrieve and whether its output is supported. CRAG is model-agnostic and retrieval-centric; Self-RAG needs a model tuned for its control tokens but gives finer-grained, token-level faithfulness control.

What metrics should I use to evaluate agentic RAG?

Use four core metrics. Context precision (are retrieved chunks relevant and well-ranked) and context recall (did retrieval find everything needed) diagnose the retriever. Faithfulness (is every claim grounded in context) and answer relevancy (does the answer address the question) diagnose the generator. Run them offline on a labelled set to gate releases, then sample online with an LLM-as-judge calibrated against human labels, backed by per-hop tracing.

How much does agentic RAG cost compared to static RAG?

Expect a multiple, not a small increment. Each hop adds LLM calls for planning, generation, and critique, plus retrieval and reranking. A four-hop query can cost five to ten times a static answer and add seconds of latency. You contain this with hard hop caps, token budgets, a wall-clock ceiling, and complexity-based routing that sends simple queries down a cheap static path.

Does agentic RAG eliminate hallucinations?

No, but it reduces them materially. The critique gate checks whether the draft is supported by retrieved context and can loop back or abstain when it is not, which catches many fabrications static RAG would ship. It does not eliminate them: the grader itself can err, and a confidently wrong retrieval can still mislead the generator. Treat faithfulness scoring plus abstention as risk reduction, not a guarantee.

Agentic RAG Architecture: Retrieval Inside the Agent Loop (2026)