Fact-Check: Is RAG Dead Now That LLMs Have Million-Token Context Windows? What the 2024-2026 Evidence Shows

Fact-Check: Is RAG Dead Now That LLMs Have Million-Token Context Windows? What the 2024-2026 Evidence Shows

Fact-Check: Is RAG Dead Now That LLMs Have Million-Token Context Windows? What the 2024-2026 Evidence Shows

Every few months a model ships with a bigger window and the same post goes viral: retrieval-augmented generation is obsolete, just paste everything into the prompt. In November 2026 the premise looks stronger than ever. Google documents a 1,048,576-token input limit for Gemini 3.1 Pro Preview, and Anthropic now bills its current Claude models’ full 1M-token window at standard per-token rates. Yet the rag vs long context debate is not settled by window size, because the window was never the binding constraint. Accuracy over long inputs, cost per query, prefill latency and KV-cache memory are. This fact-check takes the four claims that circulate most widely and scores each one against primary evidence published between 2023 and 2026: peer-reviewed benchmarks, vendor research and current official pricing pages. You will leave with a verdict per claim, worked cost and latency numbers you can rerun for your own corpus, and a decision procedure for picking RAG, long context or the hybrid that most production teams actually need.

What this covers: the four viral claims and their verdicts, what Lost in the Middle, RULER, NoLiMa, LongBench v2, Chroma’s Context Rot report, the Databricks long-context RAG study and Self-Route measured, worked token-cost and prefill math, KV-cache memory, hybrid patterns with prefix caching, a decision matrix, failure modes and an FAQ.

Context and Background

Retrieval-augmented generation (RAG) means the application selects a small set of relevant passages at query time, usually with a vector index, a keyword index or both, and places only those passages in the prompt. Long-context prompting means the application skips selection and hands the model a large slab of source text, sometimes the entire corpus, trusting attention to find what matters. The two ideas were never strict alternatives. RAG became the default in 2023 because windows of 4K to 32K tokens left no other option. The rag vs long context question only became live when windows jumped by two orders of magnitude.

The jump came quickly. Google’s Gemini 1.5 technical report (arXiv 2403.05530) claimed “near-perfect retrieval (>99%) up to at least 10M tokens” on needle-style tests and contrasted that with 200K for Claude 3.0 and 128K for GPT-4 Turbo. Meta announced Llama 4 Scout with a 10M-token window in April 2025. By late 2026, 1M tokens is a routine product tier. Google’s model page lists a 1,048,576-token input limit for Gemini 3.1 Pro Preview, and Anthropic’s models overview lists a 1M-token window for Claude Fable 5.2, Opus 5.5, Sonnet 5.5 and Haiku 5.5.

Meanwhile, the research community built harder tests. The original needle-in-a-haystack test plants one sentence that lexically matches the question, which a model can locate by near string-matching. Later benchmarks varied the needle, required multi-hop tracing, removed literal overlap, or measured real tasks instead of synthetic ones. Those benchmarks are the evidence base for this rag vs long context fact-check, and they paint a very different picture from window-size marketing.

We have already examined the narrower question of how much of a window a model genuinely uses in our fact-check of 1M-token context windows, context rot and effective context length. This post builds on that and asks the system-design question instead: given what long-context models can and cannot do, when should a production application retrieve, when should it stuff the prompt, and what does each choice cost? If you are new to the retrieval side, our guide to RAG chunking strategies, hybrid search and reranking explains the components that the long-context camp proposes to delete.

RAG vs Long Context: Four Viral Claims Checked Against the Evidence

Short answer: RAG is not dead. Long-context models win on some whole-document reasoning tasks, but accuracy degrades well before advertised limits, and per-query cost scales with every token sent. The best-supported rag vs long context answer in 2024-2026 research is a hybrid: retrieve first, escalate to long context only when retrieval cannot answer.

RAG vs long context evidence map linking four viral claims to the studies and verdicts in this fact-check
Figure 1: Evidence map for the rag vs long context fact-check. Each viral claim on the left is scored against the primary studies that tested it.

Figure 1 is the map for the rest of this rag vs long context analysis. Four claims sit on the left. The middle column groups the primary sources that bear on each one: the Self-Route and OP-RAG comparisons, the long-context stress tests (Lost in the Middle, RULER, NoLiMa), real-workload studies (Chroma’s Context Rot, Databricks), and first-principles cost math. Arrows converge on a verdict. Two studies feed more than one verdict, which is a reminder that the same data cuts in both directions.

Viral claim Verdict Strongest evidence
“Long context killed RAG” False Self-Route (2024) and OP-RAG (2024): hybrids match or beat pure long context at a fraction of the tokens
“Models use all 1M tokens equally well” False Lost in the Middle (2023), RULER (2024), NoLiMa (2025), Context Rot (2025)
“Long context is cheaper than retrieval” Misleading Per-token pricing makes cost linear in prompt size; caching narrows but does not close the gap
“RAG is always more accurate” Mixed Self-Route found long context ahead on average with strong models; OP-RAG found the reverse with a tuned RAG

Claim 1: “Long context killed RAG” — False

The most cited head-to-head is Li et al., “Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach” (arXiv 2407.16833, EMNLP 2024 Industry Track). The authors compared Gemini-1.5-Pro, GPT-4o and GPT-3.5-Turbo on nine datasets from LongBench and ∞Bench. Their RAG baseline was deliberately plain: Contriever retrieval, 300-word chunks, top-5. With resources sufficient, long context won on average. Gemini-1.5-Pro scored 49.70 with long context against 37.33 with RAG, and GPT-4o scored 48.67 against 32.60.

That sounds like a eulogy for RAG until you read the second half. RAG and long-context predictions were identical for 63% of queries, according to the paper. That observation motivated Self-Route: ask the model to answer from retrieved chunks or declare the question unanswerable, and only send unanswerable queries to the full-context path. Self-Route scored 46.41 for Gemini-1.5-Pro and 48.89 for GPT-4o, essentially level with long context, while consuming 38.39% and 61.40% of the long-context tokens respectively. The authors report cost reductions of 65% for Gemini-1.5-Pro and 39% for GPT-4o.

A second paper reached a stronger pro-RAG conclusion. NVIDIA’s “In Defense of RAG in the Era of Long-Context Language Models” (arXiv 2409.01666) introduced order-preserving RAG (OP-RAG), which keeps retrieved chunks in their original document order rather than by score. On ∞Bench En.QA, OP-RAG with Llama3.1-70B reached 47.25 F1 using 48K tokens. The same model fed roughly 117K tokens without retrieval scored 34.26 F1. GPT-4o with 117K tokens scored 32.36, and Gemini-1.5-Pro with 196K tokens scored 43.08. A tuned retrieval pipeline beat brute force with less than half the input.

Neither paper supports “RAG is dead” as a reading of the rag vs long context evidence. The first shows long context is a valuable escalation path; the second shows it is not even the best default. The honest summary is that long context removed RAG’s monopoly, not its usefulness.

Claim 2: “Models use all 1M tokens equally well” — False

Of all the rag vs long context claims, this is the one with the most evidence against it, accumulated over three years. Liu et al., “Lost in the Middle” (arXiv 2307.03172, TACL), placed the answer document at different positions among 20 retrieved documents. GPT-3.5-Turbo scored 75.8% when the answer was first but 53.8% when it sat at position 10. Its closed-book score, with no documents at all, was 56.1%. In the worst position, adding the right document made the model slightly worse than giving it nothing.

RULER from NVIDIA (Hsieh et al., arXiv 2404.06654, COLM 2024) extended needle tests with multiple needles, multi-hop variable tracing and aggregation. Across 17 long-context models and 13 tasks, nearly all models scored near-perfectly on vanilla needle retrieval. However, only about half of the models that claimed 32K or more maintained satisfactory performance at 32K. The paper popularised the idea of an “effective” context length that is often far below the advertised one.

NoLiMa (Modarressi et al., arXiv 2502.05167, LMU Munich and Adobe Research) removed the literal word overlap between question and needle, so the model had to make a one-hop associative inference. Of 13 models claiming at least 128K tokens, 11 fell to half or less of their short-context score by 32K. GPT-4o, the best performer, dropped from 99.3% to 69.7%. Defining effective length as the longest length that keeps 85% of the baseline, most models managed only about 2K tokens; GPT-4o reached 8K.

Chroma’s July 2025 technical report, “Context Rot: How Increasing Input Tokens Impacts LLM Performance” by Kelly Hong, Anton Troynikov and Jeff Huber, tested 18 models including Claude Opus 4, GPT-4.1, Gemini 2.5 Pro and Qwen3-235B-A22B. Performance degraded as input grew even on simple tasks. A single semantically similar distractor hurt accuracy, and lower question-needle similarity made the decline steeper. On LongMemEval, every model scored significantly higher with a focused prompt of about 300 tokens than with the full prompt of about 113K tokens containing the same answer. That is the cleanest real-workload argument for retrieval in the rag vs long context debate: the same information, presented compactly, gets better answers.

Claim 3: “Long context is cheaper than retrieval” — Misleading

The kernel of truth is engineering cost. A long-context prototype has no chunker, no embedding job, no index to keep fresh and no reranker to tune. For a small, stable corpus and modest query volume, that simplicity can be the cheapest option in total cost of ownership. Anthropic itself made this point in its September 2024 Contextual Retrieval post: if a knowledge base is under 200,000 tokens, about 500 pages, you can skip RAG and put it all in the prompt.

The rag vs long context cost claim turns misleading when it is generalised to inference cost at scale. Every API we checked bills input per token, so per-query cost is linear in prompt size. A 5,000-token RAG prompt and an 800,000-token stuffed prompt differ by a factor of 160 in input tokens before any discount. Prompt caching changes the multiplier, not the shape, and the next section works the numbers. Compute behaves even worse than price: self-attention prefill grows quadratically with sequence length, and KV-cache memory grows linearly. Providers absorb some of that in flat pricing, but they still pass part of it through as long-prompt tiers, rate limits and latency.

Claim 4: “RAG is always more accurate” — Mixed

In the rag vs long context argument, the RAG camp overclaims too. In Self-Route’s comparison, long context beat a plain RAG pipeline on average for both strong models, by double-digit points. The paper’s failure analysis explains why: RAG struggled with multi-step questions where the next lookup depends on the previous answer, with general or implicit questions that need an understanding of the whole text, and with long, complex queries that the retriever encodes poorly. No reranker fixes a question like “what is the overall tone of this contract’s indemnity regime?”

LongBench v2 (arXiv 2412.15204) reinforces that some tasks are inherently whole-context. Its 503 multiple-choice questions span contexts from 8K to 2M words and stress deep reasoning. Human experts working under a 15-minute limit scored 53.7%. The best model answering directly scored 50.1%, while o1-preview with extended reasoning reached 57.7%. These are tasks where you cannot pre-select five chunks because the evidence is distributed. Accuracy in rag vs long context comparisons is therefore workload-dependent, and the right verdict is Mixed.

Worked Numbers: Cost, Prefill Latency and KV-Cache Memory

A long context llm vs rag decision is ultimately a spreadsheet. This section works three numbers you can recompute for your own workload: dollars per query, time to first token, and GPU memory. Prices are from official pricing pages as fetched on 16 November 2026. The workload itself is illustrative.

Long context LLM vs RAG token flow comparing a five thousand token retrieval prompt with an eight hundred thousand token full-corpus prompt
Figure 2: Token flow per query. The RAG path sends a few thousand tokens through prefill; the long-context path pushes the whole corpus through prefill and the KV cache on every uncached request.

Figure 2 contrasts the two request paths at the heart of rag vs long context. On the left, hybrid search and a reranker cut a large corpus down to eight chunks, so the model prefills about 5,000 tokens. On the right, the entire corpus, here about 800,000 tokens, becomes the prompt. The answer step is identical, so the difference lives entirely in prefill compute, KV-cache memory and input billing.

Dollars per query: an illustrative internal-docs assistant

Assume an internal documentation assistant over a corpus of about 800,000 tokens, answering 10,000 questions per day with about 500 output tokens per answer. The RAG prompt carries eight 500-token chunks plus 1,000 tokens of instructions and question, about 5,000 tokens. These workload figures are illustrative assumptions; the prices are real list prices.

Scenario (illustrative workload) Input tokens per query Input $ per query Input $ per day
Claude Sonnet 5.5, RAG 5,000 0.010 100
Claude Sonnet 5.5, full corpus, uncached 800,000 1.60 16,000
Claude Sonnet 5.5, full corpus, cache hit 800,000 0.08 800
Gemini 3.1 Pro Preview, RAG 5,000 0.010 100
Gemini 3.1 Pro Preview, full corpus, uncached 800,000 3.20 32,000
Gemini 3.1 Pro Preview, full corpus, cached 800,000 0.32 3,200 plus storage

The inputs behind the table: Anthropic’s pricing page lists Claude Sonnet 5.5 at $2 per million input tokens, $10 per million output tokens, $2.50 for a 5-minute cache write and $0.10 for a cache read, and states that the full 1M window is billed at standard rates. Google’s Gemini API pricing page lists Gemini 3.1 Pro Preview at $2.00 per million input tokens for prompts up to 200K and $4.00 above, with context-cache reads at $0.40 above 200K and storage at $4.50 per million tokens per hour. Output costs about $0.005 per answer on Sonnet 5.5 for all scenarios, so it does not change the comparison.

Three rag vs long context cost observations follow. First, uncached full-corpus prompting costs 160 times more input than RAG on Sonnet 5.5 and 320 times more on Gemini 3.1 Pro Preview, because the long prompt crosses the 200K tier. Second, caching is the great equaliser: a cache hit brings Sonnet 5.5 down to $0.08 per query, only eight times the RAG input bill. Third, cache storage and writes are not free. Holding 800K tokens in Gemini’s explicit cache costs about $3.60 per hour, roughly $86 per day, and Anthropic charges a cache write whenever the 5-minute or 1-hour cache lapses. The RAG column also omits embedding, vector-store and reranker costs, which for a corpus this size are usually small but not zero. Our November 2026 LLM API pricing roundup and the prompt caching economics and break-even guide cover these levers in depth.

Prefill latency: back-of-envelope FLOPs

Price hides physics, so it helps to estimate the compute a provider must spend. Take an open model with published dimensions, Llama 3.1 70B: 80 layers and a model width of 8,192. A common approximation for prefill is about 2 × parameters × tokens for the dense matrix multiplies, plus about 2 × layers × width × tokens² for causal attention scores and value mixing. These are textbook estimates that ignore kernels and overheads, so treat the results as order-of-magnitude.

The dense term is about 1.4 × 10¹¹ FLOPs per token. The attention term is about 1.31 × 10⁶ × N² FLOPs for N tokens. The two are equal near N ≈ 107,000 tokens. Below that, prefill is dominated by weights and scales linearly; above it, attention dominates and cost grows quadratically.

Prompt length (illustrative) Dense FLOPs Attention FLOPs Total Time at 8 × 495 TFLOPS effective
5,000 tokens 7.0 × 10¹⁴ 3.3 × 10¹³ 7.3 × 10¹⁴ about 0.2 s
128,000 tokens 1.8 × 10¹⁶ 2.1 × 10¹⁶ 3.9 × 10¹⁶ about 10 s
1,000,000 tokens (hypothetical) 1.4 × 10¹⁷ 1.3 × 10¹⁸ 1.45 × 10¹⁸ about 6 min

The time column assumes eight H100-class GPUs, each delivering half of a roughly 989 TFLOPS dense BF16 peak, which is an illustrative utilisation figure. Llama 3.1 70B itself supports 128K tokens, so the 1M row is a hypothetical extrapolation to show scaling, not a measurement. Production frontier models use sparse or compressed attention, context parallelism and specialised hardware, so their real latency is lower. The shape still holds: a 200-fold longer prompt costs far more than 200 times the compute once attention dominates. Techniques such as native sparse attention and DeepSeek sparse attention exist precisely to bend this curve.

For interactive products, latency is often the decisive number in rag vs long context trade-offs. Our RAG latency budget guide allocates a few hundred milliseconds to retrieval and reranking. That is a rounding error next to tens of seconds of uncached prefill.

KV-cache memory: why serving long context is hard

Every prefilled token leaves keys and values in GPU memory for the rest of the generation. For Llama 3.1 70B with grouped-query attention (8 KV heads of dimension 128 across 80 layers) in BF16, each token needs 2 × 80 × 8 × 128 × 2 bytes, or 327,680 bytes, about 320 KiB. A 5,000-token RAG prompt therefore holds about 1.6 GB of KV cache. A 128K prompt holds about 42 GB. A hypothetical 1M-token prompt would need about 328 GB, more than four 80 GB GPUs for one request’s cache alone, before weights.

This is why long-context serving pushes providers towards KV compression such as multi-head latent attention versus GQA and MQA, KV offload to CPU or SSD, and aggressive prefix reuse. It is also why long-context requests typically get lower rate limits and less predictable queueing than short ones, even when the per-token price is flat.

Hybrid Patterns: Retrieve, Then Use Long Context Wisely

If neither side of the rag vs long context split wins everywhere, the engineering question becomes how to combine them. The 2024-2026 evidence supports three hybrid patterns, and they compose.

Hybrid RAG vs long context sequence showing a router that tries retrieval first, then escalates to a cached long-context prefix
Figure 3: A Self-Route style hybrid. Retrieval answers most questions cheaply; only questions the model marks unanswerable escalate to a long-context call that reuses a cached document prefix.

Figure 3 shows the request sequence. The router sends the question to hybrid search and passes the top chunks to the model with an instruction to answer or reply UNANSWERABLE. Most questions stop there. For the remainder, the router builds a long prompt whose stable part, the document set, is identical across requests, so the provider’s prefix cache or the self-hosted engine’s KV cache can skip most prefill. Only the question is new.

Pattern 1: Self-Route escalation

Self-Route is the simplest hybrid to deploy because it needs no new model. Add an explicit abstention instruction to the RAG prompt, detect the abstention string, and retry with a larger context. In the original study, the retrieval step alone handled 76.78% of queries for Gemini-1.5-Pro and 57.36% for GPT-4o. The key engineering requirement is a model that abstains honestly. Chroma’s report noted that Claude models were the most likely to abstain under uncertainty while GPT models hallucinated more often, so abstention behaviour should be measured per model rather than assumed.

Pattern 2: Retrieve documents, not chunks, then read in long context

Classic RAG retrieves 300-to-500-token chunks. Long windows make it practical to retrieve whole documents or sections instead: use retrieval to pick the five contracts or three manuals that matter, then give the model all of them, in original order. This preserves the cross-references that chunking destroys while keeping the prompt to tens of thousands of tokens instead of a million. OP-RAG’s result, that order-preserving retrieval at 16K to 48K tokens beat 117K-token brute force, is evidence for this middle ground. The paper also found an inverted-U curve: quality rose as more chunks were added, then fell as distractors accumulated. The sweet spot is workload-specific and must be measured.

Pattern 3: Cache the stable prefix

Whenever a long document set is queried repeatedly, put it at the front of the prompt and cache it. Anthropic’s cache read for Sonnet 5.5 is 0.05 times the base input price, and Gemini offers explicit caches with hourly storage. Self-hosted stacks get the same effect from KV reuse, which we compare in LLM prefix caching with RadixAttention versus vLLM automatic prefix caching. Caching attacks both cost and time to first token. However, it does nothing for accuracy: a cached 800K-token prompt suffers exactly the same context rot as an uncached one.

A decision matrix for rag vs long context

Workload Pure RAG Pure long context Hybrid Why
Support bot over 50K-page knowledge base, high QPS Strong Weak Strong Corpus exceeds any window; cost scales with QPS
Analyst Q&A on one 300-page contract Weak Strong Strong Evidence is distributed; corpus fits comfortably
Codebase assistant, 2M-token repo Medium Weak Strong Retrieve files by symbol graph, then read whole files
Small, stable policy set under 200K tokens, low volume Medium Strong Medium Simplicity wins; caching optional
Compliance answers needing citations Strong Medium Strong Retrieval gives natural provenance per chunk
Rapidly changing data, minutes-fresh Strong Weak Medium Re-caching a huge prefix on every change is costly

The rag vs long context matrix is an informed judgement, not a benchmark result. Its last two rows deserve emphasis because they rarely appear in the debate. Retrieval returns identifiable spans, which makes citation and attribution far easier, as our guide to verifiable citations for RAG answers explains. Retrieval indexes also update incrementally, whereas any change to a cached long prefix invalidates everything after it.

Decision flowchart for choosing RAG, long context, cached long context or retrieve-then-read hybrids based on corpus size, volume and task type
Figure 4: A decision procedure for the rag vs long context choice, starting from corpus size and ending in an evaluation loop.

Figure 4 turns the matrix into a procedure. Start with corpus size and stability. A small, stable corpus can go straight to long context, with caching when volume is high. Larger corpora go to RAG unless the questions require whole-document reasoning, in which case retrieve whole documents and read them in a long window. If the evidence cannot be scoped even to a few documents, use agentic or map-reduce passes over chunks rather than one giant prompt. Every branch ends in evaluation, because each published study shows that results vary by model and dataset.

Trade-offs, Gotchas, and What Goes Wrong

Benchmark transfer is weak. This is the most common rag vs long context mistake. Needle tests overstate long-context ability because the needle usually shares words with the question. NoLiMa’s collapse to roughly 2K effective tokens for most models came from removing that overlap, and real user questions rarely match document wording. Conversely, RULER-style synthetic tasks may understate a model on coherent prose. Chroma even found that shuffled haystacks produced better results than logically structured ones, which the authors called counterintuitive. Never pick an architecture from a vendor’s needle chart.

Model generations move the curve. The Databricks study (arXiv 2411.03538) ran over 2,000 experiments on 13 models with contexts from 2K to 125K tokens. Llama-3.1-405B began degrading after 32K and GPT-4-0125-preview after 64K, while GPT-4o, Claude 3.5 Sonnet and GPT-4o-mini showed little or no decline. Newer models are better, but the 2025 Context Rot results show degradation persisting in later model generations. Re-test when you upgrade; do not assume.

Long context fails in new ways. The Databricks authors found failure modes that RAG prompts rarely trigger: Claude 3 Sonnet’s copyright refusals rose from 3.7% at 16K to 49.5% at 64K, and DBRX-instruct increasingly summarised the context instead of answering, failing 50.4% of the time at 32K. Those specific numbers are for 2024 models, but the lesson is current: long prompts change behaviour, not only accuracy.

RAG fails quietly. A retriever that misses the right chunk produces a fluent wrong answer with no error. Anthropic’s Contextual Retrieval work reported that contextual embeddings plus contextual BM25 cut top-20 retrieval failures by 49%, and 67% with reranking, which tells you how large the baseline failure rate can be. Measure recall with a golden set, as in our RAG retrieval evaluation pipeline.

Security and privacy scale with the prompt. Stuffing a whole corpus exposes every document to every user query, so access control must happen before the prompt is built. Retrieval naturally filters by permissions; long context needs per-user corpora or careful prompt assembly. A larger prompt is also a larger surface for indirect prompt injection.

Caching is fragile on both sides of rag vs long context. Prefix caches match exact token prefixes. Inserting a timestamp, a user name or a reordered document at the top invalidates everything. Keep stable content first and volatile content last.

Practical Recommendations

Treat the rag vs long context question as a per-workload measurement, not an ideology. Start by sizing the corpus in tokens with the provider’s own tokenizer, because the 200K threshold suggested by Anthropic and the 200K pricing tier at Google both turn on that number. If the corpus is small and stable, a long-context prototype is the fastest way to learn what users ask. Use the logs from that prototype to build your evaluation set.

For anything larger, or anything with real query volume, default to hybrid search plus reranking and add a Self-Route style escalation. Retrieve whole documents or sections, keep them in original order, and measure where the inverted-U peak sits for your model. Put stable context at the front so prefix caching works, and budget explicitly for cache writes and storage. Finally, rerun your evaluation on every model upgrade, because the published curves move from generation to generation and sometimes regress.

A short rag vs long context checklist:

  • [ ] Count corpus tokens and growth rate; note any 100K or 200K pricing thresholds.
  • [ ] Build a golden set of at least a few hundred real questions with known answers and sources.
  • [ ] Measure retrieval recall at k before tuning generation.
  • [ ] Compare RAG, full context and Self-Route on the same golden set, with cost and time to first token.
  • [ ] Test answer accuracy with the evidence placed early, middle and late in long prompts.
  • [ ] Enforce access control before prompt assembly in both paths.
  • [ ] Structure prompts as stable prefix plus volatile suffix for caching.
  • [ ] Re-evaluate on every model or pricing change.

Frequently Asked Questions

Is RAG dead in 2026?

No. Long-context models removed the hard requirement for retrieval when a corpus fits in the window, but they did not remove the reasons to retrieve. Published benchmarks such as RULER, NoLiMa and Chroma’s Context Rot report show accuracy degrading well before advertised limits. Input billing is per token, so sending a whole corpus costs far more per query than a few retrieved chunks. Research comparing the two, including Self-Route and OP-RAG, found that hybrids of retrieval plus long context match or beat brute-force prompting while using a fraction of the tokens.

What is context rot?

Context rot is the term popularised by Chroma’s July 2025 technical report for the decline in model reliability as input length grows, even when the task stays simple. Across 18 models, Chroma found that distractors, low similarity between question and answer, and long irrelevant context all reduced accuracy. On LongMemEval, models answered markedly better from a focused prompt of about 300 tokens than from a full prompt of about 113,000 tokens containing the same answer. It is closely related to the earlier Lost in the Middle finding about positional sensitivity.

Is a million token context window useful at all?

Yes, for specific jobs. A large window lets a model read a whole contract, a long transcript or a set of related files and reason across them, which chunked retrieval handles poorly. It also simplifies prototypes and lets retrieval return whole documents rather than fragments. The mistake is treating the window as a database. Effective context is usually much shorter than the advertised maximum, so use the window to read a well-chosen subset deeply, and measure accuracy at the lengths you actually send.

Is long context cheaper than RAG?

Usually not at scale, which is the core rag vs long context cost lesson. API input is billed per token, so an 800,000-token prompt costs 160 times the input of a 5,000-token RAG prompt at the same rate, and some providers charge a higher tier above 200K tokens. Prompt caching cuts repeated-prefix cost sharply, bringing the gap down to single-digit multiples in our illustrative example, but adds write and storage charges and does not improve accuracy. Long context can still be cheaper overall for small, stable corpora with low query volume, where avoiding retrieval infrastructure dominates.

When does long context beat RAG on accuracy?

Long context tends to win when the answer depends on evidence spread across a document, on multi-step reasoning where each step depends on the previous one, or on implicit questions about tone, structure or overall meaning. The Self-Route study identified exactly these as the main RAG failure categories. Strong models showed the largest long-context advantage in that study. When questions target a specific fact in a large corpus, a well-tuned retriever with reranking and order-preserving context assembly often matches or beats long context, as the OP-RAG results showed.

How do I choose between RAG and long context for my app?

Measure rather than guess; rag vs long context has no universal winner. Count your corpus in tokens, estimate queries per day, and build a golden set of real questions. Run three variants on it: tuned RAG, full long context, and a Self-Route hybrid that escalates only unanswerable questions. Record accuracy, cost per query and time to first token for each. Small, stable corpora often favour long context with caching. Large, changing or permission-sensitive corpora favour RAG. Whole-document reasoning favours retrieving complete documents and reading them in a long window.

Further Reading

By Riju — about

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *