Fact-Check: Do 1M-Token Context Windows Actually Work? Effective Context Length, Context Rot and the Evidence (2026)

Every flagship model page now says the same thing: one million tokens. OpenAI, Anthropic, Google, DeepSeek and Mistral all ship windows at or near that size, and the marketing implication is simple. Paste in the whole codebase, the whole contract archive, the whole quarter of support tickets, and the model will reason over all of it. The research record says something narrower. A model’s effective context length, the longest input at which it still performs a given task reliably, is usually far shorter than its advertised window, and it shrinks as the task gets harder.

This matters now because the window race has stalled near 1M tokens while prices and agent workloads push teams to stuff more into every prompt. Misreading the spec is expensive: you pay for every token and still get silent errors.

You will leave with a verdict on seven common claims, the mechanisms behind context rot, worked KV-cache and prefill cost math, and a protocol for measuring effective length on your own task.

What this covers: the viral claims and their verdicts, RULER, Lost in the Middle, NoLiMa, Chroma’s Context Rot report, HELMET, LongBench v2 and Fiction.LiveBench, attention dilution, RoPE scaling and YaRN, KV-cache and prefill cost, and when long context beats retrieval.

Context and Background

Context windows grew by roughly three orders of magnitude in three years. GPT-3.5 shipped with about 4K tokens. By 2024 Google’s Gemini 1.5 Pro offered 1M tokens and later 2M, and by late 2026 the closed frontier has converged on roughly one million. I checked the vendor documentation on 14 November 2026: OpenAI lists GPT-6 Astra at 1,050,000 tokens of context and 128,000 output tokens; Anthropic’s models overview lists 1M-token windows and 128K max output for its current Claude models; and Google’s Gemini 4 Argon page lists a unified input-plus-output limit of 1,048,576 tokens. Among open weights, Meta’s Llama 4 Scout model card advertises 10M tokens, the outlier in the field.

The evaluation community spent the same three years discovering that a window is a capacity, not a capability. The first popular test, needle in a haystack (NIAH), hides one sentence in filler text and asks the model to retrieve it. Frontier models now ace it, which is exactly why it stopped being informative. Subsequent benchmarks made the needle harder to find, added distractors, required aggregation or multi-hop tracing, or abandoned synthetic haystacks for real documents.

We covered the mechanics of the most cited synthetic suite in our long-context LLM benchmarks guide to RULER and effective context, and we will not repeat its task-by-task walk-through here. This post takes a different angle. It treats the popular claims about 1M-token windows as a fact-check: state the claim, weigh the primary evidence, issue a verdict, and explain the mechanism.

My thesis is that effective context length is not a property of a model; it is a property of a model, a task and a distractor profile together. A single number on a leaderboard is a lower bound for easy lookups and an upper bound for hard reasoning. Teams that internalise this stop asking “how big is the window?” and start asking “at what length does my task fall over?”, which is the only question that predicts production behaviour.

The Verdicts: Seven Claims About 1M-Token Context Windows

Direct answer: No, a 1M-token context window does not mean a model uses one million tokens equally well. Peer-reviewed benchmarks show accuracy falling well before the advertised limit, often by 32K tokens on tasks that require inference rather than literal matching. Effective context length depends on task difficulty, distractors and position, so it must be measured per workload.

Figure 1: Where a long prompt loses information. Each stage between the raw tokens and the answer adds a failure mode, and the failures compound as input length grows.

Read the figure left to right. Tokens first receive positional information, usually rotary position embeddings (RoPE) that may have been rescaled to reach the long window. Attention then distributes a fixed probability budget over every key. The relevant span must win that competition against semantically similar distractors, the model must retrieve it, and finally it must combine retrieved facts. The dotted branches mark where each stage fails, and the later stages are the ones simple benchmarks never exercise.

Here are the seven claims and verdicts at a glance. The sections that follow give the evidence for each.

#	Claim	Verdict
1	“A 1M-token window means the model reads all of it equally well.”	False
2	“Perfect needle-in-a-haystack scores prove long-context ability.”	Misleading
3	“Models always lose information in the middle of the context.”	Misleading
4	“Context rot only kicks in near the window limit.”	False
5	“Effective context length is a fixed number you can look up.”	Misleading
6	“Filling a bigger window costs proportionally more, nothing worse.”	False
7	“Long context makes RAG obsolete.”	False

Claim 1: “A 1M-token window means the model reads all of it equally well” — False

The strongest primary evidence comes from RULER, by Hsieh and colleagues at NVIDIA (arXiv 2404.06654). RULER evaluated 17 long-context models on 13 tasks spanning retrieval, multi-hop variable tracking, aggregation and question answering. Its abstract reports that although all the models claimed 32K tokens or more, only half maintained satisfactory performance at 32K.

RULER defines effective length concretely. The threshold is Llama-2-7B’s average score at 4K tokens, 85.6%, and a model’s effective length is the longest tested length at which it stays above that bar. Under that definition GPT-4 (claimed 128K) reached 64K, Llama 3.1 70B (claimed 128K) reached 64K, Qwen2-72B (claimed 128K) reached 32K and Yi-34B (claimed 200K) reached 32K. Gemini 1.5 Pro, claimed at 1M, exceeded the longest length RULER tested, so it was reported as greater than 128K. Note what that last row means: nobody in that study verified 1M-token performance, they verified 128K.

Those models are now one or two generations old, and current frontier models do better. But no vendor I could find publishes a RULER-style effective length at the full 1M window for its current flagship, so the claim of uniform quality at 1M remains unsupported. Absence of published evidence is not proof of failure, yet the burden belongs on the claim.

Claim 2: “Perfect needle-in-a-haystack scores prove long-context ability” — Misleading

Needle tests prove one thing well: that a model can find a sentence that shares words with the question. NoLiMa, by Modarressi and colleagues (arXiv 2502.05167, ICML 2025), removed that shortcut. Its needles and questions share minimal lexical overlap, so the model must make a latent association, for example knowing that a character who “lives next to the Semperoper” lives in Dresden.

The results are stark. Of 13 models that each claimed at least 128K tokens, 11 fell to half or less of their short-context baseline at 32K. GPT-4o, the best performer in the paper, went from 99.3% at short lengths to 69.7% at 32K. Llama 3.3 70B fell from 97.3% to 42.7%. NoLiMa uses an 85%-of-baseline threshold for effective length, and under it Llama 3.1 70B’s effective length was only 2K tokens. Reasoning models did not escape: on the hardest subset, o1, o3-mini and a DeepSeek-R1 distill all dropped below 50% of baseline at 32K.

Princeton’s HELMET benchmark (project page) reached a compatible conclusion from the other direction. After evaluating 59 models across seven application categories up to 128K tokens, the authors found that synthetic NIAH results do not reliably predict downstream performance, partly because most models saturate them. A perfect needle score is necessary, not sufficient. That is why the verdict is “misleading” rather than “false”.

Claim 3: “Models always lose information in the middle” — Misleading

“Lost in the Middle” by Liu and colleagues (arXiv 2307.03172, published in TACL in 2024) documented a U-shaped curve on multi-document QA and key-value retrieval. Accuracy was highest when the relevant passage sat at the start or end of the input and dropped when it sat in the middle, including for models explicitly built for long contexts.

The finding is real and widely replicated for its era, but “always” overreaches. Chroma’s July 2025 Context Rot report, which tested 18 models from Anthropic, OpenAI, Google and Alibaba, found no notable effect of needle position in its needle-question similarity experiment. Their degradation came from length and semantic difficulty instead. The honest reading is that position bias exists, varies by model generation and task, and is not the dominant failure mode in current frontier models on simple retrieval. Putting the instruction and the most important material at the end of the prompt remains cheap insurance.

Claim 4: “Context rot only kicks in near the window limit” — False

Context rot is Chroma’s name for the steady, often non-linear decline in output quality as input length grows. The key word is steady. In the Chroma report (trychroma.com/research/context-rot, by Kelly Hong, Anton Troynikov and Jeff Huber), degradation appeared on tasks as simple as reproducing a sequence of repeated words, and it appeared across the length range rather than at a cliff near the maximum. On that repeated-words task some Gemini models began emitting random words at around 500 to 750 words of input, nowhere near any window limit.

The most practically relevant Chroma experiment used LongMemEval, a conversational memory benchmark. The authors compared a focused prompt of roughly 300 tokens, containing only the relevant conversation turns, with the full prompt of roughly 113K tokens. Models did significantly better on the focused version across 306 prompts. The 113K prompt is about 11% of a 1M window, so a model that degrades there is not failing because it ran out of room.

NoLiMa tells the same story with numbers: the big drops occur by 32K, about 3% of a 1M window. If your mental model is “quality is flat until the limit”, replace it with “quality declines from the first few thousand tokens, at a rate set by task difficulty”.

Claim 5: “Effective context length is a fixed number you can look up” — Misleading

Each benchmark defines effective length differently, so the same model gets different numbers. RULER uses an absolute threshold (85.6%, Llama-2-7B at 4K). NoLiMa uses a relative threshold (85% of the model’s own short-context score). Under RULER, Llama 3.1 70B’s effective length was 64K. Under NoLiMa it was 2K. Both are correct measurements of different things: RULER mixes literal retrieval with harder tasks, while NoLiMa isolates latent association.

Chroma’s experiments add two more variables that shift the number. Distractors, passages that are topically similar to the needle but do not answer the question, hurt performance, and four distractors hurt more than one. Haystack structure mattered in a surprising way: across all 18 models, shuffled haystacks produced better results than logically coherent ones. The authors call that result surprising and suggest it may relate to attention patterns; it is a reported observation, not a settled mechanism.

So treat any single “effective context length” figure as conditional. It is valid for that benchmark’s task mix, distractor density and threshold. Your contract-review workload with near-duplicate clauses is a high-distractor task and will likely sit below the published number.

Claim 6: “Filling a bigger window costs proportionally more, nothing worse” — False

Two costs are worse than linear or come with step changes. First, compute for the attention part of prefill grows with the square of input length, so the attention share of a 1M-token prefill is roughly 64 times that of a 128K one, not 8 times. Second, vendors price long prompts in tiers. OpenAI’s GPT-6 Astra documentation states that prompts above 272K input tokens are charged at twice the standard input and cache rates and 1.5 times the output rate, applied to the entire request. I verified that tier on the model page on the date of writing; check your provider’s current page before you budget.

There is also a latency cost the price sheet hides. Time to first token is dominated by prefill, and a fresh 1M-token prompt cannot start generating until every token has passed through every layer. Prefix caching helps when the same prefix repeats, which we cover in our prefix caching comparison of RadixAttention and vLLM, but the first request always pays in full. The worked numbers are in the next section.

Claim 7: “Long context makes RAG obsolete” — False

The evidence above cuts against this directly. If accuracy falls with length and distractors, then a retrieval step that removes irrelevant material is not a workaround for small windows; it is a quality control. Chroma’s focused-versus-full LongMemEval result is effectively a controlled experiment in favour of retrieval: same model, same question, less irrelevant text, better answers.

The claim fails for economics too. A corpus of 50 million tokens does not fit in any production window, and re-sending 1M tokens per query multiplies input spend by the ratio of corpus to retrieved context. Long context does change the RAG design space: it lets you retrieve whole documents instead of 512-token chunks and lets the model reason across them. Our fine-tuning versus RAG versus long context decision guide covers that trade in depth. The right verdict is that long context makes naive chunk-level RAG less necessary for medium corpora, not that retrieval disappears.

Deeper Analysis: Why Effective Context Length Falls Short of the Window

The verdicts describe what happens. This section explains why, with the evaluation landscape first and then three mechanisms: attention dilution, positional extrapolation, and the memory and compute economics that shape how models are trained and served.

Figure 2: The long-context evaluation ladder. Each rung removes a shortcut the previous one allowed, and each reports a shorter effective context length for the same model.

The ladder explains why marketing and research disagree. Vendors tend to cite the bottom rung, single-needle recall, where frontier models score near perfectly. Researchers climb higher. RULER adds multi-key retrieval, variable tracing and aggregation. NoLiMa removes lexical overlap. Chroma adds distractors and varies haystack structure. HELMET and LongBench v2 use realistic tasks such as summarisation, re-ranking, citation generation and expert multiple-choice questions over long documents. Fiction.LiveBench asks comprehension questions about stories that require tracking chronology, character knowledge and implied information.

LongBench v2 is a useful calibration point because it includes a human baseline. Its 503 multiple-choice questions use contexts from about 8,000 to 2 million words. Human experts limited to 15 minutes per question scored 53.7%. The best model answering directly scored 50.1%, and o1-preview with extended reasoning reached 57.7% in the original paper (arXiv 2412.15204). Long-document reasoning is hard for people as well, and a model that matches a time-limited expert is impressive but far from “reads everything perfectly”.

Mechanism 1: attention dilution

Softmax attention gives each query a probability distribution over all keys that sums to one. If a relevant token’s raw score stays fixed while the number of competing keys grows, its share of attention shrinks. With n keys of roughly equal score, each gets about 1/n of the weight. Going from 8K to 1M tokens multiplies n by 128, so a signal that relied on modest score margins can disappear into the noise.

Models fight this by learning sharper score margins, and trained heads do develop retrieval behaviour that picks out exact matches. That is why literal needles survive at length. A latent association, the NoLiMa case, produces a weaker query-key match to begin with, so it loses the competition sooner. Distractors make it worse by adding keys with high scores that point to the wrong answer. This one mechanism explains three empirical findings at once: NIAH saturation, NoLiMa’s early collapse, and Chroma’s distractor sensitivity.

Dilution also explains why multi-hop and aggregation tasks fail first. Each hop is a retrieval with some probability of error. If single-hop retrieval at a given length succeeds 90% of the time, an illustrative three-hop chain succeeds roughly 0.9 cubed, about 73%, assuming independence. Aggregation tasks, such as counting frequent words, require attending to many positions at once, which directly opposes the sharpening that rescues single retrieval.

Mechanism 2: positional encoding extrapolation

Most current open models use rotary position embeddings, introduced in the RoFormer paper by Su and colleagues. RoPE rotates query and key vectors by angles proportional to position, using a spectrum of frequencies across dimensions, so the dot product depends on relative distance. A model trained on sequences up to length L has only seen rotation angles from that range. Beyond it, low-frequency dimensions produce angles the model never encountered, and attention behaves unpredictably.

Context extension methods rescale those angles. Position interpolation squeezes long positions into the trained range. NTK-aware scaling changes the RoPE base so high-frequency dimensions, which encode local order, are disturbed less than low-frequency ones. YaRN (Peng and colleagues, arXiv 2309.00071) interpolates per frequency band and adds an attention temperature correction, and it reported extending LLaMA-family models to 128K with far less fine-tuning data than earlier methods. Our guide to RoPE scaling, YaRN and NTK-aware interpolation walks through the maths.

The fact-check implication is direct. Several open models advertise a native window plus an extended window reached with YaRN or similar scaling. Extension makes the model stop failing catastrophically at long positions. It does not by itself teach the model to reason over a million tokens, because that requires long training documents with genuine long-range dependencies, which are scarce. Expect effective context length to sit closer to the native trained length than to the extended one, and verify it rather than assume.

Mechanism 3: KV-cache and prefill economics, worked

Serving cost shapes what vendors can afford to train and ship. Here is a worked example using the published architecture of Llama 3.1 70B: 80 layers, 8 key-value heads under grouped-query attention, head dimension 128. The numbers below are illustrative arithmetic, not measurements, and Llama 3.1 70B itself supports 128K tokens, so the 1M row is a hypothetical extension of the same architecture.

The KV cache stores one key and one value vector per layer per KV head for every token. Per token that is 2 × 80 layers × 8 heads × 128 dimensions × 2 bytes in BF16, which equals 327,680 bytes, or 320 KiB.

Context length	KV cache per sequence (BF16)	Prefill FLOPs, dense weights	Prefill FLOPs, attention	Illustrative prefill time on 8 GPUs
32K tokens	10 GiB	~4.6 × 10^15	~1.4 × 10^15	under 2 s
128K tokens	40 GiB	~1.8 × 10^16	~2.3 × 10^16	~13 s
1M tokens (hypothetical)	320 GiB	~1.5 × 10^17	~1.4 × 10^18	~8 min

Illustrative arithmetic. Dense FLOPs use 2 × parameters × tokens. Attention FLOPs use 2 × n² × 8,192 × 80 layers with a causal mask. Times assume 400 TFLOP/s of sustained throughput per GPU across 8 GPUs; real kernels, parallelism and hardware vary widely.

Three things jump out. First, at 128K the attention term already overtakes the dense weight term, and at 1M it is roughly ten times larger. Eight times the tokens costs about 39 times the prefill compute. Second, a single 1M-token sequence needs 320 GiB of KV cache, more than double the roughly 140 GB of BF16 weights; one long user can occupy most of an eight-GPU node. Third, every generated token must read that whole cache, so decode becomes memory-bandwidth bound in proportion to context length.

Vendors attack these costs with grouped-query or multi-head latent attention, sparse attention, KV quantisation, context parallelism and disaggregated serving. Those techniques make 1M-token windows commercially viable. Several of them, notably sparse attention patterns and aggressive KV compression, trade some fidelity for cost, which is another reason effective context length can trail the window. Our deep dives on KV-cache optimisation and multi-head latent attention versus GQA cover them, and ring attention and context parallelism explains how training even sees sequences this long.

Figure 3: The life of a 1M-token request. Prefill pays the quadratic attention cost once, the KV cache holds the result, and every decode step re-reads the full cache.

The sequence shows where engineering choices bite. A prefix-cache hit skips most of prefill, which is why long-context workloads that reuse the same document set are dramatically cheaper than ones that send new documents every time. A miss forces full prefill, and in disaggregated serving the KV cache must then move from prefill workers to decode workers, which at 320 GiB per sequence is a network transfer, not a footnote. The decode loop is the reason long-context output feels slower per token.

What the training data cannot easily teach

A model learns to use long context only from training examples whose answers depend on distant tokens. Natural web text rarely has dependencies spanning hundreds of thousands of tokens. Books, code repositories and long legal or scientific documents help, but genuine million-token dependencies are rare, so labs rely on synthetic long-context data and staged context extension late in training. Most labs do not disclose their long-context data mix in detail, so the exact recipe for any current flagship is undisclosed.

This is the quiet root cause behind every benchmark above. Architecture determines whether a model can attend across a million tokens. Training data determines whether it learned to. Window size measures the first; effective context length measures the second.

Trade-offs, Gotchas, and What Goes Wrong

Long context is still the right tool for some jobs, and debunking the hype should not swing teams to the opposite error. The decision depends on corpus size relative to your measured effective context length, whether the task needs whole-document reasoning, how often the same material is reused, and your latency budget.

Figure 4: Choosing between long context, retrieval and a hybrid. Every path ends in validation against your own task-matched probes, because published effective lengths do not transfer.

The flow encodes one rule: start from your measured effective context length, not the advertised window. Point lookups over a large corpus favour retrieval. Whole-document reasoning over material that fits favours long context, especially when prefix caching amortises the prefill across many queries. Everything else lands on a hybrid that retrieves broadly and then lets the model read generous, whole-document context.

The gotchas I see most often in practice:

Testing with your own needles. Teams validate long-context features by asking about a fact they planted verbatim. That reproduces the bottom rung of Figure 2 and will pass. Ask questions that require paraphrase, inference or combining two distant facts.
Ignoring near-duplicates. Contracts, logs, tickets and versioned specs are distractor-dense by nature. Chroma’s distractor results predict these corpora degrade faster than benchmark haystacks.
Coherent stuffing. Concatenating whole documents produces the coherent haystack that, surprisingly, performed worse than shuffled text in Chroma’s tests. Clear separators, document IDs and a short table of contents at the end are cheap mitigations; measure whether they help your model.
Silent abstention or confabulation. Chroma found model families differ: some Claude models abstained when uncertain, while GPT models hallucinated more often on distractor tasks. Your evaluation must score “I don’t know” separately from wrong answers.
Agent context accumulation. Coding and research agents append tool output turn after turn, so they cross 100K tokens without anyone deciding to. That is context rot by accretion, and it is why the context engineering for production LLM agents patterns of compaction and sub-agent isolation exist.
Price-tier cliffs. Crossing a long-context pricing threshold can reprice the whole request, not just the excess tokens. Track prompt length distributions, not averages.
Output budget confusion. Some providers count input and output against one unified limit. A 1,048,576-token unified window holding 1M tokens of input leaves little room for output.

Practical Recommendations

Treat the advertised window as an upper bound on what the API will accept, and measure effective context length for each workload before you design around it. The measurement does not need a research team. A small harness with task-matched probes at several lengths tells you where your curve bends.

The protocol I use has four steps. Build 50 to 100 questions from your real corpus whose answers require inference or combining facts, not verbatim lookup. Embed the answer-bearing passages at several depths inside real, distractor-rich filler from the same domain. Run each question at lengths such as 8K, 32K, 128K, 256K and the largest you would use. Define effective length as the longest length that stays above 85% of the model’s own short-context score, the NoLiMa convention, which keeps the threshold model-relative.

# Minimal effective-context-length harness (sketch; plug in your client and grader)
import random

LENGTHS = [8_000, 32_000, 128_000, 256_000]
THRESHOLD = 0.85  # fraction of short-context baseline, as in NoLiMa

def build_prompt(question, evidence, filler_docs, target_tokens, depth, count_tokens):
    docs, used = [], 0
    for d in random.sample(filler_docs, len(filler_docs)):
        if used + count_tokens(d) > target_tokens:
            break
        docs.append(d)
        used += count_tokens(d)
    docs.insert(int(len(docs) * depth), evidence)
    body = "\n\n".join(f"[DOC {i}]\n{d}" for i, d in enumerate(docs))
    return f"{body}\n\nQuestion: {question}\nAnswer briefly, or say UNKNOWN."

def effective_length(model_call, grade, items, filler_docs, count_tokens):
    scores = {}
    for n in [2_000] + LENGTHS:
        hits = 0
        for it in items:
            depth = random.choice([0.1, 0.5, 0.9])
            p = build_prompt(it["q"], it["evidence"], filler_docs, n, depth, count_tokens)
            hits += grade(model_call(p), it["answer"])
        scores[n] = hits / len(items)
    base = scores[2_000]
    ok = [n for n in LENGTHS if scores[n] >= THRESHOLD * base]
    return (max(ok) if ok else None), scores

Checklist before you ship a long-context feature:

[ ] Measured effective context length on task-matched, non-literal probes, at least three depths.
[ ] Scored abstentions separately from wrong answers.
[ ] Put instructions and the question after the documents, and tested the reverse.
[ ] Modelled cost with the provider’s current long-context pricing tier and your p95 prompt length.
[ ] Enabled prefix caching for reused document sets and checked hit rates.
[ ] Added a retrieval or compaction step for any path that can exceed your measured length.
[ ] Scheduled re-measurement on each model upgrade, since effective context length changes by version.

Frequently Asked Questions

What is effective context length?

Effective context length is the longest input at which a model still performs a specific task above a defined quality bar. RULER sets that bar at 85.6%, Llama-2-7B’s score at 4K tokens, while NoLiMa uses 85% of each model’s own short-context score. It differs from the context window, which is simply the maximum number of tokens the API accepts. Because it depends on task difficulty, distractors and the threshold chosen, the same model can show an effective length of 64K on one benchmark and 2K on another.

What is context rot in LLMs?

Context rot is the gradual decline in a language model’s output quality as its input grows longer, even when the task stays the same and the relevant information is present. Chroma popularised the term in a July 2025 report that tested 18 models and found degradation on simple tasks such as repeated-word copying and conversational memory. It is not a cliff at the window limit. It starts at modest lengths and accelerates with semantic difficulty and distractors, which is why trimming irrelevant context often improves answers.

Is the needle in a haystack test still useful?

It is useful as a smoke test and nearly useless as a quality signal for frontier models. Classic needle tests plant a sentence that shares words with the question, so models can match it lexically, and most current models score near perfectly. HELMET’s authors found that such synthetic recall does not reliably predict downstream task performance. Use needle tests to catch broken long-context configurations, such as a wrong RoPE scaling setting, then rely on non-literal, multi-hop and domain-specific probes to measure real capability.

Do 1 million token context windows actually work?

They work in the sense that the API accepts the tokens and the model can often retrieve explicit facts from anywhere in them. They do not deliver uniform reasoning quality across the full window. Published studies such as RULER and NoLiMa found substantial drops well before the advertised limit, often by 32K tokens on inference-heavy tasks. No major vendor I checked publishes an independently reproducible effective length at the full 1M window, so treat million-token claims as capacity, and measure quality on your own workload.

Is long context better than RAG?

Neither wins outright. Long context is better when the material fits within your measured effective context length, the task needs reasoning across whole documents, and the same corpus is queried repeatedly so prefix caching cuts cost. Retrieval is better for large or fast-changing corpora, point lookups and distractor-heavy collections, because removing irrelevant text improves accuracy. Many production systems use a hybrid: retrieve whole documents rather than small chunks, then let a long-context model reason over that focused set.

How much memory does a 1M-token context need?

It depends on the architecture. As a worked example, a Llama 3.1 70B-style model with 80 layers, 8 key-value heads and head dimension 128 stores 320 KiB of KV cache per token in BF16. At 1M tokens that is 320 GiB for a single sequence, more than twice the roughly 140 GB of model weights. Techniques such as multi-head latent attention, KV quantisation and sparse attention cut this substantially, which is how providers serve million-token windows economically.

Fact-Check: Do 1M-Token Context Windows Actually Work? Context Rot, Effective Context Length and What the Evidence Shows (2026)