RAG Chunking: Fixed-Size, Semantic, Late Chunking, BM25 Hybrid Search and Reranking (2026)

Most retrieval-augmented generation failures that get blamed on the language model are really retrieval failures, and most retrieval failures start one step earlier, at the moment a document is cut into pieces. RAG chunking decides what the index can ever return: a fact split across two chunks may be retrievable by neither, and a chunk that says “the company grew 3% over the previous quarter” with no company name embeds as noise.

The surprising part is how little the fashionable options buy you. Peer-reviewed and vendor evidence from 2024 to 2025 shows that expensive semantic splitters often fail to beat a well-tuned recursive splitter, while two cheaper moves, lexical-plus-dense hybrid retrieval and a reranker, reliably move recall and precision. Late chunking and contextual retrieval attack the real problem, lost context, but each has a cost and a failure mode.

You will leave with a mechanism-level comparison of the main chunking strategies, verified numbers from the primary sources, runnable Python for recursive splitting, late chunking, BM25 and reciprocal rank fusion, and a decision tree for choosing the cheapest stack that clears your recall target.

What this covers: why chunk boundaries cap retrieval quality, fixed-size versus semantic versus late chunking, how BM25 hybrid search and reranking compensate, and how to evaluate and choose.

Context and Background

A RAG system has two halves. The ingestion half parses documents, splits them into chunks, embeds each chunk, and stores text and vectors. The query half embeds the question, retrieves the nearest chunks, optionally reranks them, and pastes the survivors into the model prompt. Chunking sits at the boundary and is the only step that is irreversible without re-indexing: once you have embedded 40 million chunks at a given size, changing your mind means paying the embedding bill again.

Figure 1: Where chunking sits in a RAG pipeline. The chunker feeds both the dense index and the lexical BM25 index, so one boundary decision affects every retrieval path downstream.

Figure 1 shows the dependency. The same chunk text is embedded for dense retrieval and tokenized for BM25, and both retrievers feed one candidate pool. A bad boundary therefore degrades both paths at once, which is why chunking deserves more evaluation time than it usually gets.

Why not skip chunking and use whole documents? Three constraints force the issue. Embedding models have input limits (many popular ones accept 512 tokens, and long-context models such as the Jina v3 family accept 8,192). A single vector for a 30-page document averages away every specific fact. And you pay for every token you stuff into the generation prompt, while long contexts degrade the model’s use of the middle. Anthropic notes that a knowledge base under roughly 200,000 tokens, about 500 pages, can simply be placed in the prompt with caching and no retrieval at all, which is worth checking before you build anything (Anthropic, Contextual Retrieval).

The incumbent approach is still the simple one: split by characters or tokens with a recursive splitter that prefers paragraph, then sentence, then word boundaries, usually at 200 to 1,000 tokens, with some overlap. LangChain’s RecursiveCharacterTextSplitter and LlamaIndex’s SentenceSplitter are the common implementations. Challengers fall into three families: semantic chunkers that place boundaries where embedding similarity between adjacent sentences drops; LLM-assisted chunkers that ask a model to propose boundaries; and context-restoring methods, namely contextual retrieval and late chunking, that keep the boundaries but repair the lost context.

A second axis matters as much as the splitter: how you retrieve. Dense embeddings capture paraphrase but blur exact tokens such as part numbers, error codes and names. BM25, the classic lexical ranking function, does the opposite. Combining them is the subject of our own pgvector versus Qdrant hybrid search ADR, and the latency side of the same stack is covered in the RAG latency budget guide.

One framing helps for the rest of the article. Every chunking strategy trades three quantities: boundary quality (does a chunk hold a complete idea), context retention (does the chunk know what document and section it came from), and cost (compute at ingestion, storage, and index size). Fixed-size splitting is cheapest and worst on the first two. Semantic splitting buys boundary quality at ingestion cost. Late chunking and contextual retrieval buy context retention, the first with a long-context encoder and the second with one LLM call per chunk. Hybrid search and reranking are not chunking at all; they are query-time insurance against whatever the chunker got wrong.

The Chunking Strategy Landscape: What Each Method Actually Does

The best chunking strategy is the cheapest one whose recall at your working top-k clears your target on your own evaluation set. Start with recursive splitting at 200 to 400 tokens, add BM25 hybrid retrieval and a reranker before paying for semantic splitting, and reserve late chunking or contextual retrieval for corpora where chunks lose their referent.

Figure 2: Naive chunking versus late chunking for RAG chunking pipelines. Late chunking reverses the order of embed and split, so every chunk vector carries information from the whole document.

Figure 2 contrasts the two orderings. In naive pipelines the text is split first and each piece is encoded in isolation. In late chunking the whole document goes through the transformer once, and the split happens on the output token embeddings. The long description for accessibility: the left lane runs document, split, embed each chunk, vector with no outside context; the right lane runs document, long-context encoder, split at boundaries, mean pool each span, vector that is document aware.

Fixed-size and recursive chunking

Fixed-size chunking cuts every N tokens or characters, optionally with an overlap of M. It is deterministic, parallel, and nearly free. Its failure is obvious: it cuts mid-sentence, mid-table and mid-code-block. Recursive chunking is the practical upgrade. It tries a priority list of separators (double newline, newline, sentence end, space) and only falls through to a finer separator when a piece still exceeds the size limit. The result is chunks that mostly respect paragraphs without any model in the loop.

Overlap is the most misused parameter. The intuition is that repeating the last 100 tokens of a chunk at the start of the next one protects facts that straddle a boundary. The measured effect is smaller and sometimes negative. In Chroma’s 2024 chunking evaluation, using text-embedding-3-large and five retrieved chunks averaged over several corpora, recursive splitting at 400 tokens with 200 overlap scored token-level recall of 88.1 and intersection-over-union (IoU) of 3.3, while the same size with zero overlap scored recall 89.5 and IoU 3.6. Removing overlap improved both, because overlap duplicates tokens and wastes retrieval budget (Chroma, Evaluating Chunking Strategies for Retrieval). The authors also saw that with a much smaller embedding model (all-MiniLM-L6-v2), overlap mattered more: recall fell from 82.4 to 77.1 when overlap was removed from a 250-token splitter. The lesson is to treat overlap as a hyperparameter to test, not a default to copy.

Chunk size has a clearer pattern. In the same study, smaller chunks raised precision and IoU while recall stayed in the high 80s: recursive 200 with no overlap reached recall 88.1, precision 7.0 and IoU 6.9, versus recall 85.4 and IoU 1.5 for recursive 800 with 400 overlap. Small chunks put fewer irrelevant tokens in the prompt. The cost is that each small chunk carries less context, which is exactly the problem the later sections address.

Semantic chunking

Semantic chunking replaces the size rule with a similarity rule. The common recipe, popularized in Greg Kamradt’s notebook and implemented in LlamaIndex and LangChain, embeds every sentence, computes the cosine distance between adjacent sentences, and starts a new chunk where the distance exceeds a threshold, for example the 95th percentile of all distances in the document. Variants cluster sentences globally instead of only comparing neighbours, and an LLM-based variant simply asks a model to mark topic boundaries.

The theory is sound: boundaries should follow meaning. The evidence is mixed. Qu, Tu and Bao tested semantic chunking on document retrieval, evidence retrieval and answer generation, and concluded that its extra computational cost is not justified by consistent performance gains over fixed-size chunking (Is Semantic Chunking Worth the Computational Cost?, arXiv 2410.13070). Chroma’s results nuance this. The default Kamradt-style splitter had the lowest recall of the configurations tested (83.6) and chunks averaging about 660 tokens. A modified version capped at 300 tokens reached recall 87.1. A cluster-based chunker at target size 200 reached the best precision and IoU (8.0 each), and an LLM chunker built on GPT-4o reached the best recall (91.9) at roughly 240-token chunks. Those gains over the recursive 200 baseline (recall 88.1, IoU 6.9) are real but modest, and the LLM chunker costs a model call per document.

My reading, and it is an opinion grounded in these two sources: semantic chunking is a refinement you evaluate after the cheap levers, not a foundation. Its benefit appears mainly on documents with weak formatting (transcripts, scraped prose) where recursive separators have nothing to grab. On Markdown, HTML or code with strong structure, structure-aware splitting beats it at near-zero cost.

Structure-aware and hierarchical chunking

Before reaching for embeddings to find boundaries, use the boundaries the author already provided. Split Markdown on headings, HTML on section elements, code on function and class definitions, tables on rows with the header repeated. Then attach the heading path as metadata, for example Pricing > Enterprise > Support SLAs, and prepend it to the text before embedding. This single trick restores much of the missing context at the cost of a few dozen tokens per chunk.

Hierarchical (parent-child) chunking goes one step further. You embed small child chunks, 100 to 200 tokens, for precise matching, but when a child is retrieved you return its larger parent section, 800 to 1,500 tokens, to the model. LlamaIndex calls this auto-merging retrieval and LangChain calls it a parent document retriever. The index stays precise while the generator gets enough surrounding text to answer. The price is a larger prompt per retrieved hit and a lookup from child to parent.

Late chunking

Late chunking, introduced by Günther and colleagues at Jina AI, attacks the context problem inside the embedding model rather than in the text. A transformer encoder produces one contextual embedding per token. Normally you truncate input to a chunk, run the model, and mean-pool all token vectors into one chunk vector. With late chunking you run the model over the entire document (up to its context limit, 8,192 tokens for the Jina models in the paper), keep the token-level outputs, and only then mean-pool the token vectors that fall inside each chunk’s span. Because self-attention already let every token see the whole document, the pooled vector for “its population is 3.85 million” encodes that “its” refers to Berlin (Late Chunking, arXiv 2409.04701).

Three properties make this attractive. It needs no extra training for the base technique. It adds no LLM calls. And it is orthogonal to the boundary rule: you can pair it with fixed, sentence or semantic boundaries. The requirement is a long-context embedding model that exposes token-level outputs. Documents longer than the model window are handled by “long late chunking”: split into overlapping macro-chunks of the maximum length, then apply late chunking inside each.

Contextual retrieval

Anthropic’s contextual retrieval takes the opposite route to the same goal. For every chunk, an LLM reads the whole document plus the chunk and writes a short (roughly 50 to 100 tokens) situating sentence, such as “This chunk is from ACME’s Q2 2023 SEC filing; the prior quarter’s revenue was $314 million.” That sentence is prepended to the chunk before both embedding and BM25 indexing. Anthropic reports that, on their test set measuring top-20 retrieval failure rate, contextual embeddings alone cut failures by 35 percent (5.7 percent down to 3.7 percent), adding contextual BM25 brought the reduction to 49 percent (2.9 percent), and adding a reranker brought it to 67 percent (1.9 percent). They put the one-time generation cost at about $1.02 per million document tokens using prompt caching, assuming 800-token chunks. These are the vendor’s own measurements on their own datasets, so treat them as an existence proof rather than a forecast for your corpus.

Notice what the Anthropic numbers also show: BM25 and reranking contributed as much as the context step. That ordering of effort versus benefit is the through-line of this article.

Deeper Analysis: Measuring Late Chunking, BM25 Hybrid Search and Reranking

What late chunking buys in numbers

The late chunking paper reports nDCG@10 on four small BeIR datasets (SciFact, NFCorpus, FiQA, TREC-COVID) for three embedding models. Averaged over datasets and models, fixed 256-token chunks scored 52.2 with naive chunking and 54.0 with late chunking. Five-sentence chunks moved from 52.4 to 54.3, and semantic boundaries from 52.4 to 53.8. In relative terms the paper reports improvements of 3.63 percent for sentence boundaries, 3.46 percent for fixed-size boundaries, and 2.70 percent for semantic boundaries, roughly 1.5 to 1.9 nDCG points.

Two readings follow. First, late chunking is a consistent but modest gain, not a step change; per-dataset gains range from nothing (SciFact with one model, 70.7 versus 70.6) to large (NFCorpus with the small Jina v2 model, 23.5 to 30.0). Second, the semantic-boundary row is informative: naive semantic chunking averaged 52.4, the same as naive five-sentence chunking, and fixed 256-token chunking was only 0.2 points behind. Boundary sophistication barely registered next to the effect of restoring context. The authors also note that late chunking helps most when chunks are small, can lose to naive chunking on some reading-comprehension tasks with large chunks, and gives no benefit on synthetic needle-in-a-haystack tests. They flag that the retrieval experiments use only small BeIR datasets and that the technique does not help when the relevant content is unrelated to its surroundings.

BM25: the lexical half of hybrid search

BM25 scores a document D against a query by summing, over each query term, an inverse document frequency weight multiplied by a saturated, length-normalized term frequency. Two parameters control it. The saturation parameter k1 limits how much repeated occurrences of a term keep increasing the score (Elasticsearch and Lucene default it to 1.2). The length parameter b controls how strongly long documents are penalized (default 0.75; b = 0 removes length normalization) (Elastic, Practical BM25).

BM25 matters for chunking in two ways. It rewards exact rare tokens: a query for ERR_CERT_AUTHORITY_INVALID, a SKU, a function name or a person’s surname is matched by BM25 even when an embedding model maps all error strings to similar vectors. And it interacts with chunk size. Length normalization was designed for documents of varying length; with uniform 200-token chunks it has little to normalize, and b becomes a weaker lever. Very small chunks also reduce term frequency, so saturation matters less. When you shrink chunks, re-tune k1 and b rather than inheriting defaults.

Dense retrieval complements this. An embedding model matches “how do I reset my password” to “account recovery steps” with no shared words, where BM25 scores near zero. Neither retriever dominates, and their errors are partly independent, which is the statistical reason fusion works.

Fusing the lists: reciprocal rank fusion

BM25 scores and cosine similarities live on different scales, so adding them is meaningless without normalization. Reciprocal rank fusion (RRF) sidesteps the problem by using only ranks. Each retriever returns an ordered list; a document’s fused score is the sum, over lists, of 1 / (k + rank). The constant k dampens the dominance of the very top positions; Cormack, Clarke and Buettcher introduced the method in 2009 and the conventional value is 60, which is also the default rank_constant in Elasticsearch’s implementation (Elastic, RRF reference).

A worked example with k = 60: a document ranked 1st by BM25 and 4th by dense retrieval scores 1/61 + 1/64 = 0.01639 + 0.01563 = 0.03202. A document ranked 2nd by both scores 1/62 + 1/62 = 0.03226 and wins, because agreement beats a single high rank. A document ranked 1st by one list and absent from the other scores only 0.01639. That behavior is exactly what you want for hybrid search: consensus floats up, single-retriever flukes sink but remain in the pool.

The alternative is weighted score fusion after min-max normalization. It can outperform RRF when you have labelled data to tune the weight, but it is fragile when score distributions shift between queries. RRF has no tuning surface worth speaking of, which is why most vector databases (Elasticsearch, OpenSearch, Qdrant, Weaviate, pgvector-based stacks) ship it as the default fusion method.

Figure 3: Hybrid search with BM25, dense retrieval, reciprocal rank fusion and a reranker. Each stage narrows the candidate set while spending more compute per candidate.

Reranking: spending compute where it counts

A bi-encoder embeds the query and each chunk independently, so chunk vectors are precomputed and search is a fast nearest-neighbour lookup. A cross-encoder reads the query and one chunk together, with full attention between them, and outputs a relevance score. It is much more accurate and much too slow to run over a whole index, so it is applied only to the fused shortlist.

Figure 3 shows the funnel. Anthropic’s described setup fetches the top 150 chunks from the combined retrievers, reranks them, and passes the top 20 to the model; they tested passing 5, 10 and 20 and found 20 best. Typical production values are a pool of 50 to 150 and a final context of 5 to 20, and the right pair is whatever your latency budget allows. Cross-encoder cost scales linearly with pool size: if a reranker scores one query-chunk pair of 300 tokens in about 2 ms on a GPU (illustrative figure; measure your own model and hardware), a pool of 100 costs about 200 ms if run serially, and less when batched. That arithmetic is why the pool size, not the model, is usually the first knob to turn, a trade-off detailed in the RAG latency budget guide.

Reranking also changes how you should think about chunk size. Because the reranker corrects ordering, the first-stage retriever only needs high recall at depth 100 or 150, not precision at depth 5. Smaller, more precise chunks matter less when a reranker cleans up, and slightly larger chunks (300 to 500 tokens) that carry more context often win once reranking is on. Re-test chunk size after you add the reranker; the optimum moves.

Runnable example: recursive chunks, BM25, dense retrieval, RRF

The following dependency-light Python implements the whole first-stage pipeline. It uses rank_bm25 for lexical scoring, sentence-transformers for dense vectors, and a twelve-line RRF function.

# pip install rank_bm25 sentence-transformers numpy
import re
import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer

def recursive_split(text, max_tokens=300, seps=("\n\n", "\n", ". ", " ")):
    """Greedy recursive splitter; token = whitespace word (approximation)."""
    if len(text.split()) <= max_tokens:
        return [text.strip()] if text.strip() else []
    for sep in seps:
        parts = text.split(sep)
        if len(parts) > 1:
            chunks, buf = [], ""
            for p in parts:
                cand = (buf + sep + p) if buf else p
                if len(cand.split()) <= max_tokens:
                    buf = cand
                else:
                    if buf:
                        chunks.append(buf)
                    buf = p
            if buf:
                chunks.append(buf)
            out = []
            for c in chunks:                      # recurse on oversized pieces
                out.extend(recursive_split(c, max_tokens, seps))
            return out
    words = text.split()                          # hard cut fallback
    return [" ".join(words[i:i + max_tokens]) for i in range(0, len(words), max_tokens)]

def tok(s):
    return re.findall(r"[a-z0-9_]+", s.lower())

def rrf(rankings, k=60, top=10):
    scores = {}
    for ranking in rankings:                      # each is a list of doc ids, best first
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)[:top]

docs = open("handbook.txt", encoding="utf-8").read()
chunks = recursive_split(docs, max_tokens=300)

bm25 = BM25Okapi([tok(c) for c in chunks], k1=1.2, b=0.75)
model = SentenceTransformer("BAAI/bge-small-en-v1.5")
emb = model.encode(chunks, normalize_embeddings=True)

def hybrid(query, depth=100, top=10):
    lex = np.argsort(-bm25.get_scores(tok(query)))[:depth]
    q = model.encode([query], normalize_embeddings=True)[0]
    dense = np.argsort(-(emb @ q))[:depth]
    fused = rrf([lex.tolist(), dense.tolist()], k=60, top=top)
    return [(chunks[i], s) for i, s in fused]

for text, score in hybrid("ERR_CERT_AUTHORITY_INVALID on renewal"):
    print(round(score, 4), text[:80].replace("\n", " "))

Feed the top results into a cross-encoder such as a CrossEncoder from sentence-transformers to complete the funnel. The code is deliberately plain so you can swap in your own tokenizer; BM25Okapi here uses its own IDF variant, so scores will not match Lucene exactly even with identical k1 and b.

Runnable example: late chunking with a long-context encoder

Late chunking needs token-level outputs, so you run the encoder yourself instead of calling a pooled-embedding endpoint. The sketch below uses Hugging Face Transformers with a long-context encoder. jinaai/jina-embeddings-v2-small-en (8,192-token window) is one that the paper evaluated; it requires trust_remote_code=True. Verify the model card for current loading instructions before you depend on it.

# pip install transformers torch
import torch
from transformers import AutoTokenizer, AutoModel

name = "jinaai/jina-embeddings-v2-small-en"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModel.from_pretrained(name, trust_remote_code=True).eval()

def late_chunk(document, chunk_char_spans, max_len=8192):
    """chunk_char_spans: list of (start_char, end_char) from ANY chunker."""
    enc = tokenizer(document, return_tensors="pt", truncation=True,
                    max_length=max_len, return_offsets_mapping=True)
    offsets = enc.pop("offset_mapping")[0]            # (n_tokens, 2)
    with torch.no_grad():
        token_emb = model(**enc).last_hidden_state[0]  # (n_tokens, dim)
    vectors = []
    for start, end in chunk_char_spans:
        # tokens whose character span lies inside this chunk (skip special tokens)
        mask = (offsets[:, 0] >= start) & (offsets[:, 1] <= end) & (offsets[:, 1] > offsets[:, 0])
        if mask.sum() == 0:
            vectors.append(None)                       # chunk beyond the truncated window
            continue
        v = token_emb[mask].mean(dim=0)
        vectors.append(torch.nn.functional.normalize(v, dim=0))
    return vectors

Two details bite in practice. The document must fit in the window, otherwise use the macro-chunk approach from the paper (overlapping windows, late chunking within each). And the query must be embedded with the same model in ordinary pooled mode, because query and chunk vectors need to share a space.

Evaluating and Choosing: A Decision Process You Can Defend

You cannot pick a chunking strategy by reading papers, including this one, because every result above was measured on someone else’s corpus. The only defensible method is to build a small evaluation set from your own documents and measure retrieval before generation.

Figure 4: A decision flow for RAG chunking. Each branch adds one technique only when a measured symptom justifies its cost.

Figure 4 encodes the order of operations. In prose: begin with recursive splitting at 200 to 400 tokens, measure recall at your intended top-k, and stop if it clears your target. If queries contain identifiers or jargon, add BM25 and RRF. If chunks routinely lose their referent (“the company”, “this section”), add contextual retrieval or late chunking. If the top results contain the answer but not in the first few slots, add a reranker. Finally tune size and metadata.

Building the evaluation set

You need 100 to 300 questions with known source passages. Three sources work. Mine real queries from search logs or support tickets and have a domain expert mark the passage that answers each. Generate synthetic questions by prompting an LLM with a chunk, then filter the ones that leak the chunk’s exact wording. And add a set of “hard” queries by hand: exact identifiers, negations, multi-hop questions, questions whose answer spans a boundary.

Label at the passage or span level, not the chunk level. If your gold label is a chunk ID, every change of chunker invalidates the set. If the gold label is a character span in the source document, you can score any chunker by whether the retrieved text overlaps the span. This is the idea behind Chroma’s token-level metrics: recall is the fraction of gold tokens retrieved, precision is the fraction of retrieved tokens that are gold, and IoU penalizes both misses and waste.

Metrics that matter

For retrieval, report recall@k (did the gold passage appear in the top k) at the k you will actually pass to the model, plus MRR or nDCG@10 for ranking quality. Track the token budget alongside: recall@5 with 800-token chunks is not comparable to recall@5 with 200-token chunks, since the first hands the model four times more text. Report recall per retrieved token, or fix a token budget (say 3,000 tokens of context) and compare strategies inside it.

For end-to-end quality, run an answer-correctness and groundedness check with a calibrated judge model, but only after retrieval is stable; generation metrics are noisy and conflate retrieval with prompt and model effects. The failure-analysis approach in our LLM-as-a-judge calibration guide applies directly.

A worked cost comparison

Ingestion cost differs by orders of magnitude, so do the arithmetic before committing. Take a corpus of 10 million tokens, chunked to roughly 300 tokens, which gives about 33,000 chunks (10,000,000 / 300).

Recursive chunking: zero model calls. Embedding cost is one pass over 10 million tokens.
Semantic chunking: one embedding per sentence before the final chunk embeddings. If sentences average 25 tokens, that is 400,000 sentence embeddings on top of the chunk embeddings, roughly doubling embedding compute, plus the similarity pass.
Late chunking: one pass of a long-context encoder over 10 million tokens. Attention cost grows with window length, so expect more compute per token than 300-token windows, but still a single encoder pass and no LLM calls.
Contextual retrieval: one LLM call per chunk, each reading the document. Anthropic quotes about $1.02 per million document tokens with prompt caching at 800-token chunks, so roughly $10 for this corpus; at 300-token chunks there are about 2.7 times as many calls, so the figure would rise, though caching of the document prefix limits the increase. Treat these as order-of-magnitude values and recompute with current model pricing.

At 10 million tokens, all four are affordable. At 10 billion tokens, the LLM-call approaches become a budget line and a re-indexing project, which is when late chunking and structure-aware metadata start to look attractive. The embedding-side economics are discussed further in the embedding model fine-tuning guide, including Matryoshka truncation to shrink index size.

A decision matrix

Strategy	Ingestion cost	Boundary quality	Context retention	Best for	Main risk
Fixed-size, no structure	Lowest	Poor	None	Prototypes, uniform logs	Mid-sentence cuts
Recursive, 200-400 tokens	Very low	Good on formatted text	None	Default baseline	Overlap waste
Structure-aware plus heading path	Low	Very good	Partial via metadata	Docs, wikis, Markdown, code	Needs clean parsing
Semantic (embedding or cluster)	Medium	Good on unformatted prose	None	Transcripts, scraped text	Cost not reliably justified
LLM-proposed boundaries	High	Good	None	Small, high-value corpora	Cost, nondeterminism
Parent-child	Low	Good	Good at generation	Long manuals, contracts	Larger prompts
Late chunking	Medium	Same as chosen splitter	Good	Narrative or pronoun-heavy text	Window limit, model lock-in
Contextual retrieval	High	Same as chosen splitter	Very good	Financial, legal, reports	LLM cost, stale context

The index itself also constrains your choice. Smaller chunks mean more vectors, and the vector index memory cost scales with count: see the HNSW versus DiskANN versus IVF-PQ internals comparison for how that plays out at tens of millions of vectors.

Trade-offs, Gotchas, and What Goes Wrong

Evaluating on the wrong unit. Teams compare chunkers by end-to-end answer quality with a fixed prompt and a noisy judge, then cannot tell whether a change came from retrieval or generation. Score retrieval first, with span-level labels, and hold the token budget constant. A strategy that “wins” because it retrieves more tokens has not won.

Over-trusting benchmark transfer. The late chunking results come from small BeIR datasets, and the authors say so. Chroma’s study used a specific set of corpora and embedding models, and its own alternate-model table reordered the winners on recall. Anthropic’s 67 percent figure is vendor-measured. Each is evidence about direction, not magnitude on your data.

Embedding model lock-in with late chunking. Late chunking only works with encoders that expose token embeddings and handle long inputs. If your production embedder is a hosted pooled-output endpoint, you cannot apply it without switching vendors or running a model yourself. Switching embedders means re-embedding everything, and queries must use the same model. Also, mean-pooled token vectors from a model trained with ordinary pooling can differ in distribution from its standard outputs, so verify on your evaluation set before assuming parity.

Stale or wrong context. Contextual retrieval bakes a document-level summary into each chunk. If the source document is revised, every chunk’s prefix may be wrong, and the LLM that wrote it can hallucinate a detail (a date, a figure) that then becomes searchable fact. Version your contextualization prompt, store the generated context separately from the raw text so you can display the original, and re-generate on document change.

Lexical failure modes in BM25. Tokenization decides everything. A default analyzer splits ERR_CERT_AUTHORITY_INVALID or v2.3.1 into pieces that match too much or too little; code, part numbers and non-English text need custom analyzers (n-gram or language-specific). BM25 also has no notion of synonyms, so it silently misses “car” when the document says “automobile”, which is precisely why you keep the dense retriever.

Fusion hides a weak retriever. RRF is forgiving, which makes it easy to ship a hybrid system where one leg contributes nothing. Log per-retriever recall on your evaluation set, not just fused recall. If BM25 alone reaches 70 percent recall at 100 and dense alone reaches 85, but fused is 86, the lexical leg is decoration on your queries, and removing it simplifies the system.

Metadata filters and chunk boundaries. If you filter by tenant, date or document type, small chunks multiply the filter cost and can starve a filtered ANN search of candidates. See the filtered vector search guide for how pre-filtering and post-filtering behave in HNSW.

Table, code and figure content. A table split across chunks loses its header row and becomes uninterpretable; a code block cut mid-function loses scope. Treat these as atomic units, keep them whole even when they exceed the size limit, and index a natural-language description alongside the raw content.

Chunk size mismatch with the embedder. Many embedding models are trained on sequences much shorter than their stated maximum. Feeding 2,000-token chunks to a model that technically accepts them can reduce quality. Check the model card for the training length and test two or three sizes.

Practical Recommendations

Use the cheapest approach that survives measurement. The sequence below reflects the evidence in this article: the cheap levers (size, no unnecessary overlap, hybrid retrieval, reranking) deliver most of the documented improvement, while semantic splitting and context-restoring methods are targeted tools.

Build the evaluation set first. 100 to 300 questions, span-level gold labels, fixed token budget. Without it, every later step is guesswork.
Baseline with recursive splitting at 200 to 400 tokens and zero overlap, then test overlap of 10 to 20 percent as a hypothesis rather than a default.
Exploit structure. Split on headings and code boundaries, prepend the heading path, keep tables and code atomic.
Add BM25 hybrid retrieval with RRF (k = 60) if any queries contain identifiers, names or domain jargon. Log per-retriever recall.
Add a cross-encoder reranker over a pool of 50 to 150, passing 5 to 20 chunks to the model. Then re-test chunk size upward.
Fix lost context only if failure analysis shows it: structured metadata first, then late chunking if you control the embedder, contextual retrieval if you can afford one LLM call per chunk.
Try semantic chunking last, and only on poorly formatted prose, comparing against your tuned recursive baseline at equal chunk sizes.
Monitor in production. Track retrieval hit rate on sampled real queries, and re-run the evaluation set whenever you change the embedder, the splitter or the corpus mix.

If your corpus is under about 200,000 tokens, check whether you need retrieval at all: placing the whole knowledge base in a cached prompt may beat any pipeline described here, as Anthropic notes. For a hosted embedding option, see the Cohere Embed 5 Pro explainer.

Frequently Asked Questions

What is the best chunk size for RAG?

There is no universal best, but 200 to 400 tokens is a strong starting range. Chroma’s evaluation found recursive splitting at 200 tokens with no overlap gave the best balance of recall (88.1), precision and IoU among simple splitters, while 800-token chunks with heavy overlap had the lowest efficiency. Anthropic’s examples use chunks of a few hundred tokens. Larger chunks can win after you add a reranker, so test 200, 400 and 800 on your own queries.

Is semantic chunking better than fixed-size chunking?

Not reliably. A study of document retrieval, evidence retrieval and answer generation found its extra compute cost was not justified by consistent gains over fixed-size chunking. Chroma saw modest improvements from cluster-based and LLM-based chunkers over a recursive baseline, but a default embedding-similarity splitter had the lowest recall of those tested. Try semantic chunking after hybrid search and reranking, mainly for unformatted prose such as transcripts.

What is late chunking and when should I use it?

Late chunking embeds the entire document with a long-context embedding model first, then splits the token embeddings into chunks and mean-pools each one, so each chunk vector carries document-wide context. The paper reports roughly 1.5 to 1.9 nDCG@10 points average improvement across boundary types. Use it when you control the embedding model, documents fit its window (such as 8,192 tokens), and chunks suffer from pronouns and missing referents.

Does chunk overlap improve RAG retrieval?

Often it does not. Chroma found that removing overlap from 400-token recursive chunks raised both recall (88.1 to 89.5) and IoU (3.3 to 3.6), since overlap duplicates tokens and wastes the retrieval budget. With a much smaller embedding model, overlap helped recall more. Treat overlap as a tunable hyperparameter: evaluate 0, 10 and 20 percent on your corpus, and prefer structural boundaries to repeated text.

Why combine BM25 with vector search?

They fail differently. BM25 matches exact rare terms, such as error codes, SKUs and names, that embeddings blur; dense retrieval matches paraphrases that share no words. Reciprocal rank fusion, which sums 1 / (60 + rank) across lists, merges them without score normalization. Anthropic reported that adding contextual BM25 to contextual embeddings cut top-20 retrieval failures from 3.7 percent to 2.9 percent on their test set.

Do I still need a reranker if I use hybrid search?

Usually yes, if precision in the final few slots matters. Hybrid retrieval is tuned for recall at depth 50 to 150; a cross-encoder reads query and chunk together and orders that pool far more accurately. Anthropic’s setup reranks 150 candidates to 20 and measured a drop in failure rate from 2.9 percent to 1.9 percent on top of contextual embeddings plus BM25. Latency grows with pool size, so budget it explicitly.

RAG Chunking Strategies: Fixed-Size, Semantic, Late Chunking, BM25 Hybrid Search and Reranking (2026)