RAG Reranker Benchmark: Cohere Rerank vs BGE vs Jina vs ColBERT (2026)

Most retrieval-augmented generation (RAG) systems that disappoint in production do not have a generation problem. They have an ordering problem. The right passage is somewhere in the top 100 results of the vector search, but it sits at rank 23, and the language model only ever sees the top five. A reranker is the cheapest component you can add to fix that, and choosing one is the most under-measured decision in the stack. This RAG reranker benchmark guide is a rewrite of our June post, built only on numbers that vendors and paper authors have actually published, with explicit labels wherever a figure is illustrative.

Reranking matters more in 2026 than it did two years ago because the field fractured. Cross-encoders, late-interaction models, and a new family of LLM-based and listwise rerankers now compete, and the vendor lineup has moved: Cohere shipped Rerank 4, Jina shipped a listwise v3, and Qwen3-Reranker arrived under Apache 2.0. You will leave with a mental model of the three architectures, a table of published scores with their caveats, latency and cost math you can rerun with your own numbers, and a complete evaluation harness.

What this covers: why two-stage retrieval exists, the three reranker architectures, the current model lineup, what the published benchmarks do and do not tell you, latency and cost arithmetic, harness code, failure modes, and a decision matrix.

Context and Background

Retrieval for RAG is almost always two-stage. The first stage is fast and approximate: BM25, a dense bi-encoder with an approximate nearest-neighbour index, or a hybrid of both. Its job is recall, meaning that the passages that answer the question should appear somewhere in the candidate list. The second stage is slow and precise. Its job is precision at the top of the list, because the generator has a limited context window and, as the “lost in the middle” literature showed, uses long contexts unevenly.

The bi-encoder compresses a whole passage into one vector before it has seen the question. Any nuance the query cares about that was not encoded into that vector is gone. A reranker reads the query and the passage together, so it can notice that the passage mentions the right entity but negates the claim, or that it answers a neighbouring question. That is why the gain is often larger than what you get from swapping in a bigger embedding model; our embedding models benchmark covers the first-stage side of the same trade.

The evaluation vocabulary is worth fixing early. Recall@k asks whether any relevant passage is in the top k. nDCG@10 (normalised discounted cumulative gain) asks how well the top 10 are ordered, giving more credit to relevant items at higher ranks and supporting graded relevance. MRR (mean reciprocal rank) cares only about the first relevant hit. For RAG, the right pair is recall@k of the first stage, which sets the ceiling, and nDCG@10 or recall@5 after reranking, which measures what the generator will actually receive.

The public yardsticks you will see are BEIR (a heterogeneous zero-shot suite of 18 retrieval datasets), MTEB’s reranking and retrieval tasks, MIRACL for multilingual retrieval, and MLDR for long documents. Two details are frequently glossed over. First, rerankers are scored on a candidate list produced by some specific first-stage retriever, so scores from different papers are not directly comparable. Second, most vendors report on benchmarks they also trained near, which inflates apparent generalisation. We return to both under methodology caveats. For the hybrid first stage that feeds a reranker, see our write-up of GraphRAG and hybrid retrieval patterns.

The reference text for the field’s cross-encoder baseline is the monoBERT line of work on MS MARCO, and the reference for efficient late interaction is the ColBERT family from Stanford. Everything else in this post is a variation on those two ideas, plus the newer decoder-only twist where a language model reads many candidates at once.

The Two-Stage Pipeline and Three Reranker Architectures

A reranker is a second-stage model that re-scores the 50 to 200 candidates from a fast first-stage retriever by reading query and passage jointly, then passes only the top 5 to 10 to the language model. It trades extra compute per query for precision, and it can only reorder what the first stage already found.

Figure 1: A two-stage RAG pipeline. The first stage is judged on recall, the reranker on nDCG, and a score cache and evaluation harness sit alongside both.

Figure 1 shows the pipeline we benchmark. The candidate pool is the contract between the stages: it must be large enough that the answer is in it, and small enough that the reranker finishes inside your latency budget. Everything that follows is about getting the best precision out of that second stage for the least money and time.

Cross-encoders: maximum interaction, no precomputation

A cross-encoder concatenates the query and one passage into a single sequence, runs a transformer over the pair, and emits one relevance score. Every query token can attend to every passage token in every layer, which is why this family has been the quality baseline since BERT-based rerankers beat BM25 on MS MARCO. The cost is that nothing about a passage can be computed ahead of time: each (query, passage) pair is a fresh forward pass. Scoring 100 candidates means 100 forward passes, though they batch well on a GPU.

BGE-reranker-v2-m3 from BAAI is the standard open example. Its model card describes a lightweight multilingual reranker built on the bge-m3 base, with 0.6 billion parameters, licensed Apache-2.0, that outputs a relevance score directly rather than an embedding. The model card’s own examples truncate inputs at 512 tokens, which is a practical limit to design around. Cohere Rerank is the best-known managed cross-encoder service; Cohere does not publish its architecture, so we describe it only by its documented behaviour.

Late interaction: precompute the passage side

ColBERT encodes the query and the passage separately, but instead of pooling each into a single vector it keeps one vector per token. Relevance is the MaxSim operator: for each query token vector, find the maximum similarity over all passage token vectors, then sum those maxima across query tokens. Because passage token vectors do not depend on the query, they can be computed offline and stored.

The price is storage. The ColBERTv2 paper states that late interaction inflates the space footprint by roughly an order of magnitude compared with single-vector representations, and reports that its residual compression plus denoised supervision cuts the footprint of late-interaction models by 6 to 10 times. That is the number to remember: ColBERT is cheap at query time and expensive on disk, and the engineering effort goes into index compression and candidate generation.

There is a second, quieter point. Used as a reranker over a small candidate list, ColBERT does not need a giant index: you can encode the 100 candidates on the fly and apply MaxSim, which is far cheaper than a cross-encoder forward pass per pair but gives up the full cross-attention. Used as a retriever, it needs the whole token-level index. Teams routinely conflate the two modes when they compare it with cross-encoders.

Listwise and LLM rerankers: read many candidates at once

The newest family puts the query and many candidates into one context window and lets a decoder-only model reason across them. Jina’s reranker v3 is the clearest example with published specifics. Its model page describes a 0.6 billion parameter model built on the Qwen3-0.6B backbone with 28 layers and a 131,072-token context window, processing up to 64 documents together in one window. Its paper calls the mechanism “last but not late” interaction: query and documents interact through causal attention inside the shared context first, then a contextual embedding is extracted from each document’s final token, projected through a small MLP to 256 dimensions, and compared with the query embedding by cosine similarity.

Qwen3-Reranker is the other member of this generation. It is a family of three sizes (0.6B, 4B, 8B, with 32K context) under Apache-2.0, and it is instruction-aware: you pass a task string alongside the query, and the model card reports that instructions typically add about 1 to 5 percent. A listwise design gives candidates a chance to be compared with one another rather than scored in isolation, at the price of quadratic attention cost across the window.

Figure 2: Three ways to score relevance. Cross-encoders attend across one pair, late interaction precomputes passage tokens and sums MaxSim, listwise models put a whole candidate set in one causal window.

Figure 2 contrasts the three. The decision they encode is where you spend compute. A cross-encoder spends it per pair at query time. Late interaction spends it at indexing time and on disk. A listwise model spends it on long-context attention, and amortises the query across up to 64 documents.

The 2026 Reranker Lineup, with Verified Facts

The old version of this post compared Cohere Rerank 3, BGE-reranker-v2-m3, Jina Reranker v2 and ColBERT. All four still exist, but the lineup has moved, and you should evaluate against the current generation. Below is what we could verify from primary pages this month. Where a page did not state something (pricing, context length for the newest Cohere models), we say so rather than guess.

Cohere Rerank

Cohere’s documentation currently lists five models: rerank-v4.0-pro (positioned for the highest quality and complex use cases), rerank-v4.0-fast (low latency and high throughput), rerank-v3.5, rerank-english-v3.0 and rerank-multilingual-v3.0. All handle semi-structured (JSON) documents. The v3.x models document a 4,096-token context length. The docs state that the query and each document’s tokens together count against that limit, and that a document exceeding it is automatically chunked and processed in multiple inferences. The docs page we read did not state a context length for v4.0, a maximum document count or pricing, so check the API reference and pricing page for those before you budget. Cohere’s announcement of Rerank 4 is dated December 11, 2025 and gave no benchmark figures in the text we could read.

Operationally, the pro/fast split is the useful change. It lets you put the cheap model on the hot path and reserve pro for an offline or high-value path, which is the same pattern as a two-tier retriever.

BGE reranker family

BGE-reranker-v2-m3 remains the default open cross-encoder: 0.6B parameters, multilingual, Apache-2.0, with FlagEmbedding, Transformers and sentence-transformers loaders. BAAI also publishes larger LLM-based variants: bge-reranker-v2-gemma (built on gemma-2b) and bge-reranker-v2-minicpm-layerwise (built on MiniCPM-2B, with selectable output layers from 8 to 40 so you can trade accuracy for speed). The model card’s guidance is to use v2-m3 for multilingual work or efficiency, and the gemma or layerwise variants for better performance.

Jina rerankers

Jina’s line runs from jina-reranker-v2 (0.3B, a cross-encoder) through jina-reranker-m0 (2.4B, multimodal) to jina-reranker-v3 (0.6B, listwise). The v3 paper reports a BEIR score of 61.94 nDCG@10 and claims to be much smaller than comparable generative listwise rerankers. Jina’s page does not state a license or latency figures for v3, so confirm the model card terms before commercial use, since earlier Jina models carried non-commercial terms on the open weights.

Qwen3-Reranker and mixedbread

Qwen3-Reranker-0.6B, 4B and 8B are Apache-2.0. mixedbread’s mxbai-rerank-v2 models (base 0.5B and large 1.5B per Jina’s comparison table) are Apache-2.0 on the model page we read, trained with a reinforcement-learning pipeline; the large model card reports a 57.49 BEIR average and 0.89 seconds latency on an A100, though the card does not say at what candidate count. Treat latency numbers without a stated batch and candidate count as marketing, not data.

ColBERT and Jina-ColBERT

ColBERTv2 is the reference late-interaction model, with the PLAID engine for efficient search. Jina also ships ColBERT-style multi-vector models with longer context. For RAG, the realistic ways to consume them are RAGatouille (a Python wrapper), Vespa, or a vector database with native multi-vector support.

What the Published Benchmarks Actually Say

We did not run new experiments for this rewrite, and we are not going to invent a leaderboard. The June version of this post printed nDCG@10 and latency numbers for four models and labelled them illustrative; that was an honest label but not a useful table. The numbers below are copied from the model authors’ own tables, with the source and the setup beside each.

Table 1: Jina’s comparison table for jina-reranker-v3

The table is from Jina’s announcement page. BEIR is nDCG@10 using the top 100 results from jina-embeddings-v3 as first stage (the authors’ own runs). MIRACL and CoIR are nDCG@10; MKQA is reported by Jina as a recall figure at 10 and the page does not label the metric in the table, so read it cautiously.

Model	Size	BEIR	MIRACL	MKQA	CoIR
jina-reranker-v3	0.6B	61.94	66.83	67.92	70.64
jina-reranker-v2	0.3B	57.06	63.65	67.90	56.14
jina-reranker-m0	2.4B	58.95	66.75	68.19	63.55
bge-reranker-v2-m3	0.6B	56.51	69.32	67.88	36.28
mxbai-rerank-base-v2	0.5B	58.40	55.32	64.24	65.71
mxbai-rerank-large-v2	1.5B	61.44	57.94	67.06	70.87
Qwen3-Reranker-0.6B	0.6B	56.28	57.70	65.34	65.18
Qwen3-Reranker-4B	4.0B	61.16	67.52	67.52	73.91

Table 1: Reranker scores as published by Jina (vendor-run, Jina’s own first stage). Not independently reproduced.

Read it carefully. The headline BEIR gap between jina-reranker-v3 (61.94) and Qwen3-Reranker-4B (61.16) is under one point and the 4B model wins on CoIR. The bigger story is the spread: on BEIR, BGE-reranker-v2-m3 trails v3 by about 5.4 points, but on MIRACL it leads everything in the table (69.32). On CoIR, the code-retrieval benchmark, BGE falls to 36.28 while Qwen3-4B reaches 73.91. No model wins everywhere, which is the single most important finding to carry into your own evaluation.

Table 2: Qwen’s comparison table for Qwen3-Reranker

Qwen’s model card reports scores reranking the top 100 candidates from Qwen3-Embedding-0.6B, which is a different first stage and different benchmarks, so do not mix numbers with Table 1.

Model	Params	MTEB-R	CMTEB-R	MMTEB-R	MLDR	MTEB-Code	FollowIR
Qwen3-Reranker-0.6B	0.6B	65.80	71.31	66.36	67.28	73.42	5.41
Qwen3-Reranker-4B	4B	69.76	75.94	72.74	69.97	81.20	14.84
Qwen3-Reranker-8B	8B	69.02	77.45	72.94	70.19	81.22	8.05
BGE-reranker-v2-m3	0.6B	57.03	72.16	58.36	59.51	41.38	-0.01
Jina-multilingual-reranker-v2-base	0.3B	58.22	63.37	63.73	39.66	58.98	-0.68
gte-multilingual-reranker-base	0.3B	59.51	74.08	59.44	66.33	54.18	-1.64

Table 2: Reranker scores as published on the Qwen3-Reranker model card (vendor-run).

Three observations. First, the 4B model beats the 8B on MTEB-R (69.76 vs 69.02) and FollowIR, so bigger is not monotonic, and you should not pay for 8B without testing. Second, the instruction-following column (FollowIR) is near zero or negative for the older cross-encoders: they cannot take instructions, and the number shows it. Third, the code columns again separate the families sharply, with BGE-reranker-v2-m3 at 41.38 on MTEB-Code against 73.42 for the smallest Qwen model.

Cohere is missing from both tables

Neither vendor table includes Cohere Rerank, and Cohere did not publish numbers in the pages we read. That is a real gap: the most widely deployed managed reranker has no first-party, reproducible score against these baselines. Independent leaderboards do evaluate it, but we could not verify a specific current figure this run, so we are not printing one. If Cohere is on your shortlist, you must measure it on your own data; the harness below makes that a 50-line job.

Methodology caveats you must apply to every table

Different first stages. Table 1 reranks Jina embeddings v3’s top 100; Table 2 reranks Qwen3-Embedding-0.6B’s top 100. A stronger first stage compresses the reranker’s measurable gain.
Vendor-run. Each vendor ran the baselines at their own settings. Baselines are rarely tuned as carefully as the featured model.
Benchmark proximity. BEIR and MTEB corpora are public; training sets overlap with them to a degree nobody fully audits.
Metric labelling. MKQA appears under a recall label for one vendor and nDCG for others in the wild. Check the unit before comparing.
No latency parity. The tables report quality only. Latency depends on hardware, batch, candidate count and token length.

The practical implication is that these tables are a shortlist generator, not a decision. Use them to choose three candidates, then run them against your own queries.

Latency and Cost Arithmetic

Quality tables omit the two things that decide whether a reranker ships. Let us derive them from first principles, with every input labelled so you can substitute your own.

Cross-encoder cost per query

A cross-encoder processes N candidates, each of length L tokens (query plus passage). Dense transformer inference costs about 2 times the parameter count in floating-point operations per token for the forward pass. For BGE-reranker-v2-m3 at about 0.6B parameters (the embedding matrix is a large share of that, so effective compute is lower, but we will use the full figure as a conservative bound), one 400-token pair is roughly 2 x 0.6e9 x 400 = 4.8e11 FLOPs, and 100 candidates is about 4.8e13 FLOPs per query.

On a GPU sustaining an illustrative 30 percent of 65 TFLOPs of FP16 tensor throughput (about 20 TFLOPs effective, a plausible figure for a T4-class card, which we are assuming rather than citing), that is roughly 2.4 seconds per query in the worst case. Real deployments are faster because the effective compute excludes embeddings and most of the sequence is shorter than 400 tokens, and because inference stacks overlap and batch. The point is the scaling: latency grows linearly in candidate count and, for long inputs, superlinearly in length. Cut candidates from 100 to 30 and a cross-encoder gets roughly 3 times faster. Measure rather than trust the estimate.

Listwise models change the arithmetic

For a listwise model reading 64 documents of about 300 tokens each in one window, the sequence is around 19,200 tokens plus the query. Self-attention cost grows with the square of sequence length, but at this size the feed-forward layers still dominate for a 28-layer, 1,024-hidden model, so you are paying a linear-ish cost on 19,200 tokens, comparable in total tokens to scoring 64 pairs at 300 tokens. The saving is that the query is encoded once rather than 64 times and the model can share work across documents. The risk is memory: KV-cache for a long window competes with batch size, which limits throughput on small GPUs. Jina’s page makes no latency claims for v3, so there is no vendor number to quote; benchmark it yourself.

Managed API cost model

For an API priced per search (a query with up to some number of documents), cost per month is simply queries per month times price per search, plus overage rules for documents beyond the per-search cap. Cohere’s pricing page was not retrievable in a form we could verify, so write it as a formula with your negotiated rate:

def monthly_api_cost(qpm, price_per_1k_searches, cache_hit_rate=0.0):
    """qpm = queries per month. Prices come from the vendor pricing page."""
    billable = qpm * (1.0 - cache_hit_rate)
    return billable / 1000.0 * price_per_1k_searches

# Illustrative inputs only, not a vendor quote:
print(monthly_api_cost(20_000_000, 2.00, 0.25))  # 30000.0

The self-hosted alternative has a fixed cost per GPU-hour and a variable throughput. Break-even is gpu_hour_cost / (queries_per_second_sustained * 3600) per query, compared with the API per-search price. At illustrative numbers of 3 USD per hour and 40 sustained queries per second, self-hosting costs about 0.00002 USD per query, versus 0.002 USD per search at 2 USD per thousand. That is a 100x gap that only holds if you actually keep the GPU busy; at 5 percent utilisation the gap collapses to about 5x, and once you add engineering time, on-call and redundancy, managed APIs often win below a few million queries per month.

The latency budget

Work backwards from the user. If the product target is 1.5 seconds to first token and the generator needs 600 ms to start streaming, the first stage takes 80 ms and network overhead 100 ms, the reranker has roughly 700 ms at the 95th percentile. That budget permits an API call and 100 candidates for most managed rerankers, or a self-hosted model on a modest GPU. If the budget is 300 ms total, only a small self-hosted cross-encoder over 20 to 30 candidates fits. Decide the budget first and let it choose the candidate count; picking the model first is how teams end up cutting candidates to 10 and wondering why recall collapsed.

Building the Evaluation Harness

The most valuable output of this post is not a ranking, it is a harness that lets you produce your own. The harness below computes recall and nDCG for any reranker that exposes a score(query, passages) method, on your own judged queries. It is complete enough to run; you supply the data.

import math, time, statistics
from typing import Callable, Dict, List

def ndcg_at_k(ranked_ids: List[str], rels: Dict[str, int], k: int = 10) -> float:
    """rels maps doc id -> graded relevance (0 means not relevant)."""
    dcg = sum((2 ** rels.get(d, 0) - 1) / math.log2(i + 2)
              for i, d in enumerate(ranked_ids[:k]))
    ideal = sorted(rels.values(), reverse=True)[:k]
    idcg = sum((2 ** r - 1) / math.log2(i + 2) for i, r in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0.0

def recall_at_k(ranked_ids, rels, k):
    relevant = {d for d, r in rels.items() if r > 0}
    return len(relevant & set(ranked_ids[:k])) / len(relevant) if relevant else 0.0

def evaluate(queries, candidates, qrels, reranker: Callable, k=10, n_in=100):
    """queries: id->text; candidates: id->[(doc_id, text)] from stage one;
    qrels: id->{doc_id: grade}; reranker(query, texts)->list[float]"""
    out = {"ndcg": [], "recall10": [], "recall5": [], "lat_ms": []}
    base = {"ndcg": [], "recall5": []}
    for qid, qtext in queries.items():
        cands = candidates[qid][:n_in]
        ids = [c[0] for c in cands]
        base["ndcg"].append(ndcg_at_k(ids, qrels[qid], k))
        base["recall5"].append(recall_at_k(ids, qrels[qid], 5))
        t0 = time.perf_counter()
        scores = reranker(qtext, [c[1] for c in cands])
        out["lat_ms"].append((time.perf_counter() - t0) * 1000)
        order = [i for _, i in sorted(zip(scores, ids), reverse=True)]
        out["ndcg"].append(ndcg_at_k(order, qrels[qid], k))
        out["recall10"].append(recall_at_k(order, qrels[qid], 10))
        out["recall5"].append(recall_at_k(order, qrels[qid], 5))
    lat = sorted(out["lat_ms"])
    p = lambda q: lat[min(len(lat) - 1, int(q * len(lat)))]
    return {
        "ndcg@10": statistics.mean(out["ndcg"]),
        "ndcg@10_no_rerank": statistics.mean(base["ndcg"]),
        "recall@5": statistics.mean(out["recall5"]),
        "recall@5_no_rerank": statistics.mean(base["recall5"]),
        "p50_ms": p(0.5), "p95_ms": p(0.95),
    }

Plug in rerankers behind the same signature. A local BGE cross-encoder via sentence-transformers and a hosted one via an SDK look like this:

from sentence_transformers import CrossEncoder
bge = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)
def bge_fn(q, texts):
    return bge.predict([(q, t) for t in texts], batch_size=32).tolist()

import cohere
co = cohere.ClientV2()  # reads CO_API_KEY
def cohere_fn(q, texts):
    r = co.rerank(model="rerank-v3.5", query=q, documents=texts, top_n=len(texts))
    scores = [0.0] * len(texts)
    for item in r.results:
        scores[item.index] = item.relevance_score
    return scores

Swap rerank-v3.5 for rerank-v4.0-pro or rerank-v4.0-fast as available on your account. Check the SDK reference for current parameter names, since SDK surfaces change faster than model names.

Designing the judged set

Fifty queries is the minimum to see a 3-point nDCG gap; a hundred to two hundred makes 1-point differences visible. Build them from real logs. Sample queries stratified by intent (lookup, how-to, comparison, troubleshooting), have a domain expert mark relevance on a 0 to 3 scale for the union of the top 30 of each system under test (pooling), and keep at least 20 percent queries where no good answer exists, because the reranker must still produce sensible scores when the corpus does not contain the answer. Report a bootstrap confidence interval on the difference in mean nDCG between systems; with 100 queries, deltas under about 2 points are usually noise.

Measure the first stage first

Compute recall@100 of your retriever before you touch a reranker. If recall@100 is 62 percent, 38 percent of queries cannot be fixed by any reranker, and the gain you should chase is in chunking, hybrid search or query rewriting. A useful rule of thumb is to keep improving the first stage until recall@candidate-pool is above roughly 90 percent, then invest in reranking. That is a heuristic from practice, not a published threshold.

Deeper Analysis: Choosing by Workload

Figure 3: A decision flow for reranker choice. Infrastructure and passage length eliminate options before quality differences matter.

The flow in Figure 3 encodes our view that infrastructure and data shape remove options before benchmark deltas matter. It is a heuristic, not a law.

Multilingual and code corpora

Table 1 and Table 2 show that language and domain mix reorders the field. BGE-reranker-v2-m3 posts the best MIRACL number in Jina’s table (69.32) yet has the weakest code scores in both tables. If your corpus is natural-language support content in many languages, BGE remains a strong, cheap default. If it includes source code, API docs or logs, a Qwen3-Reranker or a mixedbread model deserves a test. Industrial IoT knowledge bases are typically a mix: maintenance manuals in several languages plus PLC and configuration snippets, which is exactly where one model’s blind spot hurts.

Long passages

Chunking decisions interact with reranker context limits. A cross-encoder with a 512-token ceiling silently truncates a 900-token chunk, so the part of the passage past the cutoff cannot contribute to the score. Cohere documents the opposite behaviour for its v3.x models: it chunks over-long documents and scores them in multiple inferences, which is safer but costs more calls and requires you to understand how chunk scores are combined, which Cohere’s docs we read did not specify. Listwise models with 32K to 131K windows relax the limit, but quality still degrades when you fill the window with long documents; 64 documents at 300 tokens is a very different workload from 64 at 2,000.

Instructions as a feature

Qwen3-Reranker’s instruction input is more than a convenience. A reranker told “prefer maintenance procedures over marketing text” or “prefer documents for firmware version 4.x” can encode business logic that a fixed cross-encoder cannot. The model card reports a 1 to 5 percent gain from instructions on average, and recommends writing them in English even for multilingual retrieval. We would treat the instruction as part of your configuration, version it, and include it in your evaluation, because a changed instruction is a changed model.

When to use ColBERT-style late interaction

Late interaction shines when the corpus is large and stable, you can afford the index, and you want one system to serve both retrieval and ranking. Its weaknesses are operational: token-level indexes are large (the ColBERTv2 paper’s own motivation), updates are harder than appending vectors to an HNSW index, and the tooling is less uniform than for single-vector search. As a pure reranker over 100 candidates it is a cheap middle path between a bi-encoder and a cross-encoder, but you give up some of the full-attention accuracy. If your team has no existing multi-vector infrastructure, we would start with a cross-encoder and revisit.

Decision matrix

Situation	First pick	Why	Watch out for
No GPU, moderate volume, multilingual	Cohere Rerank (v4 fast or pro, or v3.5)	Managed, JSON-aware, documented chunking of long docs	Per-search cost, unstated v4 limits
Self-hosted, tight latency, multilingual text	BGE-reranker-v2-m3	0.6B, Apache-2.0, strong MIRACL	512-token practice limit, weak on code
Mixed text and code, GPU available	Qwen3-Reranker 4B	Highest published CoIR and MTEB-Code in tables above	Larger memory, test 4B vs 8B
Many candidates, long context, small model	Jina reranker v3	0.6B listwise, 64 docs per window, top BEIR in Jina table	Confirm license, no vendor latency figure
Huge static corpus, one system for retrieve and rank	ColBERTv2 / PLAID stack	Precomputed token vectors, query-time cheap	Index size, update complexity
Instruction-driven ranking policy	Qwen3-Reranker	Task instruction input	Version the instruction

Table 3: Decision matrix combining published results and architecture constraints. Treat the picks as starting hypotheses for your own evaluation.

Trade-offs, Gotchas, and What Goes Wrong

Rerankers fail in predictable ways. The failures are rarely in the model; they are in the system around it.

Candidate starvation. If the first stage returns 10 candidates and you rerank 10, you have reordered, not recovered. The reranker cannot surface what retrieval missed. Symptoms: reranking changes top-1 often but end-to-end answer quality does not move. Fix: raise the pool to 50 to 100, add BM25 to the first stage for exact terms such as part numbers, and measure recall@pool.

Latency tail, not median. The p50 of an API reranker may be fine while the p95 breaks your SLO because of cold paths, rate limiting or retries. Budget on p95 or p99, set a hard timeout, and define the fallback: on timeout, serve the retriever’s order. That fallback is Figure 4’s central design point.

Silent truncation. Truncating at 512 tokens drops the tail of a chunk without any error. A chunk that begins with boilerplate and puts the answer at token 600 will score low. Log the fraction of candidates exceeding the model limit; if it exceeds a few percent, shrink chunks or change model.

Score calibration. Cross-encoder scores are not probabilities across queries. A fixed threshold such as “drop everything under 0.3” works poorly because score distributions shift between queries and models. If you need abstention (answer “I don’t know” when nothing is relevant), calibrate on a labelled set, per model, and re-calibrate on model upgrades.

Position bias in the generator. Reranking puts the best chunk first, but generators also show position effects. Test both orderings, best-first and best-at-ends, with your actual model.

Model drift. Hosted rerankers are updated or deprecated. A vendor moving from v3.5 to v4 changes your score distribution, your latency and your cost. Pin the model name in config, log it with every request, and keep a frozen evaluation set so you can detect regressions on upgrade.

Licensing. Open weights are not always commercial-friendly. BGE-reranker-v2-m3 and Qwen3-Reranker list Apache-2.0 on the cards we read; the Jina v3 page we read did not state a license. Read the model card of the exact checkpoint you deploy.

Evaluating on benchmark data only. All the published tables above are on public data. A reranker that ties on BEIR can differ by many points on your jargon-heavy, multilingual maintenance manuals. The benchmark’s job is to shortlist; your own judged set decides.

Production Pipeline Design

Figure 4: A production reranking request. Cached scores skip the model, a timeout falls back to retriever order, and only the top chunks reach the generator.

Figure 4 shows the request path we recommend. Three details are worth spelling out.

First, the score cache is keyed on a hash of the query text, the chunk content hash and the model name plus version. Keying on content rather than chunk id means re-chunked or edited documents invalidate automatically. Enterprise search and agent workloads repeat queries often, so hit rates can be material, but the rate is entirely workload-dependent; measure it before promising savings. Our context engineering guide for production LLM agents covers the surrounding context-assembly layer that consumes these top chunks.

Second, only cache misses are sent to the reranker. With a managed API this reduces cost directly; with a self-hosted model it reduces GPU load. If the API prices per search regardless of document count, batching misses from the same query into one call is optimal.

Third, the fallback is a design decision, not an error handler. When the reranker times out, you can serve the retriever order, serve a cheaper reranker’s output (for example the fast variant) or return a degraded answer with a notice. Choose in advance, test it with fault injection, and alert on fallback rate. A rising fallback rate is an early warning that your reranker latency has drifted.

Observability

Track, per request: candidates in, candidates cached, reranker latency, timeout flag, model version, and the rank of the chunk the generator cited. The last metric is the most telling. If cited chunks come from rank 1 to 3 after reranking and from rank 20 to 60 before, the reranker is earning its cost. If they come from rank 1 to 3 both before and after, you are paying for nothing on those queries; segment by query type to find where it matters. This “citation rank shift” is cheap to compute and correlates with the real effect better than reranker score distributions.

Fine-tuning

Domain fine-tuning is often the highest-leverage move. BGE rerankers fine-tune through FlagEmbedding with query, positive and hard-negative triples. Hard negatives matter more than volume: mine them from your first stage’s wrong top-20, because those are exactly the confusions the reranker will face. A few thousand good triples can shift domain performance more than moving to a larger model. Keep an untouched test set, and compare against the off-the-shelf model on it so you know the tuning is a win and not an overfit.

Practical Recommendations

Start with measurement, not a model. Build a judged set of 100 real queries, measure first-stage recall at your candidate pool, and only then add a reranker. Pick three candidates from Tables 1 and 2 that fit your infrastructure: typically one managed API, one small open cross-encoder, and one newer open model such as Qwen3-Reranker or Jina v3. Run the harness, compare nDCG@10, recall@5, and p95 latency on identical candidates, and decide with a confidence interval.

For most teams the sensible default is a two-tier setup: a small, cheap reranker on the hot path with a 50 to 100 candidate pool, and an option to route high-value queries to a stronger model. Keep the model name in configuration, cache scores, set a timeout with a documented fallback, and re-run the evaluation whenever the model, the chunking or the embedding model changes.

Checklist before shipping:

Recall@pool of the first stage measured and above roughly 90 percent on your judged set.
At least 100 judged queries, including unanswerable ones, with a bootstrap interval on nDCG@10 deltas.
Three candidate rerankers evaluated on identical candidate lists.
p95 latency measured under realistic concurrency, not single-request.
Chunk sizes checked against each model’s token limit; truncation rate logged.
License of the exact checkpoint reviewed for commercial use.
Cost modelled at current volume, 10x and 100x, with and without caching.
Timeout and fallback tested with fault injection.
Model version pinned and logged; frozen eval set kept for upgrades.

Frequently Asked Questions

What is the best reranker for RAG in 2026?

There is no single winner. In vendor-published tables, Jina reranker v3 and Qwen3-Reranker-4B lead on BEIR (61.94 and 61.16 nDCG@10 in Jina’s runs), BGE-reranker-v2-m3 leads on MIRACL (69.32), and Qwen3 models lead on code benchmarks. Cohere Rerank publishes no comparable first-party numbers. Shortlist three models using the published tables, then choose with an evaluation on your own queries, latency budget and licensing constraints.

Is a cross-encoder better than ColBERT for reranking?

Cross-encoders generally achieve higher precision because every query token attends to every passage token, but they cannot precompute passage representations. ColBERT precomputes token vectors and scores with MaxSim, which is cheaper at query time but needs a large index; ColBERTv2 reports 6 to 10 times footprint reduction through compression. Used over a small candidate list, ColBERT is a fast middle path. For a large static corpus, its single-system retrieve-and-rank design can win.

How many documents should I send to the reranker?

Typically 50 to 100, then keep the top 5 to 10 for the generator. The right number depends on the first stage’s recall curve: choose the smallest pool where recall stops improving materially. Cross-encoder latency scales roughly linearly with candidates, so halving the pool roughly halves latency. If recall@100 is low, fix retrieval, chunking or query rewriting rather than raising the pool.

Do I need a GPU to run a reranker?

Not necessarily. Managed APIs such as Cohere Rerank need no GPU on your side. A 0.6B cross-encoder like BGE-reranker-v2-m3 can run on CPU for low traffic, but latency for 100 candidates will be much higher than on a GPU, so test against your budget. Models at 4B to 8B parameters, such as larger Qwen3-Reranker sizes, realistically need a GPU with enough memory for the weights and the long-context KV cache.

Does reranking always improve RAG answers?

No. If the right passage is already first, a reranker adds cost without changing the context. It helps most when relevant passages are retrievable but ranked low, when queries are ambiguous, or when the corpus is noisy. Measure with and without reranking on a judged set, and check how often the cited chunk moves into the top three after reranking. If it rarely moves, spend effort elsewhere.

Can I fine-tune a reranker on my own data?

Yes, and it is often the highest-impact change for a specialised corpus. Open cross-encoders such as BGE-reranker-v2-m3 can be fine-tuned with triples of query, positive passage and hard negatives mined from your first stage. Keep a held-out test set and compare with the off-the-shelf model. Check the license before commercial use and plan to retrain when your embedding model or chunking changes.