Fine-Tune Embedding Model for Retrieval: Contrastive Loss, Hard Negatives and Matryoshka (2026)

Most retrieval-augmented generation systems start with a general-purpose embedding model and a vector index, and most of them plateau at the same place. The demo works, the first hundred questions look fine, and then real users start asking in the vocabulary of your domain: part numbers, clause references, internal product names, abbreviations that no web-scale corpus ever taught the model. Recall at ten sits at 0.70, and no amount of prompt engineering moves it, because the right passage never reaches the prompt.

The most reliable fix is to fine-tune an embedding model on your own query-document pairs. It is cheap by modern standards, often a few GPU-hours on a base-size encoder, and it improves the one component every later stage depends on. But it is easy to do badly: the wrong loss, a tiny batch, false negatives in the training data, or an evaluation set that leaks can all produce a model that looks better and retrieves worse.

What this covers: when fine-tuning beats prompting and reranking, the bi-encoder architecture, the InfoNCE and MultipleNegativesRankingLoss math with worked numbers, batch-size effects, hard-negative mining, synthetic query generation with an LLM, Matryoshka training for truncatable vectors, runnable sentence-transformers code, evaluation without leakage, re-embedding costs, and the failure modes that waste a training run.

Context and Background

A dense retriever maps text to vectors so that a query lands near the passages that answer it. The dominant design is the bi-encoder: one transformer encodes queries, the same transformer (or a sibling) encodes documents, and retrieval is a nearest-neighbour search over precomputed document vectors. Because documents are encoded offline, query latency is one encoder forward pass plus an approximate nearest-neighbour lookup.

That efficiency has a price. The query and document never interact inside the model, so everything the retriever knows about relevance must be compressed into the geometry of a single fixed-size vector. A cross-encoder, which reads the query and document together, is far more expressive but must run once per candidate. The usual production compromise is a bi-encoder for recall over millions of documents and a cross-encoder reranker over the top fifty or a hundred.

General-purpose embedding models are trained on enormous mixtures of web pairs, question-answer sets, and synthetic data. The BEIR benchmark, which spans 18 public datasets, reported that in zero-shot settings BM25 stayed a robust baseline and dense retrievers often underperformed, which the authors read as a sign that generalisation to new domains had considerable room to improve. Models have improved since, but the underlying issue remains: a model that has never seen your corpus has no reason to know that “PN-4471-B” and “the lower bracket” refer to the same thing, or that in your contracts “term” means something different from the dictionary.

Fine-tuning is the direct response. You show the model pairs from your domain, tell it which are matches, and let contrastive learning pull them together. The cost has fallen sharply because the sentence-transformers library, now at a major version well beyond the 3.x line that introduced its Trainer-based API, wraps tokenisation, batching, mixed precision, evaluation, and checkpointing into a few dozen lines. You can verify the version you are on with sentence_transformers.__version__, and the code in this post sticks to long-stable class names so it survives minor releases.

Fine-tuning also interacts with the rest of the stack. Latency budgets decide how large a model you can afford, as we covered in RAG latency budget optimisation across retrieval, reranking and streaming. Vector dimension decides memory and index cost, which is where Matryoshka training enters, and which we explore from the serving side in Matryoshka embeddings and adaptive retrieval architecture. And the index you ultimately write into has its own tradeoffs, benchmarked in our vector database comparison.

When fine-tuning beats the alternatives

Before spending GPU time, rule out cheaper levers. Fine-tuning an embedding model is the right move when four conditions hold together.

First, your failure is a recall failure, not a ranking failure. Measure recall at 100. If the relevant passage is in the top 100 but not the top 5, a cross-encoder reranker or a better chunking strategy fixes it. If the passage is not in the top 100, no reranker can recover it, and the embedding space itself is wrong for your domain.

Second, your domain vocabulary is out-of-distribution. Legal citations, chemical names, maintenance codes, firmware identifiers, and multilingual jargon are the classic cases. Hybrid retrieval with BM25 handles exact tokens, but it cannot bridge the paraphrase between a technician’s symptom description and a manual’s procedure title.

Third, you have, or can generate, a few thousand realistic query-passage pairs. Below roughly a thousand pairs, gains are noisy and overfitting is a real risk. Above ten thousand clean pairs, results tend to be stable. These are rules of thumb from practice, not published thresholds.

Fourth, you can afford the operational consequence: every fine-tuned model change means re-embedding the corpus. That cost is often the real decision criterion, and it gets its own section below.

Prompting does not belong on this list. Instruction prefixes such as “Represent this query for retrieval” can help models that were trained with them, but a prefix cannot teach a vocabulary the encoder has never seen. Query rewriting with an LLM helps when users are vague, and it adds a model call to every request. Fine-tuning moves the cost to training time, where it is paid once.

The Bi-Encoder Training Architecture

Fine-tuning an embedding model for retrieval means training a siamese-style bi-encoder with a contrastive objective: for each query, the loss rewards a high similarity to its positive passage and a low similarity to every negative. Negatives come for free from other examples in the same batch, and from mined hard negatives that look relevant but are not.

Figure 1: Training architecture. A shared encoder embeds queries, positives and mined hard negatives; the batch similarity matrix feeds a cross-entropy contrastive loss, optionally applied again at several truncated dimensions for Matryoshka.

Figure 1 shows the whole loop. A training example is a triplet of query, positive passage, and optionally one hard negative. A batch of B triplets is tokenised and passed through the encoder, giving B query vectors, B positive vectors, and B hard-negative vectors. Pooling collapses token-level states into one vector per text, typically mean pooling or the CLS token, depending on how the base model was pre-trained. The vectors are normalised, and a B by 2B similarity matrix is computed: each query against every positive and every hard negative in the batch.

Each row of that matrix is a classification problem. The correct class is the query’s own positive; the other 2B minus 1 columns are wrong answers. Cross-entropy over the row is the contrastive loss.

The InfoNCE objective and MultipleNegativesRankingLoss

The loss is a form of InfoNCE, introduced in the contrastive predictive coding paper as a probabilistic contrastive loss made tractable by negative sampling. For a query q with positive p and a set of negatives N, using a similarity function s and a temperature-like scale, the loss for one example is:

L = -log( exp(scale * s(q, p)) / ( exp(scale * s(q, p)) + sum over n in N of exp(scale * s(q, n)) ) )

In sentence-transformers this is MultipleNegativesRankingLoss, often shortened to MNRL. It takes anchor-positive pairs or anchor-positive-negative triplets and treats every other positive, and every supplied negative, in the batch as a negative for each anchor. Its default similarity is cosine and its default scale is 20, equivalent to a temperature of 0.05; both are configurable, and you should confirm the defaults in the documentation for your installed version.

Think about what the scale does. Cosine similarity lives in a narrow band, and the difference between a good match at 0.62 and a hard negative at 0.55 is only 0.07. Multiplying by 20 turns that into a logit gap of 1.4, and the softmax can then express a meaningful preference. Without scaling, the loss surface is nearly flat and learning is slow. With too high a scale, the model is rewarded for confident separation on a handful of examples and gradients vanish on the rest.

Worked numbers: what the loss is telling you

You can sanity-check a training run with arithmetic. A model that knows nothing assigns equal probability to all candidates, so the loss equals the natural log of the number of candidates. With a batch of 32 pairs and no hard negatives, each query has 32 candidates and the chance-level loss is ln(32) = 3.47. With 256 pairs it is ln(256) = 5.55, and with 1,024 it is ln(1024) = 6.93.

Two consequences follow. A larger batch makes the task harder, so the reported loss at the same level of skill is higher, and you cannot compare loss values across batch sizes. Second, the loss starting well below the chance value on step zero means your base model already separates the pairs, which is a good sign that fine-tuning will be gentle rather than transformative.

Now a concrete step. Suppose a query has cosine similarity 0.62 to its positive and 0.55, 0.40 and 0.35 to three other candidates. With scale 20 the logits are 12.4, 11.0, 8.0 and 7.0. The exponentials are roughly 242,000, 59,900, 2,980 and 1,100. The positive’s probability is 242,000 divided by their sum of about 306,000, which is 0.79, and the loss is -ln(0.79) = 0.23. Now replace the 0.55 candidate with a hard negative at 0.61. Its logit is 12.2, the probabilities become roughly 0.55 for the positive and 0.45 for the hard negative, and the loss jumps to about 0.61, more than double. Hard negatives raise the loss, and with it the gradient, exactly where the model is confused.

The symmetric variant and what the gradient does

The basic loss is one-directional: query to document. A symmetric variant also computes the document-to-query direction, which helps when documents may be used as queries, as in deduplication or related-article lookup. For pure question answering the one-directional loss is the standard choice.

The gradient of the softmax loss has an intuitive form. For each query, the positive is pulled toward it with a weight of one minus its probability, and every negative is pushed away with a weight equal to that negative’s probability. Easy negatives, with tiny probability, contribute almost nothing. This is why training on random negatives saturates: after a few hundred steps, the loss on a random batch is dominated by candidates the model already rejects. It also explains why hard negatives are not an optional nicety. They are the only candidates still carrying gradient.

In-batch negatives and the batch-size effect

In-batch negatives are efficient because they cost no extra encoding: the positives of the other B minus 1 examples are already in memory. The number of negatives per query grows linearly with batch size, so a larger batch gives a tighter bound on the mutual information the loss estimates, and in practice better embeddings. The InfoNCE paper makes the same point: more negatives, a better estimate.

The problem is memory. A batch of 256 passages of 256 tokens, with gradients through the encoder, may not fit on a single 24 GB card for a base-size model. The standard remedy is gradient caching, implemented in sentence-transformers as CachedMultipleNegativesRankingLoss. It computes embeddings for the full batch without gradients, computes the loss and gradients with respect to the embeddings, then re-runs the encoder in mini-batches to backpropagate. Peak memory depends on the mini-batch size rather than the logical batch size, at the cost of roughly one extra forward pass. This lets a single GPU train with logical batches of several thousand.

There is a ceiling, though. A larger batch increases the chance that one in-batch candidate is a genuine, unlabelled match for another query, the false-negative problem discussed later. In a corpus with heavy topical clustering, such as a single product manual, a batch of 1,024 random pairs contains many near-duplicate topics. Returns diminish, and may reverse, which is why you should treat batch size as a hyperparameter to sweep, not a number to maximise.

Hard Negative Mining: Where the Gradient Lives

A hard negative is a passage that the current model scores highly for a query but that does not answer it. Hard negative mining is the systematic search for such passages, and it is usually the single largest quality lever after the quality of the positive pairs themselves.

Figure 2: Mining pipeline. Candidates come from BM25 and from the base model’s own ANN index, a cross-encoder filters likely false negatives, and survivors become the third column of each training triplet.

Where candidates come from

There are three common sources, and they find different kinds of difficulty.

BM25 negatives share vocabulary with the query but not necessarily meaning. They teach the model not to be fooled by lexical overlap, and they are cheap to produce, since BM25 needs no GPU. Their weakness is that they are often easy for a neural model, because the semantic gap is large.

Dense negatives come from running the base embedding model itself over the corpus and taking the top-ranked non-positives. These are the model’s own mistakes, which is exactly what you want to correct. In sentence-transformers the helper mine_hard_negatives does this: it embeds the queries and corpus, finds the nearest neighbours, and filters them using rank window, score thresholds and margins. Its signature exposes range_min and range_max to choose a rank window, max_score and min_score for similarity thresholds, absolute_margin and relative_margin to demand that a negative be sufficiently below the positive, num_negatives, a sampling_strategy of either top or random, and output_format options including triplets and n-tuples. It can use FAISS for large corpora and optionally a cross_encoder to rescore candidates.

Cross-encoder negatives are the most accurate. A reranker scores each query-candidate pair jointly, and you keep candidates whose reranker score is clearly below the positive’s. This is the safest filter against false negatives, because the reranker sees the pair together and can spot a candidate that in fact answers the question.

The false-negative trap

Mining the very top of the ranked list is dangerous. If the top-ranked non-positive is a near-duplicate chunk, a paraphrase in another document, or a passage that answers the question just as well as the labelled positive, then training pushes the query away from a correct answer. The model learns that correct answers are wrong, and recall on exactly the long-tail queries you care about drops.

Three defences work in combination. Skip the top ranks: setting range_min to 3 or 10 skips the closest candidates, which are most likely to be unlabelled positives. Enforce a margin: require the negative’s score to be below the positive’s by an absolute or relative margin, so near-ties are discarded. And use a reranker veto: drop any candidate the cross-encoder scores above a threshold.

A rule of thumb from practice, not a published constant, is to sample from ranks roughly 10 to 50 and apply a 5 percent relative margin. Treat these as starting points and inspect samples by eye. Reading fifty mined triplets takes ten minutes and routinely reveals systematic problems, such as a corpus containing three copies of the same manual that all get treated as each other’s negatives.

How many hard negatives, and curriculum

One hard negative per example is a good default, and it fits the triplet format that MNRL consumes. Adding more increases memory linearly, since each extra negative must be encoded, and has diminishing returns. If you do use several, keep them from different ranks to add diversity.

A second consideration is difficulty scheduling. Starting training with the hardest possible negatives can destabilise a model that has not yet learned the domain basics. A practical curriculum is two stages: an initial epoch with random in-batch negatives only, then a second stage with mined negatives. Another approach is periodic re-mining, which uses the partially trained model to find fresh negatives, at the cost of re-embedding the corpus between rounds. For most teams, one mining pass from the base model, filtered by a cross-encoder, is the right cost-quality point.

Getting Training Data: Real Logs and Synthetic Queries

Contrastive learning needs positive pairs, and the supply is usually the hard part. There are four sources, in descending order of quality.

Click and feedback logs are the gold standard when they exist: a user searched, opened a document, and stayed. They carry position bias and popularity bias, so debias them or restrict to sessions where the clicked result was not first. Human-labelled pairs are accurate and expensive. Document structure supplies weak pairs for free: a section heading paired with its body, a title paired with its abstract, a question in a FAQ paired with its answer.

The fourth source is synthetic: ask an LLM to write the queries a user would ask for each passage. This is the workhorse for teams with a corpus and no logs.

Figure 3: Synthetic data loop. An LLM writes several queries per chunk, a round-trip filter drops queries that the base retriever cannot find their source for, and the survivors are split by document into train and held-out sets.

A synthetic generation recipe that holds up

The idea is documented in the InPars and Promptagator line of work and in the GPL approach, which use LLM-generated queries to adapt retrievers without labelled data. The exact gains depend on the model and corpus, so measure them on your own held-out set.

A recipe that works in practice has five parts.

Sample chunks representative of production. Chunk the corpus the way your index does, because queries are written against chunks, not whole documents. Sample uniformly across sources, not proportional to size, so one verbose manual does not dominate.

Prompt for realistic queries. Give the LLM the chunk plus two or three real example queries from your users, if you have any, and ask for three different queries per chunk in different styles: a keyword query, a natural-language question, and a symptom-oriented description. Forbid copying long phrases from the passage, since queries with high lexical overlap teach the model to be a keyword matcher.

Run a round-trip filter. For each generated query, search the base retriever over the corpus. If the source chunk is not in the top 20, the query is probably ambiguous, too generic, or hallucinated, and is dropped. Be aware that this filter biases the set toward queries the base model already handles; a looser threshold such as the top 100 keeps harder examples and lets the contrastive loss do more work.

Deduplicate and balance. Near-identical queries across chunks produce false negatives in the same batch. Cluster queries by embedding and cap the count per cluster.

Split by document, not by pair. If chunks from the same document appear in both train and test, the model memorises document-specific phrasing and the evaluation overstates quality. Hold out entire documents or entire source folders.

Scale and budget

A worked example, with illustrative numbers. A corpus of 5,000 documents yielding 40,000 chunks, with three queries per chunk, gives 120,000 candidate queries. If the round-trip filter keeps 70 percent, you have 84,000 pairs. At 150 input tokens per chunk plus prompt and 60 output tokens for the three queries, generation consumes on the order of 40,000 times 300 equals 12 million tokens. At typical API prices that is a small line item, and with an open-weight model on your own hardware it is free beyond the hardware time. The point is that data generation is cheap compared with the mistakes that come from skipping the filtering and split steps.

You do not need all 84,000 pairs. Gains typically flatten between ten and fifty thousand pairs for a single-domain corpus, though this varies. Train on a subset first, plot the learning curve against held-out recall, and spend remaining effort on quality rather than quantity.

Matryoshka Representation Learning: Truncatable Vectors

Embedding dimension is a cost multiplier on everything downstream: index memory, ANN search time, network transfer, and cache footprint. Matryoshka Representation Learning, or MRL, trains a single model so that the first k dimensions of its vector are themselves a good k-dimensional embedding. You can then store the full vector, truncate it for a cheap first-pass search, and rerank with the full vector.

The original paper, by Kusupati and colleagues, shows how this is done: apply the loss at several nested prefix sizes at once, so the model is forced to front-load information into early dimensions. The abstract reports up to 14 times smaller embeddings for ImageNet-1K classification at the same accuracy, and up to 14 times real-world speedup for large-scale retrieval on ImageNet-1K and 4K. Those are vision results; text results vary by model and task and should be measured on your data.

The MatryoshkaLoss wrapper

In sentence-transformers, MatryoshkaLoss wraps another loss, such as MNRL. You pass a list of dimensions in descending order, for example 1024, 512, 256, 128 and 64, and optional weights. At each step, the wrapped loss is computed on the full embedding and again on each truncated prefix, and the results are summed with the weights. Because the same forward pass supplies all prefixes, the training cost is close to that of the plain loss.

Two practical notes. First, the largest dimension should equal the model’s native output size. Second, the weights let you favour the dimensions you will actually serve; if your production path uses 256 for first-pass search and 1024 for reranking, weight those two more heavily than 64.

What truncation buys: storage arithmetic

Work the numbers for 10 million documents stored as float32. At 1024 dimensions each vector takes 4,096 bytes, so the raw vectors occupy 40.96 GB. At 256 dimensions the footprint is 10.24 GB, a quarter. If you also quantise to int8, 256 dimensions fall to 2.56 GB, and with binary quantisation the same 256 bits take 32 bytes per vector, or 0.32 GB. These figures exclude index structures such as HNSW graph links, which add their own overhead per vector, so real totals are larger, but the ratios hold for the vector payload.

The retrieval pattern this enables is two-stage. Search a 256-dimensional index for the top 200, then rescore those 200 with their full 1024-dimensional vectors, which can live on cheaper storage. Recall loss from the truncated first stage is recovered by the full-dimension rescoring, provided the first stage’s candidate set still contains the right answers. Measure candidate recall at the truncated dimension, not only final NDCG, because a candidate that never enters the shortlist cannot be rescued.

Does Matryoshka hurt full-dimension quality?

Sometimes slightly. Forcing early dimensions to carry information constrains the full vector, so the 1024-dimension score of an MRL-trained model can be a little below a plain-trained model. Whether that matters is empirical. Evaluate both on your held-out set, at every dimension you plan to serve. If your deployment will never truncate, skip Matryoshka and keep the simpler loss.

Runnable Walk-through with sentence-transformers

The following script goes from a pairs file to a Matryoshka-trained model with mined hard negatives and a retrieval evaluator. It assumes a JSONL file where each line has query, positive and a doc_id, already split by document into train and test files. The class names used are long-standing parts of sentence-transformers; check the signature details against the documentation for your installed version before running.

import json
from datasets import Dataset
from sentence_transformers import (
    SentenceTransformer, SentenceTransformerTrainer,
    SentenceTransformerTrainingArguments,
)
from sentence_transformers.losses import (
    MultipleNegativesRankingLoss, MatryoshkaLoss,
)
from sentence_transformers.training_args import BatchSamplers
from sentence_transformers.util import mine_hard_negatives
from sentence_transformers.evaluation import InformationRetrievalEvaluator

def load(path):
    rows = [json.loads(l) for l in open(path)]
    return Dataset.from_dict({
        "anchor":   [r["query"] for r in rows],
        "positive": [r["positive"] for r in rows],
    })

train = load("train_pairs.jsonl")
base = "BAAI/bge-base-en-v1.5"   # example base model; choose your own
model = SentenceTransformer(base)

# 1. Mine hard negatives with the base model (skip the closest ranks).
train = mine_hard_negatives(
    train, model,
    anchor_column_name="anchor", positive_column_name="positive",
    range_min=10, range_max=50, relative_margin=0.05,
    num_negatives=1, sampling_strategy="random",
    batch_size=128, use_faiss=False,
)  # columns: anchor, positive, negative

# 2. Loss: MNRL wrapped in Matryoshka at several prefix sizes.
inner = MultipleNegativesRankingLoss(model, scale=20.0)
loss = MatryoshkaLoss(
    model, inner,
    matryoshka_dims=[768, 512, 256, 128, 64],
)

# 3. Evaluator on held-out documents (not pairs).
test = [json.loads(l) for l in open("test_pairs.jsonl")]
queries = {f"q{i}": r["query"] for i, r in enumerate(test)}
corpus = {r["doc_id"]: r["positive"] for r in test}
relevant = {f"q{i}": {r["doc_id"]} for i, r in enumerate(test)}
evaluator = InformationRetrievalEvaluator(
    queries, corpus, relevant,
    name="heldout", ndcg_at_k=[10], accuracy_at_k=[1, 10],
    truncate_dim=256,
)

args = SentenceTransformerTrainingArguments(
    output_dir="out/ft-embed-v1",
    num_train_epochs=1,
    per_device_train_batch_size=64,
    learning_rate=2e-5,
    warmup_ratio=0.1,
    fp16=True,
    batch_sampler=BatchSamplers.NO_DUPLICATES,
    eval_strategy="steps", eval_steps=200,
    save_strategy="steps", save_steps=200, save_total_limit=2,
    logging_steps=50,
)
trainer = SentenceTransformerTrainer(
    model=model, args=args, train_dataset=train,
    loss=loss, evaluator=evaluator,
)
print("before:", evaluator(model))
trainer.train()
print("after:", evaluator(model))
model.save_pretrained("out/ft-embed-v1/final")

Reading the script

Several choices in that script are deliberate and worth explaining.

The NO_DUPLICATES batch sampler matters. If two rows in a batch contain the same positive passage, MNRL treats one as a negative for the other, which is a guaranteed false negative. This sampler avoids duplicates within a batch. It does not catch near-duplicates, which is why the deduplication step in the data pipeline still matters.

The learning rate of 2e-5 is a conventional fine-tuning choice for base-size encoders, not a tuned value. Embedding models are sensitive to larger rates, which can erase the pre-trained geometry in a few hundred steps. One epoch with warmup is a safe first run. If the held-out score is still climbing at the end, run more epochs, but compare with checkpoints, since overfitting to a small synthetic set can start early.

The evaluator uses truncate_dim=256 here, so every evaluation reports quality at the dimension you plan to serve for first-pass search. In a real run you would instantiate one evaluator per dimension, or the same evaluator at several truncation settings, and track the whole curve.

The base model name is only an example of a widely used open embedding model; any sentence-transformers-compatible encoder works, and the matryoshka dimensions must not exceed its output size. If your base model emits 1024 dimensions, set the list to start at 1024.

A realistic training budget

An illustrative estimate, to be checked on your hardware. A base-size encoder of roughly 110 million parameters, 50,000 triplets, sequence length 256 for passages, batch size 64, one epoch: that is about 780 optimizer steps. On a single modern 24 GB GPU with mixed precision, such a run typically takes tens of minutes to a couple of hours, depending on the card and padding. Mining hard negatives over a 40,000-chunk corpus adds minutes. In other words, the whole experiment is cheap enough to run many times, and the most valuable use of that cheapness is ablation: run once with random negatives only, once with hard negatives, once with Matryoshka, and compare on the same held-out set.

Evaluation That Does Not Lie to You

A fine-tuned embedding model almost always scores better on its own training distribution. The question is whether it scores better on queries it has not seen, over documents it has not seen, while keeping the general ability you still need. Most failed fine-tuning projects fail here, not in training.

Metrics worth tracking

NDCG at 10 rewards putting relevant documents high in the list and is the headline metric in BEIR-style evaluation. Recall at k, for k of 10, 50 and 100, measures whether the right passage reaches downstream stages, which is what matters when a reranker follows. MRR at 10 is useful when there is a single correct passage. Track at least NDCG at 10 and recall at 100, and report them at each truncation dimension you serve.

The metric must match the pipeline. If a reranker takes the top 50, recall at 50 is the number that predicts final quality, and an NDCG gain at 10 that comes with lower recall at 50 is a loss in disguise.

Building a held-out set that is honest

Hold out by document or by source, not by pair, for the reasons given above. Include real user queries wherever possible, even if only a couple of hundred; a small real set is a better judge than a large synthetic one, because the synthetic generator and the fine-tuned model may share the same blind spots. When the evaluation queries are generated by the same LLM and prompt as the training queries, the model learns the generator’s style, and scores rise without any change in real-world retrieval.

Keep a second evaluation set from outside your domain. A mixed benchmark in the style of MTEB, or a few BEIR datasets, shows how much general capability the fine-tune cost. A drop of a point or two outside the domain may be fine for a dedicated domain index; a collapse signals over-training.

Leakage and contamination

Leakage hides in unexpected places. Chunks that overlap between train and test documents, boilerplate text shared across manuals, the same query template repeated for many chunks, and positives in the test set that also appear as hard negatives in training all inflate scores. A quick check: embed every test query and find its nearest train query; any pair above a high similarity threshold is a near-duplicate and should be audited. Run the same check between test passages and train passages.

Also guard against metric overfitting through repeated looks. If you tune hyperparameters on the held-out set across thirty runs, the set is no longer held out. Keep a final untouched test split that is evaluated once.

Statistical honesty

With 500 test queries, an NDCG difference of 0.01 is usually noise. Bootstrap the per-query scores, or run a paired test across queries, and report an interval. Differences should be judged paired, because the same query is easy or hard for both models. A change that helps 12 percent of queries and hurts 8 percent can have a small mean and a large real effect on users; looking at per-query wins and losses reveals which queries regressed, and those are often the ones to inspect for false-negative issues.

Domain Shift, Forgetting and Model Merging

Fine-tuning moves the embedding space toward your domain, and the movement is not free. Catastrophic forgetting means that capabilities not exercised by your training data degrade. If your users also ask general questions, such as company-wide policy questions that live in a different corpus, an over-tuned model may handle those worse than the base.

There are four standard mitigations. Use a low learning rate and a short schedule, since most of the gain arrives early. Mix in a replay set of general-purpose pairs, perhaps ten to twenty percent of the training data, drawn from a public retrieval dataset. Freeze lower layers if the corpus is small; training only the upper layers keeps the lexical foundations intact. And interpolate weights: after training, average the fine-tuned weights with the base weights using a mixing coefficient between 0 and 1. Linear interpolation of fine-tuned and original weights is a well-established technique for trading specialisation against robustness, and you can sweep the coefficient on both the in-domain and out-of-domain evaluation sets.

Domain shift also happens after deployment. Product lines change, new terminology appears, and the model that was strong in January is weaker by autumn. Log queries with no clicks, sample failed ones monthly, and add them to the next training round. A retraining cadence of quarterly or on-demand when recall drops is typical; the trigger should be a monitored metric, not a calendar.

Re-Embedding Cost and Index Versioning

The decision to fine-tune is also a decision to re-embed. Vectors from the old model and vectors from the new model live in different spaces; comparing a new-model query vector against old-model document vectors produces meaningless scores. So every model change means a full re-encode of the corpus and a new index.

Estimating the cost

A worked example, with illustrative throughput. Take 10 million chunks at 300 tokens each, which is 3 billion tokens. Suppose a base-size encoder processes 20,000 tokens per second on one GPU with batching and mixed precision; this is an assumed round figure, and real throughput depends on the card, sequence lengths and implementation. Then one GPU needs 150,000 seconds, about 41.7 GPU-hours. Spread across eight GPUs, that is roughly five hours of wall-clock time. At a cloud price of a few dollars per GPU-hour the bill is hundreds of dollars, which is modest in money but significant in coordination.

Add the index build. Constructing an HNSW graph for 10 million vectors is CPU-intensive and may take hours, and the new index briefly doubles your storage. If you use Matryoshka truncation, the serving index at 256 dimensions plus a cold store of full vectors needs both written.

Versioning and cutover

The safe pattern is blue-green indexing, shown in Figure 4 below. Tag every vector with a model version. Build the new index alongside the old one, run the new query encoder in shadow mode so you can compare results, then cut traffic over by flipping a pointer. Keep the old index until the new one has served production for a safe window, so that rollback is a configuration change rather than a six-hour rebuild.

Two details prevent incidents. The query encoder and index must always be paired: deploy them as one versioned unit, or the app will quietly query the old index with the new encoder. And ingestion must write to both indexes during the transition so documents added mid-migration are not lost.

Observability for the cutover should include the distribution of top-1 similarity scores, since a model change shifts the scale of cosine scores. Any hard-coded similarity threshold, such as “answer only if the top score exceeds 0.75,” must be recalibrated. This is among the most common silent regressions after a model swap.

Trade-offs, Gotchas, and What Goes Wrong

Fine-tuning is not free of risk, and most of the risk is in the data, not the optimiser.

Figure 4: Production lifecycle. Training outputs a versioned model; a shadow index is built and compared against the live one; cutover happens only when recall at the serving dimension and the false-negative audit both pass.

False negatives

Already discussed, and worth repeating because it is the most frequent cause of regressions. Symptoms are a training loss that plateaus high, a held-out score that improves on easy queries and degrades on queries with many equally valid answers, and mined negatives that, when read by a human, look correct. The cure is margins, rank windows, and a reranker veto.

Collapsed or degenerate embeddings

If the learning rate is too high, the scale is mis-set, or normalisation is skipped when the loss expects it, the embeddings can collapse: all vectors converge to a small region, cosine similarities cluster near one value, and the loss stalls near ln(batch size). Check for it by looking at the standard deviation of pairwise similarities on a sample; a healthy model shows a wide spread. If it is tiny, lower the learning rate, check pooling and normalisation, and re-run.

Eval contamination and the shiny-number problem

A model trained on synthetic queries, evaluated on queries from the same generator, over documents partially seen in training, will report spectacular gains. The tell is a large improvement in-domain with no improvement on the small real-query set. Trust the real queries.

Over-specialisation and the dimension trade

Matryoshka training changes the tradeoff curve: you give up a small amount of full-dimension quality for graceful truncation. Over-aggressive truncation, such as 32 or 64 dimensions, can lose rare-term discrimination exactly where your fine-tuning was meant to help. Evaluate the dimension you will serve rather than assuming that retained accuracy transfers across tasks.

Maintenance burden

Every fine-tune is a new artifact that needs a data pipeline, an evaluation harness, a registry entry, and a re-embedding job. Teams sometimes fine-tune once, ship, and then discover nobody can reproduce the run. Version the training data, the mining configuration, the base-model hash and the seed along with the model.

When not to fine-tune

If your corpus is under a few thousand chunks, hybrid search with a strong off-the-shelf model and a reranker is nearly always sufficient. If your recall problem is chunking, metadata filtering or stale content, fine-tuning will not solve it. And if you cannot afford to re-embed, which includes situations with strict re-indexing windows, consider fine-tuning a lightweight query-side adapter instead, which leaves document vectors unchanged. That approach is an active area and has real limits in expressiveness, so evaluate it carefully.

Practical Recommendations

Start with measurement, not training. Build a held-out set of at least a few hundred real queries, split by document, and record the baseline recall at 10, 50 and 100 for your current embedding model, with and without a reranker. If recall at 100 is already above 95 percent, you probably do not need a fine-tune.

If recall is the problem, generate synthetic pairs with an LLM, filter them with a round-trip check, and deduplicate. Mine one hard negative per pair from ranks roughly 10 to 50, using a margin and a cross-encoder veto, and read fifty samples by hand. Train one epoch with MNRL at the largest batch you can fit, using gradient caching if memory limits you. Wrap the loss in Matryoshka if you intend to truncate.

Compare against the base model on the held-out set at every serving dimension, with confidence intervals and per-query wins and losses. Check an out-of-domain benchmark for forgetting. Only then plan the re-embedding, blue-green cutover, and threshold recalibration.

A checklist before you ship:

Test documents are disjoint from training documents, and near-duplicate queries between splits were audited.
Recall at 100 and NDCG at 10 improved with a confidence interval that excludes zero.
Candidate recall at the truncated serving dimension was measured separately.
A sample of mined negatives was read by a human for false negatives.
Out-of-domain scores dropped by an amount you accept.
The query encoder and index are deployed as one versioned unit with a rollback path.
Similarity thresholds in application code were recalibrated against the new score distribution.
Training data, mining config, base-model identifier and seed are stored with the model.

Frequently Asked Questions

How much data do I need to fine-tune an embedding model?

A few thousand clean query-passage pairs are enough to see measurable gains on a focused domain, and gains usually flatten somewhere between ten and fifty thousand pairs. These are practical rules of thumb, not published thresholds. Quality matters more than quantity: a thousand accurate pairs with good hard negatives often beats fifty thousand noisy ones. Plot a learning curve on a held-out set before generating more.

Should I fine-tune the embedding model or add a reranker?

Diagnose first. If the right passage is in the top 100 but not the top 5, add a cross-encoder reranker. If it is missing from the top 100, the embedding space is the problem and fine-tuning is the better tool. Many production systems use both: a fine-tuned bi-encoder for recall, and a reranker for precision. Doing the reranker first is cheaper and requires no re-embedding.

What is hard negative mining and why does it help?

Hard negative mining finds passages that the current model ranks highly for a query but that do not answer it. Because softmax-based contrastive loss gives easy negatives almost no gradient, these confusable candidates carry most of the learning signal. The risk is false negatives, so skip the very top ranks, require a score margin below the positive, and use a cross-encoder to veto candidates that actually answer the query.

What does Matryoshka embeddings training do?

Matryoshka Representation Learning applies the training loss at several nested prefix sizes of the vector, so the first 256 or 512 dimensions are themselves usable embeddings. You can then store shorter vectors, or search a truncated index and rescore with full vectors. The tradeoff is a small, task-dependent change in full-dimension quality, so evaluate every dimension you plan to serve.

Do I have to re-embed my whole corpus after fine-tuning?

Yes, for a document-side change. Old and new vectors occupy different spaces, so a new query encoder cannot search old document vectors. Budget the compute, build the new index alongside the old one, cut over with a versioned pointer, and keep the old index for rollback. Recalibrate any similarity thresholds, since cosine score distributions shift after fine-tuning.

Which loss function should I use for retrieval fine-tuning?

For pairs or triplets, MultipleNegativesRankingLoss is the standard starting point in sentence-transformers because it uses in-batch negatives and supports mined hard negatives. Use the cached variant when memory limits batch size, and wrap it in MatryoshkaLoss for truncatable vectors. If you have graded relevance labels or reranker scores, distillation-style losses are an alternative worth testing.