DSPy and Programmatic Prompt Optimization: Signatures, Modules, MIPROv2 and GEPA (2026)

Most production LLM applications run on prompts that one engineer tuned by hand against a dozen examples, then froze. They work until the model is swapped, a new failure class appears, or a second prompt is added to the pipeline and the two interact in ways nobody tested. Every change is a manual search through a space of wordings, with no objective function and no regression guard.

DSPy is the most widely used framework that replaces that craft with a compile step. You declare what each LLM call must do as a typed signature, compose calls into an ordinary Python program, define a metric, and let an optimizer search for the instructions and few-shot demonstrations that maximize it. Change the model, and you re-compile instead of re-writing.

This article explains the abstractions, the compile loop, how MIPROv2 and GEPA actually search, what optimization costs, and where it fails. You should leave able to decide whether DSPy fits your system and to avoid its most common evaluation mistake.

What this covers: why hand-tuned prompts are brittle, signatures and modules, the optimizer family, MIPROv2 and GEPA mechanics, a runnable example, cost arithmetic, comparison with manual prompting and fine-tuning, and failure modes.

Context and Background

Prompt engineering started as a conversational skill and quickly became a software-maintenance problem. A single prompt string carries four jobs at once: it states the task, formats the input, specifies the output schema, and encodes lessons from past failures as extra sentences. When a pipeline has several LLM calls, such as a retriever-query rewriter, an answerer, and a verifier, the strings multiply and their quality becomes coupled. Improving the answerer prompt can degrade the verifier, and nothing in the repository tells you so.

The DSPy paper, “DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines” (Khattab et al., arXiv:2310.03714, October 2023), framed this as a missing abstraction layer. Its abstract reports that compiled pipelines on GPT-3.5 and llama2-13b-chat outperformed standard few-shot prompting by generally over 25% and 65% respectively, and outperformed pipelines using expert-written demonstrations by up to 46% and 40% on the tasks studied. Those are paper-reported results on specific benchmarks, not a guarantee for your workload, but they established the pattern: let a program search for prompts against a metric.

Since then the project has moved from research code to a maintained library. The GitHub releases page lists 3.3.1 as the latest release at the time of writing, following 3.3.0, which added an experimental dspy.Flex module that lets GEPA rewrite a module’s structure and not only its prompt text, and a ReActV2 agent module with native tool-calling history. Release 3.2.0 introduced BetterTogether chaining of multiple optimizers. Version numbers move quickly, so pin yours and read the release notes before upgrading.

DSPy is not the only approach to systematic prompting. Evaluation-first teams often build a judge-based harness first, as covered in our guide to LLM-as-a-judge evaluation, bias and calibration, and then iterate prompts manually against it. Others fine-tune. DSPy occupies the middle: it automates the prompt search while keeping the model frozen, and it can optionally hand off to weight updates later. The key prerequisite for all of these is the same, and it is not a framework: you need a metric you trust.

Why Hand-Tuned Prompts Are Brittle

Brittleness has identifiable causes, and each maps to a DSPy design decision.

Prompts conflate interface with implementation. The statement “extract the invoice total as a number” is an interface. The sentence “think step by step and ignore currency symbols, and please do not hallucinate” is an implementation detail that works for one model on one day. When both live in one string, you cannot change the second without risking the first.

Prompts are model-specific. An instruction phrasing that lifts accuracy on one model family can be neutral or harmful on another. Teams that migrate between providers for cost or latency reasons, as discussed in LLM routing and model cascades, discover that every route needs its own tuned prompt, multiplying the maintenance burden.

Prompts have no tests by default. Without a metric and a dataset, “better” means “looked better on the three examples I tried.” Hand-tuning against a handful of cases is exactly how you overfit, silently, to the examples in your head.

Multi-stage pipelines have credit-assignment problems. If the final answer is wrong, which of three prompts is at fault? The MIPRO paper (Opsahl-Ong et al., arXiv:2406.11695, EMNLP 2024) addresses this directly: it optimizes instructions and demonstrations for multi-stage language-model programs without module-level labels or gradients, using only the end-to-end metric. Its abstract reports that MIPRO outperformed baseline optimizers on five of seven diverse multi-stage programs using Llama-3-8B, with accuracy gains up to 13%.

The DSPy response is to separate concerns. The signature fixes the interface. The module fixes the inference strategy. The optimizer owns the prose. Because the prose is generated, it is treated as a build artifact, not source code you edit by hand.

The DSPy Architecture: Signatures, Modules, Metrics, Optimizers

DSPy has four core concepts. A signature declares an LLM call’s typed inputs and outputs. A module wraps a signature in an inference strategy such as direct prediction, chain-of-thought, or a tool-using ReAct loop. A metric scores outputs. An optimizer, historically called a teleprompter, searches for instructions and demonstrations that raise the metric across a dataset, producing a compiled program you save and load like any other artifact.

Figure 1: DSPy separates authoring (signatures, modules, plain Python control flow) from compilation (data, metric, optimizer). The output of compilation is saved state, not new source code.

The figure shows the one-way flow. You write the left side. The optimizer reads your program, data, and metric, and emits new instructions and demonstrations for each predictor inside the program. That emitted state is serialized and loaded in production, so serving code never calls the optimizer.

Signatures: typed contracts instead of prompt strings

A signature names the fields going in and coming out, optionally with types and short descriptions. The shorthand form is a string such as "question, context -> answer". The class form subclasses dspy.Signature, declares dspy.InputField and dspy.OutputField attributes, and uses the class docstring as the task instruction.

The framework’s adapter layer turns that declaration into a concrete prompt for whichever model is configured, and parses the completion back into typed fields. The adapter is what you give up control of and what you gain portability from. Swap the model and the adapter re-renders the same signature, so the interface is stable even though the literal prompt is not.

Typed outputs matter for pipelines. A field annotated as list[str] or a Pydantic model yields a parsed object, or a parse failure the framework can retry, instead of a string your downstream code must regex. This overlaps with the techniques in our guide to constrained decoding and structured generation, though DSPy operates at the prompt-and-parse level and does not itself mask logits.

Modules: inference strategies you can swap

A module is a parameterized callable. dspy.Predict is the plain call. dspy.ChainOfThought adds a reasoning field before the outputs. dspy.ReAct runs a thought-action-observation loop over tools you supply. dspy.ProgramOfThought and others exist for code-generating strategies. Each module that owns an LLM call contains one or more predictors, and the predictors are what optimizers tune.

The practical consequence is that changing strategy is a one-word edit. If Predict underperforms, try ChainOfThought with the same signature and compare the metric. You are no longer rewriting a prompt to add “think step by step”; you are selecting a module and letting the optimizer write the instructions that suit it.

Modules compose through ordinary Python. A pipeline is a class with a forward method that calls sub-modules, branches, loops, and calls retrievers. There is no graph DSL. This is both the strength, because anything Python can express works, and the limitation, because the optimizer sees predictors and traces, not your control-flow intent.

Metrics: the only place your judgment enters

A metric is a function that takes a gold example and a prediction and returns a score, usually a float or boolean. For extraction tasks it can be exact match or field-level F1. For open-ended generation it is often an LLM judge, which makes judge calibration part of your optimization quality, as examined in the LLM-as-a-judge guide.

Treat the metric as the specification. An optimizer is a very diligent adversary: it will find whatever the metric rewards, including shortcuts such as verbosity that fools a length-biased judge. Time spent hardening the metric returns more than time spent choosing between optimizers.

Optimizers: what the family actually does

The DSPy optimizer documentation lists the family. Condensed, with the roles verified against that page:

Optimizer	What it changes	Mechanism in one line
LabeledFewShot	Demos	Samples `k` labeled examples as demos
BootstrapFewShot	Demos	A teacher run generates demos; the metric filters them
BootstrapFewShotWithRandomSearch	Demos	Repeats bootstrapping, keeps the best program
KNNFewShot	Demos	Picks nearest-neighbor examples per input as the trainset
COPRO	Instructions	Coordinate ascent on instruction text
MIPROv2	Instructions and demos	Bayesian optimization over proposed candidates
SIMBA	Instructions or demos	Mini-batch sampling to find high-variance examples, then self-reflective rules
GEPA	Instructions	LM reflects on trajectories and evolves prompts using textual feedback
BootstrapFinetune	Weights	Distills the prompt program into fine-tuned model weights
BetterTogether	Prompts and weights	Chains prompt and weight optimization

Two orthogonal axes organize this table. The first is what is being searched: demonstrations, instructions, or weights. The second is how the search is guided: random sampling, Bayesian surrogate modeling, or language-model reflection. Choosing an optimizer is choosing a point on both axes given your data size and how rich your failure signal is.

How the Compile Loop Works

Compilation is a search loop in which every candidate evaluation is a full run of your program over examples. Understanding the loop explains both why optimization works and why it costs money: the optimizer needs program executions to score candidates, plus extra language-model calls to propose them.

Figure 2: One iteration of the compile loop. The task LM runs the program, the metric scores it, and a separate proposal or reflection LM suggests the next candidate.

Reading Figure 2 top to bottom: the optimizer runs the program on a training example, the signature is rendered into a prompt for the task model, the completion is parsed, and the optimizer records both the prediction and the trace of intermediate predictor calls. The metric scores the result. A proposal step, which differs by optimizer, generates a new candidate. The candidate is evaluated, and the best survivor is retained.

The role of traces

Traces are what make multi-stage optimization possible. For each example, DSPy records the inputs and outputs of every predictor invoked. Bootstrapping uses these traces: run the program, and if the end-to-end metric passes, keep each intermediate predictor’s input-output pair as a candidate demonstration for that predictor. You never labeled the intermediate steps; the successful trace labels them for you.

This is also the key to credit assignment. The optimizer cannot tell which module caused a failure, but it can keep demonstrations from successful runs and discard those from failures, statistically favoring modules’ behaviors that co-occur with success.

Separating the task model from the proposal model

Optimizers generally use two roles. The task model is the one you will deploy, perhaps a small cheap model. The proposal or reflection model writes candidate instructions and can be a stronger, more expensive one. In the MIPROv2 constructor, prompt_model is the model that writes instruction candidates and task_model is the one that runs the program during bootstrapping and evaluation; both default to the globally configured language model.

This split is a major economic lever. A strong model is paid for a few hundred proposal calls, while the many thousands of evaluation calls run on the cheap model you plan to serve. The result is a prompt specialized for the small model, written by the large one.

Persisting the result

A compiled program serializes to JSON with program.save("compiled.json") and loads with program.load("compiled.json") into a freshly constructed instance of the same class. The file holds the per-predictor instructions and demonstrations. Version it alongside the code and the dataset hash that produced it, because it is derived data. If the module structure changes, recompile.

MIPROv2 in Depth: Bayesian Search Over Instructions and Demonstrations

MIPROv2 (Multiprompt Instruction PRoposal Optimizer, version 2) jointly optimizes the instruction and the few-shot demonstrations of every predictor in a program. It works in three stages: bootstrap candidate demonstrations, propose candidate instructions grounded in the data and program, then run a Bayesian search over combinations, scoring them on mini-batches. Its strength is that it searches combinations rather than tuning each piece in isolation.

Stage 1: bootstrapping demonstrations

MIPROv2 first runs your program over training examples, using the trace-and-filter approach described earlier, to collect pools of demonstration sets per predictor. The constructor defaults are max_bootstrapped_demos=4 and max_labeled_demos=4, per the MIPROv2 API reference. If both are set to zero the run is zero-shot and demos are discarded, which gives an instruction-only optimizer. That mode is useful when prompt length matters or when examples contain sensitive data.

Stage 2: grounded instruction proposal

Instead of asking a model to “write a better prompt,” MIPRO grounds the proposal. According to the paper, the proposer is shown a summary of the dataset, a summary of the program’s structure, examples, and sometimes a randomly selected “tip” about prompting style, and it generates a pool of candidate instructions per predictor. Grounding gives the proposer something concrete to specialize to, such as the vocabulary of your documents or the shape of your questions.

The number of candidates per predictor is controlled by num_candidates. With auto set, that number and the trial count are determined for you; setting auto=None requires you to supply both num_candidates and num_trials yourself.

Stage 3: Bayesian search with mini-batches

With, say, 10 instruction candidates and 10 demo-set candidates for each of three predictors, the joint space has 100^3 = one million combinations. Exhaustive evaluation is impossible. MIPROv2 treats each combination as a point, evaluates a chosen point on a mini-batch, and uses a surrogate model to decide what to try next. The paper describes a surrogate built from stochastic mini-batch evaluation, and the DSPy implementation relies on a tree-structured Parzen estimator style search (an implementation detail worth confirming in the source for your installed version).

Mini-batching is the cost control. Per the API reference, minibatch defaults to True, minibatch_size defaults to 35, and minibatch_full_eval_steps defaults to 5, meaning the best candidate so far is re-evaluated on the full validation set every five trials. Noise on a 35-example batch is large, so the periodic full evaluation is what prevents the search from chasing lucky batches.

The auto presets

The auto argument accepts "light" (the default), "medium", "heavy", or None. The documentation says auto sets the run budget but, in the pages I checked, does not publish the exact candidate and trial counts each preset selects; those live in the source (dspy/teleprompt/mipro_optimizer_v2.py) and can change between releases. The optimizer guide also suggests that longer runs, on the order of 40 or more trials, suit datasets of 200 or more examples to limit overfitting. Treat the presets as budget tiers: start with light to sanity-check your metric and data plumbing, and spend medium or heavy only once the pipeline is stable.

GEPA in Depth: Reflective Prompt Evolution

GEPA (Genetic-Pareto) replaces numeric-only search with language-model reflection. Instead of treating each evaluation as a bare score, it feeds the model the full trajectory, meaning the reasoning, tool calls, outputs, and any textual feedback your metric provides, and asks it to diagnose what went wrong and rewrite the prompt. The paper claims this learns from far fewer rollouts than reinforcement learning because each rollout yields a rich natural-language learning signal, not a scalar.

What the paper reports

The GEPA paper (Agrawal et al., “GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning”, arXiv 2507.19457, submitted July 2025, revised February 2026) reports that across six tasks GEPA outperformed GRPO, a reinforcement-learning method, by 6% on average and up to 20%, while using up to 35 times fewer rollouts. It also reports gains of over 10% against MIPROv2, including a 12% accuracy gain on AIME-2025. The arXiv page does not name a peer-reviewed venue. These are the authors’ benchmark results on their chosen tasks and models, and their size on your task is an empirical question.

The mechanism

As described in the paper, GEPA maintains a pool of candidate programs. Each iteration it selects a candidate, runs it on a mini-batch, collects trajectories, and asks a reflection model to propose a revised instruction for one module based on those trajectories and feedback. The child is evaluated, and if it improves, it joins the pool. Candidate selection is Pareto-based: instead of always mutating the single best-average candidate, GEPA tracks which candidates are best on at least one individual training instance and samples from that frontier. This preserves diversity and avoids converging on a local optimum. It can also merge complementary candidates.

The metric contract: feedback is the fuel

In DSPy the GEPA metric signature is richer than a plain scorer. The GEPA API overview documents the call shape (gold, pred, trace, pred_name, pred_trace, program_trace=None), returning either a float or a dspy.Prediction(score=..., feedback=...). If you return only a float, GEPA falls back to a score-only message, and you lose most of its advantage.

Good feedback is specific and diagnostic. “Wrong” is useless. “The answer cites document 3, but the claim appears only in document 1; the retriever returned the right documents, so the failure is in the answer module” tells the reflection model which instruction to change. The pred_name argument tells your metric which predictor is being reflected on, so you can return module-specific feedback in a multi-stage pipeline.

The overview page also shows multi-objective scoring: returning objective_scores={"quality": ..., "privacy": ...} together with gepa_kwargs={"frontier_type": "objective"} keeps a Pareto frontier across named objectives. Release notes for 3.3.1 add parallel multi-candidate GEPA evaluation, which matters for wall-clock time on slow pipelines. The reflection model is passed as reflection_lm; confirm the current argument names and budget options (such as auto presets and metric-call limits) in the GEPA documentation for your installed version, since I could not confirm every constructor argument against a single canonical page.

Where GEPA’s structure edge is going

Release 3.3.0 introduced an experimental dspy.Flex module that allows GEPA to rewrite a module’s structure, not only its prompt text. This is new and marked experimental, so treat it as a research feature. It points at the direction of travel: from tuning strings toward evolving programs.

Choosing an Optimizer

No optimizer dominates. The right choice depends on three inputs: how many labeled examples you have, whether your metric can explain failures in words, and whether you ultimately want prompt-only or weight updates.

Figure 3: A practical selection path for DSPy prompt optimization, starting from dataset size and metric richness.

The documentation gives rough data-size guidance: with around 10 examples use BootstrapFewShot; with 50 or more consider BootstrapFewShotWithRandomSearch; use MIPROv2 for instruction optimization, with longer runs on 200 or more examples; and consider BootstrapFinetune when a smaller model of around 7B parameters or more suffices and you want efficiency. Those thresholds are guidance, not laws. A hard task with 30 examples and a noisy judge can overfit any optimizer.

Decision matrix

Situation	First choice	Why	Watch out for
Under 20 examples, quick win	BootstrapFewShot	Cheap, demos only, little to overfit	Gains plateau fast
50-200 examples, scalar metric	MIPROv2 light then medium	Searches instruction and demo combinations	Noisy mini-batch scores
Metric can emit diagnostic text	GEPA	Learns from trajectories, strong with few rollouts	Reflection LM cost; feedback quality
Zero-shot required (long inputs, privacy)	MIPROv2 with zero demos	Instruction-only output	Smaller ceiling
Need cheaper serving	BootstrapFinetune or BetterTogether	Moves behavior into weights	Training infrastructure, data volume
Agentic ReAct pipeline	GEPA or MIPROv2 on ReAct	Optimizes tool-use instructions	Long traces inflate cost

For agents, the evaluation question is harder because correctness depends on trajectories, not final strings. Our guide to agent trajectory evaluation and pass^k reliability covers building metrics that an agent optimizer can safely maximize.

A Runnable Example: Optimizing a Support-Ticket Classifier and Summarizer

The example below builds a two-stage pipeline: classify an incoming ticket, then draft a one-line summary conditioned on the class. It uses a small dataset, an exact-match-plus-keyword metric, and MIPROv2. Model identifiers are examples; substitute any provider LiteLLM-style identifier your account supports. Install with pip install dspy and set your provider key as an environment variable.

import dspy
from dspy.evaluate import Evaluate

# 1. Models: cheap task model, stronger model to write instructions.
task_lm = dspy.LM("openai/gpt-4o-mini", temperature=0.0)
prompt_lm = dspy.LM("openai/gpt-4o", temperature=1.0)
dspy.configure(lm=task_lm)

# 2. Signatures: typed contracts, no prompt prose.
class Classify(dspy.Signature):
    """Classify the support ticket into exactly one category."""
    ticket: str = dspy.InputField()
    category: str = dspy.OutputField(desc="billing, bug, access, or other")

class Summarize(dspy.Signature):
    """Write a one-sentence summary an on-call engineer can act on."""
    ticket: str = dspy.InputField()
    category: str = dspy.InputField()
    summary: str = dspy.OutputField()

# 3. Program: ordinary Python composing modules.
class TicketPipeline(dspy.Module):
    def __init__(self):
        super().__init__()
        self.classify = dspy.ChainOfThought(Classify)
        self.summarize = dspy.ChainOfThought(Summarize)

    def forward(self, ticket):
        cat = self.classify(ticket=ticket).category.strip().lower()
        out = self.summarize(ticket=ticket, category=cat)
        return dspy.Prediction(category=cat, summary=out.summary)

# 4. Data: split BEFORE optimizing. Replace with your own labeled rows.
rows = [
    {"ticket": "I was charged twice for March", "category": "billing", "must_mention": "charge"},
    {"ticket": "App crashes when I upload a PDF", "category": "bug", "must_mention": "upload"},
    # ... 150+ more labeled rows in a real run ...
]
examples = [dspy.Example(**r).with_inputs("ticket") for r in rows]
n = len(examples)
train = examples[: int(n * 0.5)]
dev = examples[int(n * 0.5): int(n * 0.8)]
test = examples[int(n * 0.8):]

# 5. Metric: score plus a boolean-friendly float.
def metric(gold, pred, trace=None):
    cat_ok = pred.category == gold.category
    mention_ok = gold.must_mention.lower() in pred.summary.lower()
    return 0.7 * cat_ok + 0.3 * mention_ok

# 6. Baseline on the held-out test set BEFORE any optimization.
evaluate = Evaluate(devset=test, metric=metric, num_threads=16, display_progress=False)
baseline = evaluate(TicketPipeline())

# 7. Compile with MIPROv2.
optimizer = dspy.MIPROv2(
    metric=metric,
    prompt_model=prompt_lm,
    task_model=task_lm,
    auto="light",
)
compiled = optimizer.compile(
    TicketPipeline(),
    trainset=train,
    valset=dev,
    max_bootstrapped_demos=3,
    max_labeled_demos=3,
    requires_permission_to_run=False,
)

# 8. Measure the held-out result and persist the artifact.
after = evaluate(compiled)
print("baseline", baseline, "compiled", after)
compiled.save("ticket_pipeline_v1.json")

A few details deserve attention. The module is constructed fresh in compile, which returns an optimized copy and leaves the original untouched. The valset is what the optimizer uses to pick winners, which is why a separate test split must exist; otherwise your reported number is the number the optimizer maximized. And requires_permission_to_run=False suppresses an interactive cost confirmation, so do not set it in notebooks where you have not estimated spend.

To use GEPA on the same program, swap the optimizer and upgrade the metric so it explains failures:

def metric_with_feedback(gold, pred, trace=None, pred_name=None, pred_trace=None):
    cat_ok = pred.category == gold.category
    mention_ok = gold.must_mention.lower() in pred.summary.lower()
    score = 0.7 * cat_ok + 0.3 * mention_ok
    notes = []
    if not cat_ok:
        notes.append(f"Category was {pred.category} but ticket is {gold.category}. "
                     "Check whether payment words should map to billing.")
    if not mention_ok:
        notes.append(f"Summary omitted the key detail: {gold.must_mention}.")
    return dspy.Prediction(score=score, feedback=" ".join(notes) or "Correct.")

gepa = dspy.GEPA(metric=metric_with_feedback, reflection_lm=prompt_lm, track_stats=True)
compiled_gepa = gepa.compile(TicketPipeline(), trainset=train, valset=dev)

The reflection_lm and budget arguments are the ones I flagged earlier as needing confirmation against your installed version; if the constructor rejects a budget option, consult the GEPA documentation or run help(dspy.GEPA). Notice that the feedback names the failing predictor’s concern. In a deployment, feedback can include judge rationales, unit-test output, schema-validation errors, or retrieval diagnostics.

What Optimization Costs: Worked Arithmetic

Optimization cost scales with program executions and tokens per execution. The optimizer documentation says a typical simple optimization run costs on the order of $2 and takes around ten minutes, and that costs range from a few cents to tens of dollars depending on the model, dataset, and configuration. The arithmetic below shows how to estimate your own. All prices and counts are illustrative; plug in your provider’s current rates.

A budget model

Let a program have P predictors and a typical end-to-end run consume T tokens (input plus output across all predictors). A MIPROv2-style run performs these workloads:

Bootstrap: roughly one program run per training example sampled for demos.
Proposal: several proposal calls per predictor per candidate, on the stronger model.
Trials: each trial evaluates a candidate on a mini-batch of size B; every k-th trial runs a full dev evaluation of size D.

Illustrative numbers: P = 2, T = 1,500 tokens per run, 100 bootstrap runs, 30 trials, B = 35, D = 100, full evaluation every 5 trials. Trial runs = 30 x 35 = 1,050. Full evaluations = 6 x 100 = 600. Bootstrap = 100. Total task-model program runs are about 1,750. At 1,500 tokens each, that is 2.6 million tokens.

If the cheap task model costs an illustrative $0.40 per million blended tokens, evaluation costs about $1.05. Add proposal calls: 2 predictors x 10 candidates x about 3,000 tokens each is 60,000 tokens on a strong model at an illustrative $5 per million blended, or $0.30. The order of magnitude, around $1-2, matches the documentation’s “on the order of $2” statement, which is why the table below is a sanity check and not a prediction.

Component	Count	Tokens each	Total tokens	Illustrative price/M	Cost
Bootstrap runs	100	1,500	150,000	$0.40	$0.06
Trial mini-batches	1,050	1,500	1,575,000	$0.40	$0.63
Full dev evals	600	1,500	900,000	$0.40	$0.36
Proposal calls	20	3,000	60,000	$5.00	$0.30
Total			2,685,000		about $1.35

What blows the budget

Cost grows linearly in T and in dataset size, and multiplies when you use a large task model, a ReAct agent whose single run makes 10 to 20 tool calls, or an LLM-judge metric that adds a second call per evaluation. The judge multiplier is the quiet one: a metric that calls a strong model per example can cost more than the program it scores. Cache aggressively; DSPy caches LM calls by default so repeated evaluation of an identical prompt and input does not pay twice, which is also why temperature settings and cache configuration affect measured variance.

GEPA’s budget logic differs. Its claim is fewer rollouts per unit of improvement, but each iteration adds reflection calls with long contexts, because trajectories and feedback are included in the prompt. Compare optimizers at equal dollar cost and equal wall-clock time on your own task, not at equal trial counts.

Optimization cost versus serving savings

The payoff is amortized. Suppose a compiled program lets you replace a larger model with a smaller one at the same quality, cutting serving cost by an illustrative 60% on a workload of 5 million requests per month. A one-time spend of tens of dollars on optimization is negligible next to the monthly delta. The reverse also holds: if your workload serves a few thousand requests a month, a day of engineer time spent on optimization may exceed any savings, and the case for DSPy rests on quality and maintainability instead.

DSPy Versus Manual Prompting Versus Fine-Tuning

These three approaches are not exclusive, and the honest comparison is about what each costs you per unit of quality gained. Manual prompting is cheapest to start and most expensive to maintain. Programmatic optimization has a setup cost in data and metric work and then makes re-tuning a command. Fine-tuning has the highest ceiling for narrow tasks and the highest operational burden.

Dimension	Manual prompting	DSPy optimization	Fine-tuning
Upfront effort	Low	Medium: data, metric, program	High: data, training infra
Labeled data needed	A handful for spot checks	Tens to hundreds	Hundreds to thousands or more
Model swap cost	Re-write and re-test by hand	Re-compile	Re-train
Handles multi-stage pipelines	Poorly	Designed for it	Per-stage, costly
Interpretability of result	High, you wrote it	Medium, generated text is readable	Low, weights
Latency at serving	Prompt length dependent	Prompt length dependent	Often shorter prompts
Risk of overfitting	Hidden	Visible if you hold out data	Visible if you hold out data

Prompt optimization and fine-tuning sit on a continuum, which is why BetterTogether exists as a meta-optimizer that chains prompt and weight optimization, and why BootstrapFinetune distills a prompt-based program into weights while keeping the same program steps. A common progression is to optimize prompts first, because it is cheap and yields a labeled-trace dataset as a by-product, and to fine-tune only once volume justifies it. If you do fine-tune, our notes on synthetic data, distillation, and quality filtering cover the data hygiene that determines whether the traces are good training material.

When the problem is retrieval, not generation, neither prompt optimization nor weight tuning will rescue you. If your RAG pipeline returns the wrong documents, an optimized answer prompt will only produce more fluent errors. Optimize retrieval budgets and quality first, as covered in the RAG latency budget guide, then optimize the generation modules.

DSPy also differs from agent frameworks that focus on orchestration. A framework comparison such as LangGraph versus CrewAI versus the Agents SDK versus PydanticAI addresses how to route control and state between agents. DSPy addresses how each LLM call’s prompt is produced. They are complementary: DSPy programs can sit inside a graph node, and a graph’s nodes can be DSPy modules, though you should check how your orchestrator passes traces if you plan to optimize end to end.

Evaluation Pitfalls: Overfitting the Dev Set

The most common failure with programmatic optimization is reporting a number the optimizer itself maximized. An optimizer with hundreds of trials and mini-batch selection is a powerful overfitting machine. The remedy is a three-way data split with discipline about which split is touched when.

Figure 4: The train split generates demos and reflections, the dev split selects winners, and the test split is evaluated once to estimate real performance.

Why dev-set overfitting happens here

Three mechanisms compound. First, selection bias: when you pick the best of many candidates by dev score, the winner’s dev score is optimistically biased even if candidates are equally good, an effect that grows with the number of trials. Second, demonstration leakage: bootstrapped demos come from the training split, and if train and dev contain near-duplicate tickets, queries, or documents, the demos effectively contain the answers. Third, reflection leakage in GEPA: textual feedback derived from specific training failures can bake those failures’ specifics into the instruction, such as a hard-coded product name, which helps on train and does nothing on new data.

The documentation’s guidance of 200 or more examples for long MIPROv2 runs reflects this: more data per trial budget means less selection bias. With fewer examples, prefer shorter runs and simpler optimizers.

A defensible protocol

Deduplicate and group-split first. Split by customer, document, or session, not by row, so related items stay on the same side.
Freeze a test split before any optimization and keep it out of every notebook that touches the optimizer.
Measure the un-optimized baseline on test, so the reported gain is relative to the right reference.
Record the dev-minus-test gap. A large gap signals overfitting; a small one is necessary but not sufficient evidence.
Re-run with a different random split or seed; if the gain disappears, it was noise.
Keep a regression suite of known-hard cases, and re-check it on every re-compile.

Judge-based metrics introduce a further trap. If the same model family writes outputs and judges them, self-preference can inflate scores, and an optimizer will amplify whatever the judge likes. Calibrate the judge against human labels on a sample, and consider a different judge model than the task model, per the bias and calibration guidance in our LLM-as-a-judge article.

Trade-offs, Gotchas, and What Goes Wrong

You trade prompt visibility for automation. The generated instructions are readable but not designed; they may be verbose, odd, or include phrasing no engineer would write. Review them before release, especially for safety-relevant behavior. Do not assume that an optimized prompt preserves guardrails you implied in the original: the optimizer only preserves what the metric measures. If refusal behavior matters, put it in the metric or in a separate filter, and test adversarial inputs as in our guide to LLM red teaming and guardrails.

Compiled programs are tied to a model. The artifact is tuned to the task model it saw. Changing the model without re-compiling can lose much of the gain, and sometimes worsens results relative to a plain baseline. Treat the model name as part of the artifact’s version.

Adapters and parsing are a hidden surface. Typed outputs depend on the adapter’s formatting and parsing behavior. Version upgrades can shift how fields are rendered. Release 3.3.0 also shipped a breaking change requiring explicit factories for Image, Audio, and File types. Pin DSPy, run your regression suite on upgrades, and keep the compiled JSON alongside the version that produced it.

Poisoned or leaky data flows into prompts. Demonstrations are stored in plain text in the compiled artifact. If training tickets contain personal data, that data now sits in a JSON file and in every production prompt. Scrub the training set, for example with the approach in our guide to PII detection and redaction pipelines, or compile zero-shot.

Noise masquerades as gain. LLM outputs vary even at low temperature across providers and caches. A 2-point improvement on 100 examples is inside noise for many metrics. Report confidence intervals or paired comparisons on the same examples, and re-run the baseline.

Optimizers game weak metrics. A metric that rewards keyword presence will produce summaries stuffed with the keyword. Inspect a sample of high-scoring outputs by hand every time you optimize. The better your metric encodes what users want, the less this happens, and no optimizer can fix a metric that is wrong.

Observability still matters. The compile step does not replace production monitoring. Trace each predictor call so you can see which stage degrades after a model update, using the conventions in our write-up of OpenTelemetry GenAI semantic conventions for LLM and agent observability.

Practical Recommendations

Adopt DSPy when you have at least a few dozen labeled examples, a metric you trust, and either a multi-stage pipeline or a plan to swap models. Do not adopt it to avoid thinking about evaluation: the framework relocates the hard work into the dataset and metric, where it belongs, and it will not rescue a task whose success criterion you cannot state.

Start small. Port one prompt to a signature and measure the un-optimized module against your current string prompt; this alone verifies that the adapter does not regress behavior. Then build the dev and test splits, run BootstrapFewShot for a cheap first gain, and only then escalate to MIPROv2 or GEPA. Escalate to GEPA when you can write feedback that explains failures; escalate to MIPROv2 when you have a clean scalar metric and more data. Spend the proposal-model budget on the strongest model you can afford and the task-model budget on the model you will serve.

Checklist before shipping a compiled program:

[ ] Data split by group, with a frozen test set that never touched the optimizer.
[ ] Baseline and compiled scores reported on the test set, with the dev-test gap recorded.
[ ] Metric audited by reading at least 30 high-scoring and 30 low-scoring outputs.
[ ] Generated instructions reviewed by a human for safety, tone, and leaked specifics.
[ ] Training data scrubbed of personal or secret data, since demos are stored in plain text.
[ ] DSPy version, task model, proposal model, dataset hash, and optimizer settings stored with the artifact.
[ ] Regression suite and per-predictor tracing in place, with a re-compile trigger for model changes.
[ ] Optimization cost measured and compared with expected serving savings.

Frequently Asked Questions

What is DSPy used for?

DSPy is a Python framework for building LLM applications as programs of typed modules and then automatically optimizing the prompts and few-shot examples inside them against a metric. Teams use it for multi-stage pipelines such as RAG, classification-plus-generation flows, and agents, particularly when they expect to change models or keep improving quality over time and want re-tuning to be a repeatable command, not a manual rewrite.

What is the difference between MIPROv2 and GEPA?

MIPROv2 proposes grounded instruction and demonstration candidates and uses Bayesian optimization with mini-batch scoring to find the best combination, driven by a scalar metric. GEPA instead reflects on full trajectories and textual feedback to rewrite prompts and maintains a Pareto frontier of candidates. GEPA’s paper reports over 10% gains against MIPROv2 on its benchmarks, but results vary by task and by feedback quality.

How much data do I need to optimize prompts with DSPy?

The documentation suggests roughly 10 examples can support BootstrapFewShot, 50 or more suit BootstrapFewShotWithRandomSearch, and longer MIPROv2 runs of about 40 trials or more fit 200 or more examples to limit overfitting. Some runs work with 5 to 10 training inputs. Whatever the size, reserve a held-out test split so you can measure real generalization instead of the dev score that the optimizer maximized.

How much does DSPy optimization cost?

The official documentation says a typical simple run costs on the order of $2 and takes about ten minutes, with a range from a few cents to tens of dollars. Cost scales with the number of program runs, tokens per run, the models used, and any LLM-judge metric. Estimate it as trials times mini-batch size times tokens per run, plus proposal or reflection calls, then check the first run’s actual usage.

Does DSPy replace fine-tuning?

No. DSPy optimizes prompts and demonstrations while the model stays frozen, which is cheap and fast to iterate. Fine-tuning changes weights and can lower serving latency and cost for narrow, high-volume tasks. DSPy includes BootstrapFinetune and BetterTogether to bridge the two: optimize the prompt program first, then distill into weights if volume justifies the operational cost of training and hosting.

Can I use DSPy with any LLM provider?

DSPy configures language models through dspy.LM using provider-prefixed model identifiers, and it supports hosted APIs and local servers. The release notes mention a move toward a typed, provider-neutral LM boundary and decoupling from LiteLLM, so check the documentation for your version. Compiled programs remain tied to the model they were optimized on, so re-compile after changing the model.