Prompt Version Management for LLM Apps: Registries, Eval Gates and Safe Rollouts

The most expensive LLM regression most teams ship is not a bad prompt. It is a prompt that was fine yesterday, paired with a model that changed underneath it, behind a config nobody remembers editing, in an environment where the only audit trail is a Slack thread. Someone tweaks one sentence in a system prompt, quality dips on a customer segment nobody sampled, and the rollback is a guessing game because “the previous version” was never a defined thing.

This matters now because LLM features have moved from demos to revenue paths, and the moving parts have multiplied: prompts, model identifiers, sampling parameters, tool schemas, retrieval settings and output validators all change on different clocks. Prompt version management therefore has to mean managing a whole release, not a text file.

You will leave with a concrete design: an immutable release bundle that pins prompt, model and parameters together, a registry with movable environment pointers, a statistical eval gate you can run in CI, and a canary rollout with automatic rollback. Code is included and runnable.

What this covers: why prompts alone are the wrong unit, the bundle and registry architecture, model pinning and deprecation calendars, eval gates with confidence intervals, shadow and canary rollouts, rollback mechanics, and the failure modes that still bite.

Context and Background

Software teams solved most of this problem for code a decade ago: immutable artifacts, a registry, promotion between environments, automated tests as a gate, and progressive delivery with a fast revert. LLM applications inherit the problem but break two assumptions that make the code playbook work. The first is determinism: the same input does not produce the same output, so “the tests pass” is a statistical statement, not a boolean. The second is that the largest dependency, the model, is hosted by someone else and has its own lifecycle.

Tooling has converged on the registry idea. MLflow’s prompt registry stores prompts as versioned objects, describes existing versions as immutable, lets each new version carry a commit message, and supports aliases such as production that point at a version; applications load a prompt through a URI of the form prompts:/<name>@<alias>, and a reserved @latest alias tracks the newest version (MLflow prompt registry documentation). Observability vendors and gateways ship comparable features under different names. Our earlier post on MLflow 3 experiment tracking, model registry and GenAI tracing covers that platform in general, and the existing overview of prompt management and versioning for LLM apps introduces the registry-gate-rollout loop. This post takes a narrower and, I think, more useful angle: the thing you version and promote is the release bundle, and everything else follows from that.

The second assumption matters because model providers publish lifecycle rules you can plan against. Anthropic’s documentation states that each Claude model ID identifies a pinned version whose underlying model remains constant for the lifetime of that ID, that updated versions ship under new IDs, and that every ID has its own deprecation and retirement schedule. It also notes that convenience aliases exist for some earlier models and resolve to the most recent dated snapshot for that minor version, so an alias can move under you (Anthropic model IDs and versions). The same page cautions that infrastructure updates can occasionally produce minor behavioural differences even on a stable ID. OpenAI’s deprecations page lists shutdown dates with recommended replacements, describes minimum notice periods by model category, and recommends reading a shutdown_date field from the models endpoint to check programmatically (OpenAI deprecations). Dates and model names on those pages change often; treat any specific model I mention as an example, not a recommendation, and re-check the source before you plan around it.

Put those two facts together and the implication is plain. A prompt version is only meaningful relative to a model ID and a parameter set. Version the text alone and you have recorded a third of the experiment. That is the gap this post closes.

The Release Bundle: One Immutable Unit for Prompt, Model and Config

A release bundle is an immutable, content-addressed record that pins every input that determines an LLM call’s behaviour: prompt templates, model identifier, sampling parameters, tool and output schemas, and retrieval configuration. You promote and roll back bundles, never individual prompts, so any production answer can be reproduced from one bundle hash.

Figure 1: A release bundle ties prompt, model pin, parameters, schemas and retrieval config under one hash; environment aliases point at bundles.

Figure 1 shows the shape. Authors edit source files in git. A build step assembles a bundle, computes a hash over its canonical serialisation, and registers it in a registry. Environments such as dev, staging and prod are aliases, small mutable pointers that name a bundle hash. The application resolves its alias at startup and on a refresh interval, and stamps the bundle hash onto every trace it emits. The registry is therefore the only mutable thing in the system, and its mutations are tiny, auditable and reversible.

What goes inside the bundle

The temptation is to version the prompt template and call it done. Resist it, because the following inputs each change output quality as much as a rewritten sentence does.

The model identifier is the obvious one, and it must be a pinned ID rather than a floating alias. Sampling parameters come next: temperature, top-p, maximum output tokens, stop sequences and any reasoning-effort or thinking-budget setting the provider exposes. A change from 0.2 to 0.7 temperature is a behavioural release even if no word of the prompt changed.

Tool definitions and structured-output schemas belong inside the bundle too. If your prompt tells the model to call lookup_order and the tool schema renames a field, the pair is broken, and only a bundle that versions them together can catch it. The same logic applies to retrieval configuration: chunk size, top-k, reranker choice and the index version. An answer that was grounded last week may be ungrounded after a re-index, and the prompt never changed.

Finally, include the evaluation contract: which dataset version and which metric thresholds this bundle was promoted against. That turns the bundle into a self-describing claim, “this configuration met these bars on this data”, which is far more useful in an incident than a bare version number.

Content addressing beats counters

Registries typically hand out sequential integers: version 1, 2, 3. Counters are convenient for humans and treacherous for machines, because two branches can each create “version 7” with different content, and a counter says nothing about whether two bundles are identical. Hash the canonical JSON of the bundle and use the first twelve hex characters as the identity, with a human-friendly counter or semantic tag as a label on top.

Content addressing gives you three properties for free. Deduplication: re-registering an identical bundle returns the existing record rather than a spurious new version. Tamper evidence: if the bundle in the registry no longer hashes to its ID, something edited it. And cache correctness: eval results, shadow comparisons and prompt-cache warm-ups can key on the hash, so a result can never silently attach to the wrong configuration.

Here is a minimal, dependency-free implementation of the bundle and an in-memory registry. It is deliberately small so the semantics are visible; the same logic maps onto MLflow, a database table, or a git repository.

import hashlib
import json
from dataclasses import dataclass, field, asdict
from typing import Optional


@dataclass(frozen=True)
class Bundle:
    name: str
    prompt: dict          # e.g. {"system": "...", "user": "..."}
    model: str            # pinned model ID, never a floating alias
    params: dict          # temperature, max_tokens, ...
    schemas: dict = field(default_factory=dict)   # tools / output schema
    retrieval: dict = field(default_factory=dict)
    eval_contract: dict = field(default_factory=dict)

    def canonical(self) -> str:
        return json.dumps(asdict(self), sort_keys=True, separators=(",", ":"))

    @property
    def bundle_id(self) -> str:
        return hashlib.sha256(self.canonical().encode()).hexdigest()[:12]


class Registry:
    def __init__(self):
        self._bundles: dict[str, Bundle] = {}
        self._aliases: dict[tuple[str, str], str] = {}
        self._history: list[tuple[str, str, Optional[str], str]] = []

    def register(self, b: Bundle) -> str:
        self._bundles.setdefault(b.bundle_id, b)   # idempotent
        return b.bundle_id

    def set_alias(self, name: str, alias: str, bundle_id: str, actor: str):
        if bundle_id not in self._bundles:
            raise KeyError(f"unknown bundle {bundle_id}")
        prev = self._aliases.get((name, alias))
        self._aliases[(name, alias)] = bundle_id
        self._history.append((f"{name}@{alias}", bundle_id, prev, actor))

    def resolve(self, name: str, alias: str) -> Bundle:
        return self._bundles[self._aliases[(name, alias)]]

    def rollback(self, name: str, alias: str, actor: str):
        # last history entry for this pointer that has a previous value
        for key, _, prev, _ in reversed(self._history):
            if key == f"{name}@{alias}" and prev:
                self.set_alias(name, alias, prev, actor)
                return prev
        raise RuntimeError("nothing to roll back to")

Note what the alias history records: pointer, new target, previous target and actor. Rollback is then a pointer move to a known previous value, not a rebuild. That single design choice is what makes recovery take seconds.

Aliases as environments, not branches

Treat aliases as environments with promotion rules, not as long-lived branches. staging should only ever point at a bundle that passed the CI gate. prod should only ever point at a bundle that was staging first and survived a rollout. Encoding those rules in the registry’s set_alias call, rather than in a wiki page, is the difference between a policy and a hope. A further refinement is a prod-previous pointer maintained automatically on every promotion, so a human under pressure never has to look up which bundle to restore.

MLflow’s reserved @latest alias illustrates the danger of the opposite approach. It is useful in development, where you want the newest draft, and dangerous in production, where an unreviewed version would go live on its next refresh. A good convention is that production code paths are forbidden from resolving latest, enforced by a lint rule or a startup assertion.

Pin models, and keep a deprecation calendar

Model pinning deserves its own discipline because it is the one input you do not control. Pin to the most specific identifier the provider offers, and record it in the bundle. Where a provider offers both a floating alias and a pinned snapshot, the alias is a convenience for experimentation; the Anthropic documentation is explicit that an alias for earlier models resolves to the latest dated snapshot, which means your behaviour can change without a deploy.

Pinning buys stability but not permanence. Every provider retires models. The OpenAI deprecations page, for example, publishes announcement dates, shutdown dates and recommended replacements in tables, and lists an alias with its dated snapshots in the same row sharing one shutdown date. Anthropic keeps a separate deprecations page with per-ID schedules. The practical response is a small job that runs weekly, reads each provider’s lifecycle information (the OpenAI page recommends the shutdown_date field returned by the models endpoint), compares it against the model IDs in every bundle currently aliased in any environment, and opens a ticket when a pinned model is within a configurable horizon, say 120 days, of shutdown. That ticket is the trigger for a migration bundle that goes through the same gate and rollout as any other change.

Treat a model upgrade as a release of its own, never as a bundled side effect of a prompt edit. When both change in one bundle and quality moves, you cannot attribute the cause. Change one axis per release whenever you can.

Eval Gates in CI: Turning “Looks Fine” into a Statistical Claim

An LLM eval gate is a CI check that runs a candidate bundle and the current baseline over a versioned golden dataset, scores both, and blocks promotion unless the paired score difference clears a regression margin with statistical confidence. It replaces eyeballing a few outputs with a repeatable, auditable decision.

Figure 2: The gate runs candidate and baseline on the same golden set, combines deterministic and judge scores, and decides on a confidence interval.

Figure 2 shows the flow. The key design choice is that candidate and baseline are evaluated on the same items, so you can compare them as pairs. Paired comparison removes item difficulty from the noise: a hard question is hard for both bundles, and only the difference matters. That typically lets a modest dataset detect regressions an unpaired comparison would miss.

Layer the checks, cheapest first

Run deterministic checks before any judge call, because they are fast, free and unambiguous. Does the output parse against the JSON schema? Did the model call the expected tool with valid arguments? Is the answer within length limits? Does it avoid forbidden strings such as leaked system-prompt text? A bundle that fails a hard check should fail immediately, with no judge spend.

Judge-based scoring comes second, for qualities no regex can see: faithfulness to retrieved context, helpfulness, tone, policy compliance. Judges have well-known biases, including position bias in pairwise comparisons and a preference for longer answers, and they need calibration against human labels. Our deep dive on LLM-as-a-judge evaluation, bias, calibration and pairwise rubrics covers how to build one you can trust. For a gate, the minimum viable discipline is to randomise answer order in pairwise comparisons, pin the judge model in the bundle’s eval contract, and re-validate the judge whenever you change it.

A third layer, often forgotten, is a small set of human-reviewed canary cases: ten to thirty examples that encode past incidents. Each time production burns you, add the failing input to this set. It becomes institutional memory that no amount of generic benchmarking replaces.

Compute a confidence interval, not just a mean

A mean score that is 0.8 points lower on a 100-point scale means nothing without a sense of the noise. The standard tool is a paired bootstrap: resample the per-item score differences with replacement many times, compute the mean each time, and read the interval off the percentiles. Block promotion when the lower bound of the interval on the difference (candidate minus baseline) falls below your allowed regression margin. A separate rule can demand a lower bound above zero for changes whose entire purpose is improvement.

import random
import statistics
from typing import Sequence


def paired_bootstrap(cand: Sequence[float], base: Sequence[float],
                     n_boot: int = 5000, alpha: float = 0.05, seed: int = 7):
    assert len(cand) == len(base) and cand, "need paired, non-empty scores"
    diffs = [c - b for c, b in zip(cand, base)]
    rng = random.Random(seed)
    means = []
    for _ in range(n_boot):
        sample = [diffs[rng.randrange(len(diffs))] for _ in diffs]
        means.append(statistics.fmean(sample))
    means.sort()
    lo = means[int((alpha / 2) * n_boot)]
    hi = means[int((1 - alpha / 2) * n_boot) - 1]
    return statistics.fmean(diffs), lo, hi


def gate(cand, base, margin: float = -0.02):
    """Scores in [0, 1]. Pass if the CI lower bound is above the margin."""
    mean, lo, hi = paired_bootstrap(cand, base)
    ok = lo > margin
    return {"mean_diff": round(mean, 4), "ci": (round(lo, 4), round(hi, 4)),
            "margin": margin, "pass": ok}


if __name__ == "__main__":
    # Illustrative synthetic scores, not real benchmark data.
    rng = random.Random(1)
    base = [min(1, max(0, rng.gauss(0.80, 0.12))) for _ in range(300)]
    cand = [min(1, max(0, b + rng.gauss(0.005, 0.06))) for b in base]
    print(gate(cand, base))

The margin is a product decision, not a statistical one. A customer-support bot might tolerate a two-point dip in a style metric for a large cost saving, while a medical-triage assistant tolerates none on safety rules. Write the margins into the bundle’s eval contract so they are versioned with the thing they judge. The synthetic data above exists only to make the script runnable; it says nothing about any real system.

Dataset hygiene and the CI wiring

The golden set is itself a versioned artifact. Store it with a hash, record that hash in the eval contract, and treat edits as reviewable changes, because a dataset that quietly gets easier lets every bundle pass. Stratify it by the segments that matter to your product, such as language, customer tier or document type, and report per-stratum results. An aggregate improvement that hides a 10-point drop for one language is a regression for that language’s users.

Cost and latency control keep the gate usable. Cache model outputs keyed by the pair of bundle hash and item ID, so re-running CI after an unrelated change does not repay for generations. Run the full set on merge to main and a smaller sample on each pull request, with a nightly full run that catches drift in the judge or in the provider. Because sampling is stochastic, run each item more than once where the budget allows, or set temperature low in the eval profile and accept that you measure the configured behaviour rather than the sampling tail.

In CI, the job is a few steps: register the candidate bundle, run the harness, upload the report as a build artifact, and set a status check. Only a passing check allows the registry’s set_alias call for staging. This is where the policy enforcement described earlier pays off: the registry refuses the pointer move unless it can find a passing eval record for that exact bundle hash.

One caution on what the gate proves. It proves the candidate did not regress on your dataset. It does not prove the dataset reflects tomorrow’s traffic. That is the job of the rollout stage.

Safe Rollouts: Shadow, Canary and Rollback for LLM Apps

An LLM canary rollout sends a small, sticky fraction of real traffic to the new bundle while guardrail metrics are compared against the baseline, widening the fraction in steps and moving the alias back automatically on a breach. Shadow mode comes first: the new bundle runs on mirrored traffic and its answers are never shown to users.

Figure 3: Rollout stages. Each widening step requires guardrails to hold; any breach moves the alias back to the previous bundle.

The gate tested a dataset; the rollout tests reality. Real traffic contains inputs your golden set never imagined, user behaviour that reacts to the new answers, and load patterns that expose latency and cost differences. Progressive delivery for LLM apps follows the same logic as for services, and our post on feature flag architecture, OpenFeature and progressive delivery covers the evaluation and targeting machinery in depth. What differs is the metrics you watch and the cost of an observation.

Shadow mode: free evidence, paid tokens

In shadow mode the application sends each eligible request to both the production bundle and the candidate. Users receive only the production answer. The candidate’s answer, latency, token counts and tool calls are logged and compared offline. Shadow is excellent for catching format failures, tool-call errors, refusals and latency regressions on real inputs, with zero user risk.

It has two limits. First, it doubles the model spend for the sampled slice, so sample a fraction rather than everything. Second, it cannot observe anything that depends on the user’s reaction, such as follow-up questions, thumbs-down clicks or abandoned sessions. Shadow also needs care with side effects: if the candidate can call tools that write data, shadow runs must use stubbed or read-only tools, or you will perform every action twice.

Canary with sticky bucketing

For the live canary, assignment must be deterministic and sticky. A user who sees the new behaviour on one request and the old on the next gets an incoherent experience, and your per-user metrics are contaminated. Hash a stable identifier together with the experiment name and map it to a bucket. Salting with the experiment name matters: without it, the same users would land in the canary slice for every experiment you run, which concentrates risk on them.

import hashlib


def bucket(user_id: str, experiment: str, buckets: int = 10_000) -> int:
    h = hashlib.sha256(f"{experiment}:{user_id}".encode()).digest()
    return int.from_bytes(h[:8], "big") % buckets


def choose_bundle(user_id: str, name: str, canary_pct: float,
                  prod_id: str, cand_id: str) -> str:
    """canary_pct in [0, 100]. Same user always gets the same arm."""
    exp = f"{name}:{cand_id}"
    if bucket(user_id, exp) < canary_pct * 100:   # 10_000 buckets = 0.01% each
        return cand_id
    return prod_id


if __name__ == "__main__":
    n, hits = 100_000, 0
    for i in range(n):
        if choose_bundle(f"user-{i}", "support-bot", 5.0, "aaa", "bbb") == "bbb":
            hits += 1
    print(f"canary share: {hits / n:.2%}")   # close to 5%, sticky per user

Keep each bucketing decision out of the model path: compute it once per request, attach the chosen bundle hash to the trace, and pass it down. Every span for that request then carries the same label, which is what makes per-bundle dashboards trivial.

Guardrail metrics and stop conditions

Choose guardrails before the rollout, not during it. A workable set has four families. Quality proxies: judge scores on a sampled slice of live traffic, user feedback rates, escalation-to-human rate. Reliability: schema-violation rate, tool-error rate, timeout rate, refusal rate. Cost: input and output tokens per request, and cache-hit rate, because a longer prompt silently raises both bill and latency. Latency: time to first token and end-to-end percentiles. Our post on LLM inference SLOs, TTFT, TPOT and goodput explains why the percentile, not the mean, is the right lens.

Write each guardrail as a comparison against the baseline arm running at the same time, not against a historical number. Traffic mix shifts by hour and by day, so a fixed threshold produces false alarms at peak and misses regressions at night. A concurrent control arm absorbs those shifts.

Statistical power is the honest constraint. A 1 percent canary on a low-traffic feature may take days to collect enough samples to detect a modest quality change; it will catch crashes and format breaks within minutes but not subtle degradation. Decide the step durations from the traffic you actually have: the number of judged samples needed to detect your regression margin sets the minimum dwell time per step. If a step cannot reach that sample size in a reasonable time, say so and accept a slower rollout or a wider first slice.

Stepping typically goes 1, 10, 50, 100 percent, with automated advancement when guardrails hold for the dwell time and automated rollback when a hard guardrail trips. Hard guardrails, such as schema-violation rate doubling or an error-rate spike, should trigger rollback without waiting for significance. Soft guardrails, such as a small judge-score dip, should hold the rollout and page a human.

Runtime Resolution, Observability and Rollback Mechanics

Rollback is only as fast as the slowest component that must notice it. In this design that is the application’s alias refresh interval, so the interval is a deliberate trade-off between rollback speed and registry load.

Figure 4: Runtime sequence. The app polls the alias, stamps each trace with the bundle hash, and picks up a rollback on its next refresh.

Figure 4 traces a promotion and a rollback. The operator, or the rollout controller, moves the prod alias to bundle B. Each application instance polls the registry on an interval, for example every 30 seconds, receives B, and begins stamping its traces with B’s hash. A guardrail breach surfaces from the trace store, the alias moves back to A, and each instance picks up A on its next poll. Worst-case time to full rollback is therefore roughly the refresh interval plus any in-flight request duration. Thirty seconds is an illustrative setting, not a recommendation; pick yours from the registry’s capacity and your tolerance for exposure.

Make the resolver boring and safe

The resolver sits in the hot path of every request, so it must be local and failure-tolerant. Cache the resolved bundle in memory and refresh in a background task; never call the registry per request. If a refresh fails, keep serving the last known good bundle and emit a metric, rather than failing requests or falling back to an unreviewed default. If the very first load fails at startup, fail the instance’s readiness check so the orchestrator does not route traffic to it.

Validate before swapping. When the resolver receives a new bundle, check that its hash matches its content, that the pinned model is in the allowed list, and that the schemas parse. Only then replace the in-memory pointer, atomically, so a request never sees half of one bundle and half of another. Per-request code should capture the bundle object once at the start and use it throughout, so a mid-request refresh cannot mix prompt A with parameters from B.

Tag everything with the bundle hash

The single highest-value observability habit is stamping the bundle hash on every trace, log line and metric label. With that, “which version produced this answer?” becomes a lookup. Incident review becomes a comparison between two hashes rather than an archaeology exercise. Dashboards can split any metric by bundle, which is exactly what the canary needs. Beware metric-cardinality blowup: put the hash on traces and logs freely, but on metrics use it only for the bundles currently in rollout, or map hashes to a small set of labels like baseline and candidate.

Tracing standards are evolving here. OpenTelemetry has been developing semantic conventions for generative AI spans, but I have not verified their current stability status for this post, so check the specification before depending on attribute names. Whatever convention you adopt, a custom attribute such as app.bundle_id costs nothing and survives convention changes.

Rollback is not always a pointer move

Moving the alias back restores the prompt, model and parameters. It does not undo side effects of the bad bundle. If the candidate wrote bad data to a store, sent incorrect emails, or polluted a cache, the revert leaves that mess. Three practical mitigations help. Tag every side-effecting write with the bundle hash so affected records can be found and repaired. Keep semantic and response caches keyed by bundle hash so stale answers from a bad bundle are never served to a good one. And limit the blast radius of risky bundles through the canary itself: a 1 percent slice makes a bad bundle a small cleanup rather than a large one.

State is the other trap. If a bundle changes the format of conversation memory, summaries or stored tool results, rolling back the code path may leave behind data the older bundle cannot read. Version stored state formats separately and make them forward and backward tolerant for at least one release, the same expand-and-contract discipline database migrations use.

Trade-offs, Gotchas, and What Goes Wrong

The bundle model is not free, and several failure modes survive even a careful implementation.

Over-versioning creates friction. If every one-word prompt tweak must clear a 20-minute gate and a multi-day canary, authors route around the process by editing prompts in production consoles or hard-coding strings. Tier your changes. Copy edits to low-risk prompts can take a fast path with a small gate and a short canary; model swaps, tool-schema changes and safety-prompt edits take the full path. The registry can enforce tiers by labelling bundles with a risk class derived from which fields changed.

Gates overfit. Teams iterate prompts against the same golden set until the set stops being independent evidence. Hold out a private slice that authors never see, refresh the set from sampled production traffic on a schedule, and watch for a widening gap between gate scores and live metrics, which is the signal that the dataset has gone stale.

Judge drift can masquerade as regression. If the judge model is a floating alias or is updated by its provider, scores shift without any change to your bundle. Pin the judge, store a fixed calibration set of human-labelled items, and re-run it whenever the judge changes. When a gate suddenly fails everything, check the judge before blaming the candidate.

Pinned does not mean frozen. Anthropic’s documentation says weights and configuration of an existing ID are not updated, yet also acknowledges that infrastructure changes can produce minor behavioural differences. Other providers’ policies differ, and I have not verified each one for this post. The defence is the nightly full-gate run against the production bundle: if the same bundle scores differently on the same data, something outside your control changed, and you want to learn that from a dashboard rather than a customer.

Prompt caching interacts with versioning. Provider-side prompt caches key on prompt prefixes, so a canary arm with a new system prompt starts cold and shows higher latency and cost than the baseline for reasons unrelated to quality. Compare cost and latency guardrails with that in mind, and see our analysis of prompt caching economics across providers for how prefix stability changes the break-even.

Dynamic prompts defeat hashing. If your prompt template interpolates user data, retrieved passages or timestamps, the bundle hash covers the template, not the rendered text. That is correct, but it means two requests under one bundle can differ wildly. Log the rendered prompt (with appropriate redaction) alongside the bundle hash so incidents remain debuggable.

Secrets and compliance. Prompts often embed business rules, internal URLs or customer-specific instructions. A registry that stores them needs the same access control and audit logging as source code, and in regulated settings the alias-history log, who promoted what and when, is part of your evidence trail. Separate the right to author bundles from the right to move the prod alias.

Organisational anti-pattern: the prompt owner who is nobody. Registries do not assign accountability. Name an owner per bundle, make the owner the default responder to its guardrail alerts, and review ownership when people change teams.

Practical Recommendations

Start with the smallest version of this design that gives you reproducibility, and add rigour where incidents or cost justify it. The order below follows the return on effort I would expect, though your own incident history should override it.

First, define the bundle and stamp its hash on every trace. This alone converts “which prompt was that?” from a mystery to a lookup, and it needs no new infrastructure beyond a hash function and a log attribute. Second, pin model IDs and add the weekly deprecation check, because a surprise retirement is an avoidable outage. Third, put prompts and configs in git and register bundles from CI, so the registry is a product of review rather than a place people edit by hand.

Fourth, add the eval gate with deterministic checks first and a small judged set second. Even a 100-item golden set with a paired bootstrap catches embarrassing regressions. Grow it from incidents. Fifth, introduce aliases with enforced promotion rules and a prod-previous pointer, then rehearse the rollback until it takes under a minute. Finally, add shadow and canary stages for the changes that deserve them, with guardrails written as comparisons against a concurrent control.

A checklist you can paste into a pull-request template:

Bundle hash recorded; prompt, model ID, parameters, schemas and retrieval config all inside it.
Model ID is a pinned identifier, not a floating alias, and its shutdown date is known.
Only one axis changed (prompt, or model, or parameters), or the reason for combining is written down.
Golden set version and judge model pinned in the eval contract; gate report attached.
Lower bound of the paired confidence interval clears the stated margin; per-stratum results reviewed.
Rollout plan names stages, dwell times, hard and soft guardrails, and an owner.
Rollback tested: alias history shows a previous target, and the resolver refresh interval is known.
Side-effecting writes and caches are keyed or tagged by bundle hash.

Frequently Asked Questions

What is prompt version management?

Prompt version management is the practice of storing every prompt change as an immutable, identifiable version, linking it to the model and parameters it was tested with, and controlling how versions move to production. Done well it provides reproducibility, an audit trail, evaluation before release and fast rollback. In practice the unit worth versioning is a bundle of prompt, model ID, parameters and schemas rather than the text alone.

Should I store prompts in git or in a prompt registry?

Use both for different jobs. Git is the right home for authoring, review and history, since prompts are code-like and benefit from pull requests. A registry is the right home for serving: it holds immutable built bundles and the mutable environment aliases that applications resolve at runtime, so you can promote or roll back without a redeploy. A CI step that builds bundles from git into the registry connects the two.

How do I roll back an LLM prompt change quickly?

Make rollback a pointer move. If production resolves an alias such as prod to a bundle hash, rollback is setting that alias to the previous hash, and each application instance picks it up on its next refresh, typically seconds. Keep a prod-previous pointer so nobody searches for the old version mid-incident. Remember that rollback does not undo side effects such as bad writes or poisoned caches.

How large should an LLM eval set be for a CI gate?

There is no universal number; it depends on the effect size you need to detect and the variance of your scores. A paired comparison needs far fewer items than an unpaired one, and a few hundred items often detects moderate regressions, while subtle ones need more. Compute the confidence interval on the paired differences, report it, and grow the set from production incidents and stratified samples of real traffic.

Do pinned model IDs guarantee identical behaviour?

Not entirely. Provider documentation I reviewed states that a pinned Claude model ID keeps constant underlying weights for the ID’s lifetime, but also acknowledges that infrastructure updates can produce minor observable differences. Pinning prevents the large, announced changes that come from a floating alias moving to a new snapshot. Run a scheduled evaluation of the production bundle so that unannounced drift shows up in your own metrics.

What is the difference between shadow mode and a canary?

Shadow mode runs the candidate bundle on mirrored traffic and discards its answers, so users are never exposed and you can compare outputs, latency and cost safely, but you learn nothing about user reactions. A canary serves the candidate’s answers to a small sticky share of real users, which exposes real risk but measures real outcomes. Mature rollouts use shadow first, then a staged canary.

Prompt and Model Version Management for LLM Apps: Registries, Eval Gates and Safe Rollouts (2026)