A/B Testing for ML Systems: CUPED, Sequential Testing, Interleaving and Contextual Bandits

Your new ranking model beat the old one by four points of NDCG offline, and then lost 0.3% of revenue in a two-week online test. Or it won, and nobody could say whether the win was real, because the dashboard was checked every morning and the test was stopped the first time the p-value dipped under 0.05. This is the ordinary failure mode of ab testing for machine learning: the statistics taught for button colors quietly break when the treatment is a model that learns from, and interacts with, the very population being measured.

This matters more in 2026 than it did five years ago. Models are retrained continuously, LLM-backed features have noisy outputs and noisy graders, and marketplaces couple every user’s outcome to everyone else’s. A test that is merely “statistically significant” is no longer a decision you can defend.

You will leave with a working mental model and runnable code: how to pick a randomization unit, size a test with real numbers, shrink variance with CUPED, peek safely with sequential methods, use interleaving for ranking, decide between a bandit and a fixed test, and diagnose why offline and online metrics disagree.

What this covers: randomization and interference, power and minimum detectable effect (MDE) math, CUPED, sequential testing, interleaving, bandits, offline-online gaps, guardrails, sample ratio mismatch (SRM), novelty effects and LLM-judge experiments.

Context and Background

Controlled online experiments are the oldest reliable tool in product measurement. The canonical reference is Kohavi, Tang and Xu’s Trustworthy Online Controlled Experiments, distilled from experimentation programs at Microsoft, Google, LinkedIn and others. Its central lesson is unglamorous: most ideas fail, most surprising wins are bugs, and the infrastructure around the test (assignment, logging, validity checks) matters as much as the test statistic.

Machine learning changes three assumptions that the classic playbook leaves implicit. First, the treatment is not a fixed page variant but a function with a training distribution, so its quality depends on the data it saw and the data it will generate. Second, outcomes are often heavy-tailed and sparse (revenue per user, long-term retention), so the t-test’s normal approximation needs far more samples than a conversion-rate intuition suggests. Third, the unit that is convenient to randomize, the user, is frequently not the unit across which effects propagate.

The community has responded with a toolbox. Variance reduction using pre-experiment data (CUPED, from Deng, Xu, Kohavi and Walker at WSDM 2013) cuts required traffic. Always-valid inference (Johari, Pekelis and Walsh, 2015) lets teams monitor continuously without inflating false positives; the paper is available at arXiv 1512.04922. Interleaving, developed for search ranking evaluation, compares two rankers inside a single result list. Bandit algorithms, including the LinUCB contextual bandit from Li, Chu, Langford and Schapire (WWW 2010, arXiv 1003.0146), trade exploration against exploitation when the goal is reward during the test rather than a clean inference afterwards.

None of this removes the need for offline evaluation. Offline metrics are cheap, fast and reproducible; they are how you decide which candidates deserve live traffic. The discipline of building that offline layer is covered in our guide to the LLM evaluation harness architecture, and the data plumbing that keeps training and serving features consistent is in the feature store architecture post. This article starts where those end: the moment a candidate meets real users and you need to decide, with quantified uncertainty, whether to ship it.

A useful framing is that an experiment platform is a measurement instrument, and instruments have error budgets. Measurement uncertainty is a topic IoT engineers know well from sensor work; the same vocabulary of bias, variance and uncertainty budgets applies, as discussed in our piece on sensor calibration, drift compensation and measurement uncertainty. An A/B test is just a differential measurement with a noisy, drifting sensor made of people.

The Reference Architecture: Assignment, Exposure, Analysis

An A/B test for an ML system is a pipeline of three contracts: deterministic assignment of a randomization unit to a variant, trustworthy logging of who was actually exposed, and an analysis that respects the dependence structure of the data. Break any one contract and the p-value is meaningless, however sophisticated the statistics.

Figure 1: Assignment pipeline. Entity IDs are hashed with an experiment-specific salt into buckets, the randomization unit is chosen to match where interference lives, and exposure logs feed an SRM check before any metric is computed.

Figure 1 shows the shape of the system. Traffic arrives keyed by some entity identifier. A hash function with a per-experiment salt maps the identifier to a bucket (commonly 0 to 9,999), and bucket ranges map to variants. The salt matters: reusing a salt across experiments correlates assignments and lets one experiment’s treatment leak into another’s baseline. The choice of randomization unit is the first real design decision, and the log at the end is the only record the analysis will ever see.

Deterministic hashing, layers and exposure

Assignment must be a pure function of the identifier and salt, so any service can recompute it without a lookup and a returning user always sees the same variant. A minimal implementation:

import hashlib

def bucket(entity_id: str, salt: str, n_buckets: int = 10_000) -> int:
    h = hashlib.sha256(f"{salt}:{entity_id}".encode()).digest()
    return int.from_bytes(h[:8], "big") % n_buckets

def variant(entity_id, salt, split=(0.5, 0.5)):
    b = bucket(entity_id, salt) / 10_000
    acc = 0.0
    for i, p in enumerate(split):
        acc += p
        if b < acc:
            return i
    return len(split) - 1

Large organizations organize experiments into layers: mutually exclusive experiments share a layer and split its buckets, while independent layers re-hash with different salts so that experiments in different layers are orthogonal. This is described in Google’s “Overlapping Experiment Infrastructure” paper (Tang et al., KDD 2010). For ML work, put anything that changes the same model or the same feature (two ranking models, say) in the same layer; otherwise their interaction will contaminate both readouts.

Equally important is the distinction between assignment and exposure. A user assigned to treatment who never reaches the surface where the model runs contributes noise, not signal. Log an exposure event at the moment the variant’s behavior is actually shown, and analyze either the intent-to-treat population (everyone assigned, diluted but unbiased) or the exposed population with care, because exposure itself can be affected by treatment. A model that makes a page slower might reduce exposures, and conditioning on exposure then hides the harm.

Choosing the randomization unit

The unit of randomization should be the coarsest entity across which outcomes interact, and no coarser than necessary. Four options cover most ML cases:

User or device level. The default. It keeps the experience consistent across sessions and is right when one user’s outcome does not depend on other users’ assignments. Cross-device identity gaps cause leakage: the same human in both arms through two cookies.
Session or request level. Higher traffic efficiency, but the same user sees both experiences, which contaminates learning effects and breaks any metric defined per user.
Cluster or geo level. Randomize whole cities, social-graph clusters or warehouses when interference runs through a shared resource. Effective sample size collapses to the number of clusters, so power drops sharply.
Time slice (switchback). Alternate the whole system between variants over time windows. Used when there is only one shared supply pool, as in dispatch or pricing.

A unit mismatch also biases standard errors. If you randomize by user but compute metrics per session and treat sessions as independent, the sessions of one heavy user are correlated, and the naive standard error is too small. Either aggregate to the randomization unit or use the delta method or cluster-robust errors. This single mistake explains a large share of “significant” results that fail to replicate.

Interference in marketplaces and LLM applications

The stable unit treatment value assumption (SUTVA) says one unit’s outcome depends only on its own assignment. Marketplaces violate it by construction. Suppose a new ranking model for buyers is better at converting. Treated buyers consume scarce inventory (the same listings, drivers or appointment slots) that control buyers would otherwise have bought. The measured treatment-minus-control difference then overstates the true effect of shipping to everyone, because in the full launch there is no untreated pool to lose out. The bias can also run the other way when treatment degrades the shared pool.

Mitigations form a ladder of cost. Cluster randomization on geography or on demand-side market segments reduces leakage at the price of fewer independent units. Two-sided or budget-split designs randomize both buyers and sellers and compare the cells. Switchback experiments sidestep spatial interference but need care with carryover effects between windows, usually handled by discarding a washout period at the start of each slice. None is free; the right move is to estimate how large interference is before committing, for example by running the same experiment at two randomization granularities and comparing.

LLM applications introduce a subtler coupling. Shared caches mean that a treatment prompt can warm a cache that control then benefits from. Shared rate limits and shared GPU capacity couple latency across arms. Conversation memory or retrieval indexes updated from user interactions let treatment behavior leak into the corpus control retrieves from. Human-in-the-loop systems are the worst case: a support agent who sees AI suggestions on some tickets changes how they handle all of their tickets. The fix is to randomize at the agent or team level, which is again a trade of power for validity.

Power, MDE and Variance Reduction: Worked Numbers

Power analysis answers a planning question: how many units do you need to detect an effect of a given size with acceptable error rates? For a two-sided test on the difference of two means with equal arms, the per-arm sample size is approximately

n = 2 (z_{1-α/2} + z_{1-β})² σ² / δ²

where σ² is the metric variance per unit, δ is the absolute effect you want to detect (the MDE) and the z terms are standard normal quantiles. With α = 0.05 and power 0.8 the sum of quantiles is 1.96 + 0.84 = 2.80, and its square is about 7.85, so n ≈ 15.7 σ² / δ². The practical rule of “16 σ² over δ squared” is accurate enough for planning.

A worked conversion-rate example

Take a recommendation widget with a baseline click-through rate of 5%. For a Bernoulli metric σ² = p(1−p) = 0.0475. Suppose the team wants to detect a 2% relative lift, which is an absolute δ of 0.001. Using the exact quantiles (1.95996 and 0.84162), the per-arm requirement is

n = 2 × 7.849 × 0.0475 / 0.001² ≈ 745,600 users per arm.

That is about 1.49 million users in total. At 200,000 eligible users per day split across two arms, the test runs roughly seven and a half days at full allocation. Now consider a smaller feature: a 0.5% relative lift (δ = 0.00025) requires sixteen times as many users, around 11.9 million per arm, because n scales with 1/δ². This quadratic penalty is why “small but real” improvements to mature models are effectively unmeasurable without variance reduction, and why experimentation teams argue so much about which effects are worth testing at all.

For revenue-like metrics the arithmetic is harsher. If revenue per user has a mean of $4 and a standard deviation of $40 (a plausible shape for a heavy-tailed metric; these figures are illustrative), then detecting a 1% lift (δ = $0.04) needs n = 15.7 × 1,600 / 0.0016 ≈ 15.7 million per arm. Heavy tails make σ dominate the cost. Capping or winsorizing extreme values at a pre-registered percentile, or analyzing a ratio with the delta method, often cuts σ severalfold at the price of a slightly different estimand. State that estimand explicitly in the experiment spec.

MDE is a planning input, not a prediction

A common mistake is to treat the MDE as the expected effect. It is the smallest effect worth being able to detect. If the true effect is smaller than the MDE, the test is underpowered for it, and “not significant” does not mean “no effect”. Conversely, an underpowered test that does reach significance tends to exaggerate the effect (the winner’s curse, or Type M error), because only unusually lucky samples cross the threshold. When a launch decision rests on a surprisingly large measured lift from a small test, shrink the estimate toward your prior from past experiments before believing it.

CUPED: using pre-experiment data to cut variance

CUPED, short for Controlled-experiment Using Pre-Experiment Data, subtracts the part of the outcome that is predictable from a covariate measured before the experiment began. Because randomization makes the pre-period covariate independent of treatment, the adjustment does not bias the estimate; it only removes noise. The adjusted metric is

Y_cuped = Y − θ (X − mean(X)), with θ = Cov(X, Y) / Var(X).

The variance of the adjusted metric is Var(Y) (1 − ρ²), where ρ is the correlation between X and Y. A correlation of 0.7 gives a variance factor of 0.51, so the same precision needs about half the users, or equivalently the MDE shrinks by a factor of √0.51 ≈ 0.71. Figure 2 shows the computation.

Figure 2: CUPED. A pre-experiment covariate X predicts the in-experiment outcome Y. Subtracting θ(X minus its mean) leaves the treatment effect untouched while the variance falls by a factor of one minus ρ squared.

CUPED works best for metrics that are persistent per user: sessions per week, spend, watch time. It does nothing for a first-ever-purchase metric where the pre-period value is zero for everyone, and new users, who have no history, get no variance reduction. The covariate should be the same metric from the pre-period, though any pre-treatment variable is valid, and modern platforms generalize to regression with several covariates or machine-learned predictions of the outcome (the CUPAC approach published by DoorDash engineers). One caution applies: covariates must be measured strictly before assignment. A feature that is influenced by treatment, even indirectly, reintroduces bias.

Runnable CUPED in numpy

The following script simulates 400,000 users with a pre-period metric and an in-experiment metric whose true treatment effect is +0.15, then compares the raw and adjusted estimates.

import numpy as np

rng = np.random.default_rng(7)
n = 200_000
x = rng.gamma(2.0, 10.0, 2 * n)            # pre-period metric
y = 0.8 * x + rng.normal(0, 12, 2 * n)     # in-experiment metric
t = np.repeat([0, 1], n); rng.shuffle(t)   # random assignment
y = y + 0.15 * t                           # true effect = 0.15

def cuped(y, x):
    theta = np.cov(y, x, ddof=1)[0, 1] / np.var(x, ddof=1)
    return y - theta * (x - x.mean()), theta

def diff(v, t):
    a, b = v[t == 1], v[t == 0]
    est = a.mean() - b.mean()
    se = np.sqrt(a.var(ddof=1) / len(a) + b.var(ddof=1) / len(b))
    return est, se

print("raw  ", diff(y, t))
yc, theta = cuped(y, x)
print("cuped", diff(yc, t), "theta", theta, "rho", np.corrcoef(x, y)[0, 1])

Running it with this seed, the raw estimate was 0.114 with a standard error of 0.0520, and the CUPED estimate was 0.114 with a standard error of 0.0379. The correlation was 0.684, so the predicted standard error ratio is √(1 − 0.684²) ≈ 0.729, and the observed ratio 0.0379 / 0.0520 ≈ 0.729 matches. Both estimates sit within about one standard error of the true 0.15, as expected from sampling noise. (This is a synthetic simulation, not a production benchmark.) Estimate θ on the pooled data of both arms, or on pre-experiment data only, and never separately within arms in a way that lets the arms differ.

In practice compute θ and the adjusted metric inside the same pipeline that handles the denominators of ratio metrics, and treat missing pre-period values by imputing the mean with an indicator, so that new users stay in the sample without distorting the fit.

Sequential Testing: Peeking Without Cheating

The fixed-horizon test assumes you decide the sample size in advance and look at the data exactly once. Real teams look daily, because waiting two weeks for a broken model to show harm is unacceptable. If you run a z-test at the 5% level after every day for fourteen days and stop at the first significant result, the true false positive rate under the null climbs well past 5%, to roughly 19% for ten looks in the classic Armitage-McPherson-Rowe calculation. The p-value from a single look is simply not the right object once the stopping rule depends on the data.

Figure 3: Choosing a sequential method. If peeking is not needed, plan a fixed horizon. If the look schedule is known, use group sequential boundaries. If looks are continuous or unscheduled, use mSPRT or another always-valid procedure.

Group sequential designs

Group sequential methods fix a small number of interim analyses in advance and spend the overall α across them. The Pocock boundary uses the same critical value at every look, making early stopping easy but making the final look stricter. The O’Brien-Fleming boundary is extremely conservative early and close to the nominal threshold at the end, so it rarely stops on early noise and costs little power overall. The Lan-DeMets alpha-spending approach generalizes both and allows looks at unplanned information fractions, which suits online experiments where traffic varies by day.

For a design with five equally spaced looks and α = 0.05, O’Brien-Fleming critical z values are roughly 4.56, 3.23, 2.63, 2.28 and 2.04 at looks one through five (standard published values for this design). The first look therefore requires |z| above 4.5, which almost never happens by chance but does catch a catastrophic regression within the first fifth of the planned sample. The final-look threshold of 2.04 is only slightly above 1.96, so total sample size inflates by a few percent relative to a fixed test.

mSPRT and always-valid inference

When you want to monitor continuously and stop whenever the evidence warrants, the mixture sequential probability ratio test (mSPRT) is the standard tool, popularized for A/B testing by Johari, Pekelis and Walsh. It places a prior (a mixing distribution) over the alternative effect size and computes the average likelihood ratio against the null. By Ville’s inequality, a nonnegative martingale starting at 1 exceeds 1/α with probability at most α at any time, so rejecting when the mixture likelihood ratio Λ_n crosses 1/α controls Type I error no matter when or why you stop. The always-valid p-value is p_n = min(1, 1/max_{k≤n} Λ_k).

For a normal mean difference with known variance σ², observed mean difference m_n over n paired observations (or per-arm n with variance 2σ²), and a N(0, τ²) mixing prior, the closed form is

Λ_n = sqrt(σ² / (σ² + n τ²)) · exp( n² τ² m_n² / (2 σ² (σ² + n τ²)) ).

import numpy as np

def msprt_pvalue(diff_mean, n, sigma2, tau2):
    """Always-valid p-value contribution at sample size n (per arm).
    sigma2 is the variance of the difference estimator per unit pair."""
    lam = np.sqrt(sigma2 / (sigma2 + n * tau2)) * np.exp(
        (n**2 * tau2 * diff_mean**2) / (2 * sigma2 * (sigma2 + n * tau2))
    )
    return min(1.0, 1.0 / lam)

def run(stream_a, stream_b, sigma2, tau2, alpha=0.05):
    running_min_p = 1.0
    for n in range(10, len(stream_a) + 1, 10):
        m = stream_a[:n].mean() - stream_b[:n].mean()
        running_min_p = min(running_min_p, msprt_pvalue(m, n, sigma2, tau2))
        if running_min_p < alpha:
            return n, running_min_p
    return None, running_min_p

The tuning knob is τ². Choose it to match the effect size you most want to detect early: a mixing variance near the square of the MDE gives good power around that effect. A mismatched τ² does not invalidate the test, it only shifts power. Always-valid tests are more conservative than a single fixed-horizon test at the same sample size, which is the price of being allowed to stop at any time. Modern variants use confidence sequences, which provide a running interval that covers the true effect at all times simultaneously, and are often easier to communicate than p-values.

Sequential methods combine with CUPED: apply the adjustment to the metric first, then run the sequential test on the adjusted values, using the adjusted variance. Variance must be estimated from data too, and with unknown variance the mSPRT is approximate for small n, so impose a minimum sample before any decision. Peeking safety also does not repair other problems: looking at many metrics, slicing by many segments or running many variants still needs multiplicity control such as Benjamini-Hochberg or Holm corrections.

Interleaving and Contextual Bandits: Two Ways to Spend Traffic Better

Fixed A/B tests are statistically clean but traffic-hungry and blind to what you could learn on the way. Two families of methods attack that: interleaving, which makes ranking comparisons far more sensitive, and bandits, which shift traffic toward better options while the experiment runs.

Interleaving experiments for ranking

In an A/B test of two rankers, each user sees one ranker’s list, and you compare aggregate clicks or conversions across users. The between-user variance is huge relative to the small difference between two similar rankers. In an interleaving experiment, every user sees a single merged list built from both rankers, and clicks are credited to the ranker that contributed the clicked item. The comparison is within-user, so user-level variance cancels.

Team-draft interleaving (Radlinski, Kurup and Joachims, CIKM 2008) works like picking teams: at each round a coin flip decides which ranker picks first, each picks its highest-ranked item not already in the list, and the list records which ranker contributed each slot. Balanced interleaving and probabilistic interleaving are alternatives with different bias properties. Netflix engineers reported that interleaving needed substantially fewer users than A/B tests to detect a ranking preference, and published work from Bing and Yahoo reached the same qualitative conclusion; the precise sensitivity gain depends on the metric and the rankers, so measure it on your own historical tests rather than assuming a number.

import random

def team_draft(rank_a, rank_b, k=10):
    merged, owner = [], []
    ia = ib = 0
    while len(merged) < k and (ia < len(rank_a) or ib < len(rank_b)):
        a_first = random.random() < 0.5
        for who in ((0, 1) if a_first else (1, 0)):
            r, i = (rank_a, ia) if who == 0 else (rank_b, ib)
            while i < len(r) and r[i] in merged:
                i += 1
            if i < len(r) and len(merged) < k:
                merged.append(r[i]); owner.append(who); i += 1
            if who == 0: ia = i
            else: ib = i
    return merged, owner

def credit(clicked_positions, owner):
    a = sum(1 for p in clicked_positions if owner[p] == 0)
    b = sum(1 for p in clicked_positions if owner[p] == 1)
    return (a > b) - (a < b)   # +1 A wins, -1 B wins, 0 tie

Aggregate the per-query outcomes with a sign test or a binomial test on the win fraction. Interleaving has clear limits. It measures preference between rankers on click-like signals, not absolute business impact such as revenue or retention. It cannot test changes that alter page layout or latency, because both rankers share the same page. And it requires the ranking to be a list of discrete items. Use it as a fast filter between offline evaluation and a full A/B test, and confirm the winners with a conventional test on the business metric.

Contextual bandits versus A/B tests

A/B tests and bandits answer different questions. A test estimates an effect with a controlled error rate so that a decision can be justified. A bandit maximizes cumulative reward during the learning, accepting biased or adaptive estimates. If the goal is “which of these fifty headlines should the homepage use today” and the content decays within hours, a bandit wins. If the goal is “should we replace our production model”, a test usually wins, since the bandit’s adaptive allocation complicates inference and the regret saved is small compared with the cost of a wrong permanent decision.

Figure 4: Contextual bandit loop. The policy returns an arm plus its selection probability, which the serving layer logs with the context. The trainer updates the policy from batches, and the logged propensities enable later off-policy evaluation.

Thompson sampling maintains a posterior over each arm’s reward and plays the arm whose sampled value is highest, which naturally balances exploration and exploitation. For binary rewards each arm keeps a Beta(α, β) posterior:

import numpy as np
rng = np.random.default_rng(0)
true_p = np.array([0.04, 0.05, 0.06])
a = np.ones(3); b = np.ones(3)
for _ in range(50_000):
    arm = np.argmax(rng.beta(a, b))
    r = rng.random() < true_p[arm]
    a[arm] += r; b[arm] += 1 - r
print((a + b - 2).astype(int))   # pulls per arm concentrate on arm 2

LinUCB adds context. For each arm it fits a ridge regression of reward on a feature vector x and plays the arm maximizing the estimated reward plus an uncertainty bonus:

import numpy as np

class LinUCB:
    def __init__(self, n_arms, d, alpha=1.0):
        self.A = [np.eye(d) for _ in range(n_arms)]
        self.b = [np.zeros(d) for _ in range(n_arms)]
        self.alpha = alpha

    def choose(self, x):
        scores = []
        for A, b in zip(self.A, self.b):
            Ainv = np.linalg.inv(A)
            theta = Ainv @ b
            scores.append(theta @ x + self.alpha * np.sqrt(x @ Ainv @ x))
        return int(np.argmax(scores))

    def update(self, arm, x, reward):
        self.A[arm] += np.outer(x, x)
        self.b[arm] += reward * x

Three practical issues dominate production use. First, log the propensity, the probability with which each arm was chosen, because without it you cannot evaluate a new policy offline from the logs using inverse propensity scoring or doubly robust estimators. UCB policies are deterministic, so for offline evaluation add explicit randomization such as an epsilon floor. Second, rewards are delayed (a purchase hours after a click), so updates must tolerate late feedback, and a naive loop treats pending outcomes as failures. Third, non-stationarity: user preferences drift, so use discounting or sliding windows, or the posterior becomes overconfident in stale conclusions.

A pragmatic hybrid is common: keep a small fixed holdout under a stable baseline policy, run the bandit on the rest, and compare the bandit’s aggregate outcome with the holdout. This gives you an unbiased estimate of the whole system’s lift, which the bandit’s own logs cannot provide.

Why Offline Wins Fail Online, and How to Guard the Launch

The most expensive habit in ML experimentation is trusting offline metrics as proxies for online outcomes. Several mechanisms produce the gap, and each has a diagnostic.

Feedback loops and selection bias. Logged data reflects what the previous model chose to show. A new ranker evaluated offline against clicks on the old ranker’s lists is rewarded for agreeing with the old ranker, and cannot be credited for surfacing items that were never shown. Counterfactual estimators such as inverse propensity scoring address this but need logged propensities and have high variance when the new policy differs much from the logging policy.

Metric misalignment. NDCG, AUC and log loss measure ranking or calibration quality on a labeled set. Business metrics depend on latency, diversity, trust and many things the label does not encode. A model can improve AUC while hurting the long tail of items and thereby catalog engagement.

Train-serve skew. If the production feature pipeline computes a feature differently from the training pipeline, offline metrics describe a model that does not exist online. This is the strongest argument for the parity discipline described in the feature store article linked above; check it before blaming user behavior.

Distribution shift and latency. The offline test set is from last month, while online sees today. A heavier model that adds 80 ms of latency can lose conversions that offline evaluation never charged for. Treat latency as a first-class outcome.

Guardrail metrics

A guardrail is a metric that must not get worse, regardless of whether the primary metric improves. Typical guardrails for ML launches include latency percentiles (p95 and p99, not the mean), error and timeout rates, cost per request, unsubscribe or complaint rates, and fairness or coverage metrics across segments. Define each guardrail with a non-inferiority margin before the test, and test non-inferiority explicitly: the lower confidence bound of the difference must exceed the negative margin. Merely failing to find a significant harm is weak evidence, especially in an underpowered test. Guardrails are also the right place for sequential monitoring: stop early for harm using a conservative boundary, while requiring the full planned sample to claim benefit.

Sample ratio mismatch

Before looking at any metric, compare the observed arm sizes with the designed split using a chi-squared goodness-of-fit test. If you designed a 50/50 split and observe 50,400 versus 49,600 users out of 100,000, the chi-squared statistic is (400² / 50,000) × 2 = 6.4, with p ≈ 0.011 on one degree of freedom. At that p-value, many platforms would flag SRM, since a strict threshold such as p < 0.001 is typical for automated alerts and this case sits in a gray zone worth investigating. A confirmed mismatch means the arms are no longer comparable, and no metric result should be trusted. Common causes are bot filtering that treats variants differently, a redirect or crash in one arm that drops users before logging, caching that serves one variant disproportionately, and triggering logic that differs between arms. Fix the cause and rerun; reweighting cannot repair selection that depends on outcomes.

from scipy.stats import chisquare
obs = [50400, 49600]
print(chisquare(obs, f_exp=[50000, 50000]))   # statistic 6.4, p about 0.011

Novelty and primacy effects

Users respond to change itself. A new recommendation carousel gets curiosity clicks that fade (novelty), or experienced users resist a new layout until they adapt (primacy). Either way, the first days are not representative of steady state. Diagnose by plotting the daily treatment effect against days since first exposure: a decaying curve indicates novelty. Mitigate by running long enough to see stabilization, analyzing only users in a later exposure window, or using a long-lived holdback that keeps a small fraction of users on the old experience for months to measure the cumulative, durable effect. ML systems add a second time effect: a model that learns online or is retrained on its own exposure data will itself change over the test, so record model version with every exposure.

Experimenting on LLM Applications

LLM features break several comfortable assumptions at once: outputs are stochastic, quality is multidimensional, and the quickest quality signal is another model.

LLM-as-judge in online and offline loops

Teams often use a strong model as a judge to score outputs for helpfulness, groundedness or tone, then treat those scores like a metric in the experiment. This is workable if you treat the judge as a noisy instrument with a calibration problem. Judges exhibit position bias (preferring the first of two answers), verbosity bias (favoring longer answers) and self-preference (favoring outputs from their own model family); these effects are documented in the MT-Bench and Chatbot Arena work by Zheng et al. (NeurIPS 2023 datasets and benchmarks track). Mitigations include swapping answer order and averaging, controlling for length, using a rubric with explicit scales, and, most importantly, validating the judge against a sample of human labels and reporting the agreement rate with its confidence interval.

Judge noise has a statistical consequence. If the judge’s label has error, the measured effect is attenuated, and its variance includes both sampling and judging noise. Prediction-powered inference offers a principled way to combine a large pool of cheap judge labels with a small set of human labels to get valid confidence intervals; at minimum, bootstrap over both the sampled prompts and the repeated judge calls. A judge score should usually be a secondary metric or a diagnostic, with a behavioral signal (acceptance rate, edit distance, task completion, escalation to a human) as the primary.

Randomization, caching and the nondeterminism budget

Randomize by user or by conversation, never by individual message, since a multi-turn conversation that switches variants mid-way is uninterpretable. Pin the prompt version, model version, sampling temperature and retrieval index snapshot as part of the variant definition and log them. Because outputs are stochastic, repeated identical inputs produce a spread of results; for offline replay, run several samples per prompt and treat the within-prompt variance as part of the noise budget. Provider-side model updates can change behavior silently under a stable model alias, so pin dated model versions where the provider offers them, and watch for drift in both arms equally using an unchanged control.

Cost and latency deserve guardrail status: a prompt variant that improves quality by a small amount while doubling token usage may be a net loss. Track cost per successful task, not cost per request. Evaluating the pipeline offline first with a harness that handles contamination, as discussed in the evaluation harness article linked above, limits the number of weak candidates that ever reach live traffic.

Trade-offs, Gotchas, and What Goes Wrong

Power versus validity in interference designs. Cluster and switchback designs buy validity by spending sample size. With 40 cities as clusters, you effectively have 40 data points, and an effect that a user-level test would detect in a week may be undetectable in a quarter. Be explicit about what you are willing to trade, and consider whether the interference is large enough to matter before paying the price.

Variance reduction can hide problems. CUPED shrinks standard errors and therefore makes small differences significant. That is what you want when they are real; it also makes any bias in the data pipeline show up as a confident wrong answer. Never skip the SRM check because the intervals look tight.

Adaptive methods and inference. Estimates from bandits are biased, since arms that look good early get more samples and arms that were unlucky early get abandoned. Naive means from bandit logs overstate the winner. Use inverse-propensity or doubly robust estimators when you need a defensible effect size from adaptive data.

Multiple comparisons. A dashboard with twenty metrics and ten segments yields two hundred hypotheses. At α = 0.05, expect roughly ten false positives. Pre-register a single primary metric, apply corrections to the secondary family, and label exploratory findings as hypotheses for the next test.

Heterogeneous effects and Simpson’s paradox. An overall win can hide a loss in an important segment, and aggregate trends can reverse when a segment mix shifts between arms. This is another reason SRM and segment-level balance checks come first.

Long-term versus short-term. Two weeks measures what changes quickly. Models that affect user trust, supply-side behavior or content ecosystems have effects that accumulate. Surrogate indices and long holdbacks help, but they are assumptions, and you should label them as such in the decision memo.

Learning during the test. If the treatment model is retrained daily on all traffic, including treatment-affected traffic, the control arm trains on data shaped by treatment. Decide deliberately whether to train separately per arm, and document it.

Practical Recommendations

Start by making the platform trustworthy before making it clever. Deterministic hashing with per-experiment salts, exposure logging at the point of treatment and an automatic SRM check on every experiment catch the majority of invalid results. Only then add variance reduction and sequential monitoring.

Choose the randomization unit by asking where interference lives. If users share inventory, drivers or a content corpus, estimate interference with a pilot at two granularities before launching expensive tests. If you randomize by user, aggregate metrics to the user, or use the delta method for ratios.

Make CUPED the default for persistent metrics, with a pre-period of two to four weeks and the same metric definition, and report both raw and adjusted estimates in early rollouts to build trust. Use group sequential or mSPRT monitoring for guardrails so harm is caught early while benefit claims wait for the planned sample.

Use interleaving as a pre-A/B filter for ranking models, and bandits for short-lived, high-churn decisions with a holdout for honest measurement. For LLM features, define the variant as the full tuple of prompt, model version, parameters and retrieval snapshot, make the behavioral metric primary and the judge score a calibrated secondary.

A launch checklist:

Primary metric, estimand, MDE and sample size written before the test starts
Randomization unit matches the interference structure; salt unique to the experiment
SRM check passes (chi-squared p above the platform threshold) before any metric review
Guardrails with non-inferiority margins, including p95 latency and cost per task
Monitoring plan: fixed horizon, group sequential or always-valid, chosen in advance
Multiplicity plan for secondary metrics and segments
Novelty check: effect by days since first exposure
Model version and training-data policy per arm recorded
Long-term holdback if effects may accumulate

Frequently Asked Questions

What is A/B testing for machine learning, and how is it different from normal A/B testing?

It is controlled online experimentation where the treatment is a learned model, prompt or policy. The statistics are the same, but ML adds interference through shared inventory and caches, heavy-tailed metrics, feedback loops from retraining on exposure data, and treatments that change over time. Those features mean you must think harder about the randomization unit, variance reduction, versioning and offline-online alignment before trusting a p-value.

How much does CUPED actually reduce sample size?

The variance falls by a factor of 1 − ρ², where ρ is the correlation between the pre-experiment covariate and the outcome. A correlation of 0.7 roughly halves the required sample, while a correlation of 0.3 saves only about 9%. Gains are largest for persistent per-user metrics such as spend or sessions, and negligible for metrics with no history, like first purchases by new users.

Can I peek at my A/B test every day?

Not with a fixed-horizon test, since repeated looks inflate the false positive rate far above the nominal 5%. You can peek safely if you use a sequential method designed for it: group sequential boundaries for planned looks, or mSPRT and confidence sequences for continuous monitoring. These cost some power compared with one fixed look but give valid error control whenever you stop.

When should I use a bandit instead of an A/B test?

Use a bandit when the goal is maximizing reward during the experiment, options are many or short-lived, and a wrong choice is cheap and reversible, such as headlines or promotions. Use an A/B test when you need a defensible effect estimate for a lasting decision like replacing a production model. A hybrid with a fixed holdout lets a bandit run while still measuring system-wide lift.

Why do offline metric gains not show up online?

Common causes are selection bias in logged data, metrics like AUC that do not encode latency or diversity, train-serve feature skew, and distribution shift between the test set and live traffic. Check feature parity first, include latency and cost as outcomes, and use counterfactual estimators with logged propensities where possible. Treat offline results as a filter for candidates, not as a forecast of the lift.

How do I run A/B tests when the metric comes from an LLM judge?

Calibrate the judge against human labels and report its agreement with a confidence interval, randomize answer order to counter position bias, control for length, and bootstrap over prompts and repeated judge calls so that judge noise enters the uncertainty. Keep the judge score secondary to a behavioral metric such as task completion or acceptance rate, and pin model and prompt versions in the variant definition.

A/B Testing for ML Systems: CUPED, Sequential Testing, Interleaving and Contextual Bandits (2026)