ML Model Monitoring in Production: Data Drift, Concept Drift, PSI and KS Tests (2026)

A deployed model rarely fails with an exception. It fails by being quietly, confidently wrong. The service returns HTTP 200, latency is flat, CPU is boring, and the dashboards your platform team built are all green, while the fraud model approves a new attack pattern, the demand forecaster keeps predicting last summer, and the defect classifier slowly mislabels a new camera’s output. Nothing in classic infrastructure monitoring can see this, because the thing that broke is not the software. It is the relationship between the world and the function you trained.

That is why ML model monitoring has become its own discipline in 2026, and why it is harder than it looks: the ground truth you need to measure accuracy usually arrives days or months late, the statistical tests you reach for are tuned for textbook problems rather than for 300 features checked every hour, and a naive setup produces so many alerts that the on-call engineer mutes the channel. This post gives you a working mental model and a working pipeline.

What this covers: the four kinds of drift and why they need different detectors, the mathematics of PSI, the Kolmogorov-Smirnov test, Wasserstein distance, Jensen-Shannon divergence and chi-square, the delayed-label problem and performance estimation without labels, a reference monitoring architecture, runnable Python with Evidently, embedding drift for LLM and RAG systems, alert design, retrain triggers, and the failure modes that make monitoring itself unreliable.

Context and Background

Software monitoring answers “is the system up and fast?” Model monitoring must answer a different question: “is the system still right?” The two diverge because a model is a frozen summary of a data distribution. Training fits a function to samples drawn from some joint distribution P(X, Y) of inputs and targets. Serving silently assumes that production traffic is drawn from the same distribution. Whenever that assumption weakens, accuracy degrades without any code change.

The literature on this goes back well before MLOps tooling. Credit-risk teams have monitored scorecards with the Population Stability Index (PSI) for decades, because regulators expect evidence that a scoring model still describes the population it is applied to. The statistics community formalised the problem as “dataset shift”, and the machine-learning community later split it into covariate shift, prior probability shift and concept drift. Those names matter operationally because each implies a different detector, a different severity, and a different fix.

The tooling landscape has consolidated around a few patterns. Open-source libraries such as Evidently compute drift statistics and render reports from a reference dataset and a current dataset. NannyML focuses on estimating performance without labels. Profile-based loggers such as whylogs capture lightweight statistical summaries at the edge so you never ship raw payloads. Commercial observability platforms such as Arize, Fiddler and similar vendors add storage, dashboards and alert routing. The core mathematics under all of them is the same handful of two-sample tests, which is why understanding the tests is worth more than memorising any one product’s UI.

Monitoring also sits at a specific point in the lifecycle. Upstream, a feature store with online and offline parity removes an entire class of false drift, where training and serving compute a feature differently. Downstream, a retrain pipeline closes the loop, which is the topic of the edge MLOps pipelines for industrial IoT post, where sensors and operating regimes change constantly. Monitoring is the sensor in that control loop. If it is miscalibrated, everything downstream either retrains too often or not at all.

One framing to keep throughout: drift is a property of data, degradation is a property of the model’s error, and the two are only loosely coupled. A feature can drift enormously without hurting accuracy, if the model ignores it or the decision boundary is far away. Accuracy can collapse without any input drift at all, if the labelling rule changes. A monitoring system that treats “drift detected” as synonymous with “model broken” will be wrong in both directions, and most of the craft below is about telling those cases apart.

The Four Kinds of Drift and the Reference Architecture

Direct answer: ML model monitoring in production compares live inference data against a reference dataset, using statistical tests per feature plus prediction and performance metrics. It separates covariate drift (inputs change), prior shift (label frequencies change), concept drift (the input-to-label relationship changes) and upstream data bugs, because each needs a different detector and a different response.

A taxonomy that maps to actions

Write the joint distribution as P(X, Y) = P(Y | X) P(X). Different factors can move, and the factor that moves decides what you can detect without labels.

Covariate shift is a change in P(X) while P(Y | X) stays the same. A mobile app gains a new user segment, a sensor is recalibrated, or a marketing campaign brings younger customers. The model’s function is still correct where it was trained, but it is now evaluated on a different mix of inputs, and it may be asked about regions with little training support. Covariate shift is detectable from inputs alone, which makes it the cheap case. Importance weighting and performance-estimation methods can partly correct for it.

Prior probability shift (also called label shift) is a change in P(Y) while P(X | Y) stays the same. The fraud rate doubles during a holiday weekend, or the share of defective parts falls after a process improvement. A classifier’s calibrated probabilities are now off, because the base rate that was baked into them has moved. You see it in the distribution of predictions, which is why prediction-score monitoring is a first-class signal and not a luxury.

Concept drift is a change in P(Y | X). The same inputs now deserve different outputs. A fraudster adopts a behaviour that previously indicated a good customer, a pandemic changes what a normal commute looks like, or a regulation changes what counts as an eligible applicant. This is the dangerous case, because inputs may look perfectly normal. Without ground truth there is, in general, no way to detect pure concept drift. NannyML’s documentation says as much: its estimators fail when P(Y | X) changes and “there is generally no way to detect this without ground truth.”

Concept drift is further classified by its temporal shape: sudden (a regime switch), gradual (a slow blend of old and new), incremental (a continuous slide), and recurring (seasonal). The shape affects window size. A sudden change wants a short window and a fast alarm. A gradual one wants a long window and a trend test, otherwise you alert on noise.

Upstream data bugs are not drift in the statistical sense, but they dominate real incident logs. A column arrives in different units, a join starts returning nulls, a default value of -1 replaces missing data, a timezone change shifts every timestamp by five and a half hours, a categorical level is renamed. These produce abrupt, large, often single-feature shifts that look like drift to a detector but are fixed by a data engineer, not by retraining. The practical rule is to run schema and quality checks (null rate, range, cardinality, unseen categories) as a separate, cheaper tier that fires before statistical drift tests do. If you retrain on a corrupted feed you bake the bug into the model.

Shift type	What moves	Detectable without labels?	Typical fix
Covariate	P(X)	Yes, from inputs	Retrain on recent data, reweight, widen coverage
Prior / label	P(Y)	Partly, via prediction distribution	Recalibrate, adjust thresholds
Concept	P(Y given X)	No, needs labels or proxies	Retrain with new labels, redesign features
Upstream bug	Pipeline output	Yes, via quality checks	Fix the pipeline, do not retrain

The reference architecture

A monitoring system is a data pipeline with a statistical core. Figure 1 shows the shape that works across batch and online serving.

Figure 1: A reference ML model monitoring architecture. Inference payloads and predictions are logged, joined later with delayed labels, windowed, compared against a reference dataset, and routed to alerts and retrain triggers.

The flow has six stages. The serving layer emits an inference log record per request or per batch: a request identifier, a timestamp, the model version, the feature vector exactly as the model saw it (after preprocessing, not before), the raw prediction and score, and any context such as region or device type. A log store persists these, typically as partitioned Parquet in object storage for volume, with a stream such as Kafka in front when you need near-real-time checks. A label joiner later attaches ground truth when it arrives, keyed by request identifier. A windowing job slices the log into comparison windows. A drift and quality engine computes tests of each window against the reference dataset. A sink writes metrics to a time-series store and fires alerts.

Three design decisions determine whether this architecture is trustworthy.

What is the reference? The reference dataset is the definition of “normal”, and choosing it is a modelling decision, not a convenience. The training set is the obvious choice, but it often includes preprocessing, resampling or class balancing that production never sees. A better reference is often a held-out validation set processed through the same serving code path, or a recent production window that was verified healthy because labels showed acceptable performance. For seasonal domains, a rolling reference (the same weeks last year, or the trailing 28 days) avoids flagging every Monday as drift. NannyML adds a useful constraint: for confidence-based performance estimation the reference data must not be the monitored model’s training set, because calibrating on training data introduces bias.

What is the window? Too small a window gives noisy statistics and false alarms. Too large delays detection. A typical compromise is a window sized so the test has the power to detect a shift you actually care about, with a fixed-size sliding or tumbling window per feature. We will put numbers on this shortly.

Log what the model saw. The most common architectural mistake is logging raw request data rather than model-ready features. Then the monitor tests the wrong distribution, and a preprocessing bug (the very thing you want to catch) is invisible. Log post-transformation features, plus a small sample of raw inputs for debugging. For privacy-constrained domains, log summaries rather than rows, as covered in the whylogs approach discussed later.

Where the signals come from

Think of monitoring as five layers, ordered by cost and by how directly they measure harm.

Operational health: latency, error rate, throughput, resource use. Standard observability. For LLM systems, see the OpenTelemetry GenAI semantic conventions for tracing.
Data quality: schema conformance, null rates, ranges, cardinality, unseen categories, duplicate rates.
Input drift: per-feature distribution comparison, plus multivariate checks.
Prediction drift: distribution of scores and predicted classes. This catches prior shift and many covariate shifts through their combined effect.
Performance: accuracy, precision, recall, calibration, business KPIs, either measured with labels or estimated without them.

Layers 1 to 4 need no labels and run in near real time. Layer 5 is the one you care about and the one that lags. A mature setup uses the cheap layers as early warnings and the performance layer as confirmation, and it is explicit about which alerts mean “investigate” and which mean “page someone.”

Detection Statistics: PSI, KS, Wasserstein, Jensen-Shannon and Chi-Square

Every drift detector reduces to one question: are two samples drawn from the same distribution? The choices differ in what they are sensitive to, how they scale with sample size, and whether they return a p-value or a distance.

Population Stability Index with worked bins

PSI comes from credit scoring. Bin the feature (or score) into k bins using the reference distribution, compute the share of observations in each bin for the reference (expected, e_i) and the current window (actual, a_i), and sum:

PSI = sum over bins of (a_i – e_i) * ln(a_i / e_i)

Each term is non-negative, because (a – e) and ln(a / e) always share a sign. PSI is a symmetrised KL divergence: it equals KL(a || e) + KL(e || a). It is zero when distributions match and grows as mass moves.

A worked example, with illustrative numbers. Take a transaction-amount score binned into five reference quantile-style bins with expected shares 10, 20, 40, 20 and 10 percent. In production the current window has 5, 15, 35, 25 and 20 percent.

Bin	Expected e	Actual a	(a – e)	ln(a/e)	Term
1	0.10	0.05	-0.05	-0.693	0.0347
2	0.20	0.15	-0.05	-0.288	0.0144
3	0.40	0.35	-0.05	-0.134	0.0067
4	0.20	0.25	+0.05	+0.223	0.0112
5	0.10	0.20	+0.10	+0.693	0.0693

The total is about 0.136. The widely quoted rule of thumb from credit-risk practice reads PSI below 0.1 as no significant change, 0.1 to 0.25 as moderate shift worth investigating, and above 0.25 as a major shift. Treat those cut-offs as convention, not theorem: they were calibrated on scorecard populations, they depend on the number of bins, and they are not derived from a null distribution. By that convention our example sits in the moderate band, and the table shows why: bin 5, the high tail, contributes half the total because its share doubled. PSI is dominated by where small reference bins change proportionally.

PSI has real weaknesses. It needs binning, and results move with the bin count and edges. Empty bins make ln(a/e) blow up, so implementations add a small epsilon (commonly something like 1e-4) or merge sparse bins. It is symmetric and unsigned, so it says nothing about direction. And because it has no sampling distribution built in, a PSI of 0.08 from 200 rows and 0.08 from 20 million rows mean different things. Its strength is interpretability and a long audit history, which is why regulated teams keep it even when better tests exist.

The Kolmogorov-Smirnov test

The two-sample KS test compares empirical cumulative distribution functions. The statistic D is the maximum vertical gap between the two CDFs, and the null hypothesis is that both samples come from the same continuous distribution. SciPy exposes it as scipy.stats.ks_2samp(data1, data2, alternative='two-sided', method='auto'), returning the statistic, the p-value, and the location where the gap occurs. With method='auto' it uses the exact distribution for small arrays (the documentation describes exact computation being attempted when both sample sizes are under 10,000) and the asymptotic Kolmogorov-Smirnov distribution for larger ones.

KS is nonparametric and needs no binning, which makes it attractive for continuous features. Its weaknesses are specific. It is most sensitive near the centre of the distribution and relatively insensitive to changes in the tails. It is defined for continuous data, so ties from discrete or heavily rounded features distort it. And, crucially, it has high power at large n. The approximate critical value at significance 0.05 is 1.358 times the square root of (n + m) / (n m). With 1,000 reference and 1,000 current rows that is about 0.061. With 100,000 and 100,000 it falls to about 0.006. A shift in the empirical CDF of less than one percentage point will then be “statistically significant” and operationally irrelevant. This is the single most common reason KS-based monitors drown teams in alerts.

Wasserstein distance

The first Wasserstein distance (earth mover’s distance) measures how much mass must move, and how far, to turn one distribution into the other. For one-dimensional empirical samples it equals the area between the two CDFs. Unlike KS, it responds to the magnitude of the shift in the feature’s own units, including in the tails, and it does not saturate. The cost is that the raw distance depends on the feature’s scale, so a threshold of “0.1” means nothing unless you normalise. Evidently’s approach, per its documentation, is to compute a normalised Wasserstein distance for numerical columns on larger datasets; I describe the exact defaults below and recommend verifying against your installed version.

Jensen-Shannon divergence and chi-square

For categorical features, compare category frequency vectors. The chi-square test takes observed counts in the current window against expected counts implied by the reference proportions, and returns a p-value. Like KS, it becomes oversensitive as n grows, and it misbehaves when expected counts per category are small (a common rule is at least five). The Jensen-Shannon divergence is a smoothed, symmetric, bounded version of KL divergence. With natural logarithms it lies between 0 and ln 2 (about 0.693), and its square root is a true metric. For our five-bin example it works out to roughly 0.017, which illustrates a practical point: JS values look small and need thresholds calibrated to the domain. Because it is a distance, not a hypothesis test, it does not inflate with sample size the way a p-value does, apart from estimation noise.

What Evidently actually does by default

Evidently’s documentation describes a sample-size-aware default policy, which is a good template to copy. For small datasets (up to 1,000 reference rows), numerical columns with more than five unique values use the two-sample KS test, categorical columns or numerical ones with five or fewer unique values use chi-square, and binary categorical columns use a z-test on proportions, all flagging drift when p is at or below 0.05. For larger datasets, numerical columns use Wasserstein distance with a threshold of 0.1 and categorical ones use Jensen-Shannon divergence with a threshold of 0.1. For text, it trains a domain classifier and flags drift when ROC AUC exceeds 0.55 on large data. PSI and KL divergence are available as selectable methods but are not defaults. Versions change defaults, so check the documentation for your installed release. The design lesson is the switch itself: p-value tests for small samples, effect-size distances for large ones.

Figure 2: Choosing a drift statistic by feature type and sample size. Small samples favour p-value tests such as KS and chi-square; large samples favour effect-size distances such as Wasserstein and Jensen-Shannon.

Choosing a statistic

Statistic	Feature type	Returns	Strength	Weakness
PSI	Binned numeric or categorical	Index, convention thresholds	Auditable, intuitive	Binning sensitivity, no null distribution
KS test	Continuous	D and p-value	No binning, nonparametric	Over-sensitive at large n, weak in tails, bad with ties
Wasserstein	Continuous	Distance in feature units	Magnitude-aware, tail-aware	Scale-dependent, needs normalisation
Jensen-Shannon	Categorical or binned	Bounded distance	Symmetric, bounded, smooth	Needs calibrated thresholds
Chi-square	Categorical	Statistic and p-value	Standard, exact for counts	Over-sensitive at large n, small-count issues
Domain classifier	Text, embeddings, multivariate	ROC AUC	Catches joint shifts	Needs training per check

Univariate tests miss joint drift

Every statistic above looks at one feature at a time. Two features can each keep an unchanged marginal distribution while their correlation flips, and a model that depends on the interaction will break. A domain classifier handles this: label reference rows 0 and current rows 1, train a classifier (for example gradient-boosted trees) to tell them apart on held-out folds, and read the ROC AUC. An AUC near 0.5 means indistinguishable. An AUC near 1.0 means a large shift, and feature importances from that classifier point to which features differ. It costs a training run per check, so run it daily rather than per minute. Other multivariate options include maximum mean discrepancy and, for embeddings, centroid distance, which we return to in the LLM section.

Deeper Analysis: Delayed Labels, Estimation Without Labels, and a Runnable Pipeline

The delayed-label problem

The metric you want is accuracy, or whatever business loss the model serves. The label you need to compute it is often unavailable when you need it. A loan default label arrives after months. A fraud chargeback arrives after 30 to 120 days. A churn label requires waiting out the churn window. A predictive-maintenance label arrives only if the machine actually fails, which, if the model works, it did not. Recommenders face selection bias on top: you only observe outcomes for items you showed.

Three strategies compose. First, join late labels and backfill: keep a durable inference log keyed by request identifier, and compute true performance per prediction cohort as labels land. This is the ground truth, but it tells you about last quarter. Second, use proxy metrics that arrive faster: click-through instead of purchase, early repayment behaviour instead of default, human-review overrides, customer complaints, or the agreement rate with a slower, more accurate “teacher” model on a sampled subset. Third, estimate performance without labels, covered next.

Sampling labels deliberately is underrated. If a human can label 400 randomly chosen recent predictions per week, the accuracy estimate has a standard error of roughly sqrt(p(1-p)/n). At p = 0.9 and n = 400 that is 0.015, so a 95 percent interval of about plus or minus 3 percentage points. That is enough to detect a 10-point collapse within a week and not enough to detect a 1-point decay, a useful reminder that label budgets set detectable effect sizes just as sample size does for the drift tests.

Performance estimation without labels

NannyML’s two estimators formalise the idea that, under covariate shift, you can predict a model’s performance from its inputs and outputs. Per its documentation, CBPE (confidence-based performance estimation) targets classification. It uses each prediction’s confidence score as the probability that the prediction is right, builds an expected confusion matrix over the window, and derives metrics such as accuracy, precision, recall and ROC AUC from it, with no targets required. DLE (direct loss estimation) targets regression: a “nanny model” (LightGBM by default) is trained on reference data to predict the child model’s per-row loss, and averaging the predicted losses estimates metrics like MAE.

The documented constraints are the whole story, so read them closely. CBPE requires well-calibrated probabilities (NannyML fits isotonic regression on reference data when this improves expected calibration error). It needs a reference set that is not the training set. It degrades on small analysis samples, for near-random models, and when drift moves data into regions the model never saw. Both estimators handle covariate shift and fail under concept drift, because a change in P(Y | X) invalidates the assumption that confidence means correctness. The mental model is this: label-free estimation turns a covariate-shift alarm into a number in the units you care about, which makes it far more actionable than “feature 17 has p less than 0.05”, but it cannot see the case that matters most when the world changes the rules.

A hand-computable CBPE-style illustration (illustrative numbers, not from NannyML): a binary classifier outputs calibrated probabilities. In a window of five predictions the model predicts positive with confidences 0.9, 0.8, 0.6, 0.7 and 0.95. If calibrated, the expected number correct is the sum of confidences, 3.95 of 5, so estimated accuracy is 79 percent with no labels. When covariate shift pushes more traffic toward the 0.6 region, the estimate falls even though no label has arrived. That is precisely the mechanism, and precisely why calibration quality bounds the estimate’s validity.

Figure 3: Estimating performance without labels. Predictions and confidences feed a label-free estimator immediately, while ground truth arrives later and validates or recalibrates the estimate.

The workflow in Figure 3 treats the estimator as a provisional measurement. When true labels arrive, compare estimated against realised metrics for the same cohort. Persistent gaps tell you the estimator’s assumptions (calibration, no concept drift) have broken, and that is itself a signal worth alerting on.

Windowing and statistical power

Windows turn a stream into samples. Use a tumbling window (non-overlapping, say one per hour or day) for dashboards and reproducible reports, and a sliding window with a stride for faster detection at higher compute cost. Fix the window size by sample count where traffic varies, not by time alone, otherwise a quiet night produces a 40-row window that either never alerts or always does.

Statistical power sets the minimum useful size. For a KS test at the 0.05 level the smallest detectable CDF gap is about 1.36 times sqrt((n + m) / (n m)); with a reference of 10,000 and a window of 500 rows this is roughly 0.064, meaning shifts smaller than about six points of CDF are statistically invisible at that window. That is acceptable if you care about big shifts, and it is a design input you should write down. Conversely, if your window is 500,000 rows against a 500,000 reference, anything over 0.003 is “significant”, and you must switch to effect-size thresholds, which is exactly why Evidently’s defaults flip from p-values to distances above 1,000 reference rows.

The multiple-testing trap

Monitor 200 features, each tested at 0.05, and under the null hypothesis (no drift whatsoever) the chance that at least one flags is 1 – 0.95^200, which is effectively 100 percent. Even at 50 features it is about 92 percent, and at 10 features about 40 percent. A per-feature alert policy will therefore page you every window, forever. Four mitigations work in practice.

First, aggregate: alert on the share of drifted features rather than any single one (a policy such as “dataset drift when at least half the columns drift”, which Evidently’s documentation gives as an example and lets you customise). Second, correct: apply Bonferroni (divide alpha by the number of tests) or, less conservatively, Benjamini-Hochberg false discovery control. Third, weight by importance: a drifting feature the model barely uses matters less than a drifting top-three feature, so rank by model feature importance and alert on weighted drift. Fourth, require persistence: alert only if the signal holds for k consecutive windows, which turns one-off noise into a rare event and costs you only k windows of latency.

A runnable drift report with Evidently

The following script builds a synthetic reference and a drifted production window, runs Evidently’s data drift preset, and independently computes PSI and the KS statistic so you can sanity-check the library. The Evidently calls follow the pattern in its current quickstart (Dataset.from_pandas, DataDefinition, Report([DataDriftPreset()]), report.run(current, reference)); the API changed substantially in the 0.7 line, so verify imports against your installed version. At the time of writing, PyPI lists 0.7.23 (released September 11, 2026) under Apache 2.0, requiring Python 3.10 or later.

# pip install evidently pandas numpy scipy
import numpy as np
import pandas as pd
from scipy import stats

from evidently import Dataset, DataDefinition, Report
from evidently.presets import DataDriftPreset

rng = np.random.default_rng(42)
N_REF, N_CUR = 5000, 1500

def make(n, shift=0.0, drift_cat=False):
    amount = rng.lognormal(mean=3.0 + shift, sigma=0.8, size=n)
    age = rng.normal(40, 12, n).clip(18, 90)
    p = [0.5, 0.3, 0.2] if not drift_cat else [0.25, 0.30, 0.45]
    channel = rng.choice(["web", "app", "pos"], size=n, p=p)
    score = 1 / (1 + np.exp(-(0.02 * (age - 40) + 0.001 * amount - 0.5)))
    return pd.DataFrame(
        {"amount": amount, "age": age, "channel": channel, "score": score}
    )

reference = make(N_REF)
current = make(N_CUR, shift=0.25, drift_cat=True)   # covariate shift

# 1) Library report
schema = DataDefinition(
    numerical_columns=["amount", "age", "score"],
    categorical_columns=["channel"],
)
ref_ds = Dataset.from_pandas(reference, data_definition=schema)
cur_ds = Dataset.from_pandas(current, data_definition=schema)
report = Report([DataDriftPreset()])
result = report.run(cur_ds, ref_ds)
result.save_html("drift_report.html")
summary = result.dict()          # inspect keys for your version

# 2) Independent PSI using reference quantile bins
def psi(expected, actual, bins=10, eps=1e-4):
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, eps, None), np.clip(a, eps, None)
    return float(np.sum((a - e) * np.log(a / e)))

for col in ["amount", "age", "score"]:
    ks = stats.ks_2samp(reference[col], current[col])
    print(f"{col:7s} PSI={psi(reference[col], current[col]):.3f} "
          f"KS D={ks.statistic:.3f} p={ks.pvalue:.2e}")

# 3) Categorical: chi-square on counts vs reference proportions
ref_p = reference["channel"].value_counts(normalize=True).sort_index()
cur_c = current["channel"].value_counts().sort_index()
chi = stats.chisquare(cur_c.values, f_exp=ref_p.values * cur_c.sum())
print("channel chi2 p =", chi.pvalue)

Expect amount and channel to flag strongly, age to stay quiet, and score to move modestly because it is derived from the drifted amount. Two details are worth internalising. The PSI function bins on the reference quantiles, so each reference bin holds about 10 percent by construction and the denominator never collapses, then clips tiny shares to an epsilon. And the library result and your hand computation should agree on direction, even if thresholds differ, which is a cheap regression test for your monitoring code.

For production, do not run this in a notebook. Wrap it in a scheduled job (Airflow, Dagster, Argo, or a cron-triggered container), read the last window from the log store, write the per-feature statistics to a metrics table, and let your existing alerting stack consume that table. Keep the HTML report as a debugging artefact attached to the alert, not as the monitoring mechanism itself.

Tooling in context

Evidently, as shown, is a library: you own the scheduling and storage. NannyML specialises in label-free performance estimation and in tracing drift to performance impact. whylogs takes a different route, capturing mergeable statistical profiles (counts, histograms, sketches) at the point of inference so you can compare profiles across time without retaining raw rows, which suits privacy-sensitive or very high-volume settings; check the project’s current maintenance status before adopting it. Commercial platforms such as Arize add managed storage, embeddings views and routing. The selection question is rarely which test is “best”. It is whether you need raw-data retention, whether you can build a scheduler, and whether your labels arrive fast enough to make estimation unnecessary.

Embedding Drift for LLM and RAG Systems

Classical drift tests assume tabular features. LLM applications have two awkward properties: inputs are free text, and there is often no label at all. The workable approach is to monitor representations and proxies instead of raw tokens.

Embedding drift. Embed each prompt (or retrieved chunk, or response) with a fixed, versioned embedding model and compare distributions in vector space. Cheap options are the cosine distance between window centroids, the average distance of window embeddings to the reference centroid, or a domain classifier on embeddings (Evidently’s text approach follows this idea, flagging drift when the classifier’s ROC AUC passes 0.55 on large data). Pin the embedding model: if you upgrade it, every vector moves and you will see a spurious, total drift. Reduce dimensionality (PCA to 20 to 50 components) before running tests like MMD, because distances concentrate in high dimensions.

Retrieval health for RAG. Track the distribution of top-k similarity scores, the fraction of queries whose best score falls below a floor, the share of queries that retrieve nothing, and document-age distribution. A new product line or a changed knowledge base shows up as a fall in top-1 similarity long before users complain. This is covariate drift in the query space, plus corpus drift on the other side.

Output and behaviour proxies. Response length, refusal rate, citation-presence rate, groundedness scores from an evaluator model, toxicity scores, user thumbs-down rate, and tool-call failure rates. These are noisy individually and meaningful as time series. Beware of evaluator drift: an LLM-as-judge is itself a model that changes when its provider updates it, so version it and calibrate it against a human-labelled sample.

Security shifts also look like drift. A wave of adversarial prompts shifts the input embedding distribution and the refusal rate together, which is a useful cross-check with the practices in the LLM red teaming and guardrails post. Monitoring will not tell you the intent, but a sudden cluster of unusual prompts is the cue to look.

Alerting, Retrain Triggers, and Shadow Deployment

Detection is only half a system. The other half decides what happens next, and Figure 4 shows a decision flow that separates quality incidents, drift without harm, and drift with harm.

Figure 4: From alert to action. Data quality failures go to the data owner, drift with confirmed or estimated degradation triggers retraining and a shadow deployment, and harmless drift only raises the watch level.

Tiered alerts

Define at least three tiers. Page (wake someone) only for confirmed or strongly estimated harm: a performance estimate below the agreed floor for several consecutive windows, a schema break, a missing feature, a prediction distribution collapse such as all scores identical. Ticket (next working day) for drift in important features with no estimated impact. Dashboard only for everything else. Attach to every alert the feature, the statistic, the window, the reference, a link to the HTML report and the model version. An alert that cannot be acted on in five minutes of context is a future muted alert.

Retrain triggers

There are three families. Schedule-based retraining (weekly, monthly) is simple and robust when data is cheap and labels flow. Drift-triggered retraining reacts to detected change, but retraining on data that is drifted because of a pipeline bug bakes the bug in, and retraining during a transient event (a promotion, an outage) learns an abnormal regime. Performance-triggered retraining reacts to measured or estimated degradation and is the most defensible. Most teams run a schedule as a floor and a performance trigger as an accelerator, with drift as a diagnostic that raises the priority of the investigation.

A retrain is not a fix until proven. Evaluate the candidate on a recent labelled window, not only on a random split of the new training data, and compare against the incumbent on the same window.

Shadow and canary deployment

Before promoting a retrained model, run it in shadow mode: it receives live traffic and logs predictions, but its output is not served. Compare shadow against the incumbent on agreement rate, score distribution, latency and, once labels arrive, accuracy. Then canary a small traffic slice with automatic rollback conditions defined in advance. Shadow deployment also exposes training-serving skew, since the candidate runs the real serving path. If the candidate is better on estimated metrics but its output distribution is wildly different, treat that as a finding, not a win.

Trade-offs, Gotchas, and What Goes Wrong

Statistical significance is not practical significance. At scale, KS and chi-square flag trivial changes. Prefer effect-size distances with thresholds set from historical behaviour, for example the 99th percentile of the metric measured between healthy adjacent windows. That empirical null is more honest than any textbook threshold, including PSI’s 0.1 and 0.25 conventions.

The reference rots. A reference frozen at launch becomes obsolete as the world legitimately evolves, and everything drifts forever. A reference that follows production too closely hides slow drift, because each window looks like its predecessor. Pair a long-lived anchor reference (detects cumulative change) with a short rolling one (detects sudden change).

Seasonality masquerades as drift. Weekday and holiday cycles produce regular PSI spikes. Compare like with like (same weekday, same season) or detrend before testing.

Feedback loops contaminate the signal. If model decisions influence future data (loan approvals shape who applies, recommendations shape what is clicked), the data you monitor is partly a product of the model. Drift can be self-inflicted, and labels are censored by your own decisions. Hold out a small random exploration slice to keep an unbiased view.

Monitoring the wrong thing. Testing raw inputs instead of model-ready features, hashing high-cardinality identifiers into tests designed for continuous data, or comparing features that the model never uses all produce noise. Start from the model’s top features.

Calibration silently breaking. Label-free estimators lean on calibrated confidences. Retraining, class reweighting or temperature changes alter calibration, and an estimator calibrated on yesterday’s model misleads. Recalibrate on each model version and track expected calibration error as a metric.

The monitor can fail silently. A scheduler that stops, a log pipeline that drops half the records, or an empty window all yield “no alerts”. Monitor the monitor: alert on missing windows, on log volume dropping against traffic, and on stale metric timestamps.

Privacy and cost. Logging every feature vector may be restricted by law or cost. Sampling a fixed percentage (stratified so rare classes survive) or storing mergeable profiles keeps both under control.

Practical Recommendations

Begin with the cheapest honest signals and add sophistication when evidence demands it. Log model-ready features, predictions, scores and a request identifier from day one, because you cannot reconstruct this later. Put data-quality checks in front of drift tests so that pipeline bugs route to data owners rather than triggering retrains. Choose a statistic per feature type and sample size, with a p-value test for small windows and an effect-size distance for large ones, and set thresholds from the empirical distribution of healthy windows rather than from folklore.

Control false alarms structurally: aggregate across features, weight by importance, apply a multiple-testing correction, and require persistence over several windows. Treat labels as a budget: join late labels automatically, sample a small human-labelled batch each week, and use label-free estimation only where calibration holds and concept drift is unlikely. Version your reference dataset alongside the model. Gate every retrain behind a shadow run and a canary with rollback criteria written down in advance.

A starting checklist:

[ ] Inference logging of post-transformation features, scores, model version and request ID
[ ] Schema, null-rate, range and unseen-category checks as a separate tier
[ ] Per-feature statistic chosen by type and sample size, thresholds from healthy history
[ ] Prediction-score drift monitored independently of input drift
[ ] Windows sized for the smallest shift you care about, written in the runbook
[ ] Alerts aggregated, importance-weighted, persistent, and tiered
[ ] Late-label join plus a weekly random labelled sample
[ ] Reference versioned with the model, with an anchor plus a rolling comparison
[ ] Shadow and canary gates before any promotion
[ ] Monitoring of the monitor itself: missing windows and log volume

Frequently Asked Questions

What is the difference between data drift and concept drift?

Data drift (covariate shift) means the distribution of inputs P(X) has changed while the relationship between inputs and labels may still hold. Concept drift means P(Y given X) has changed, so identical inputs now deserve different outputs. Data drift is detectable from inputs alone. Pure concept drift generally needs ground truth or proxy outcome metrics to detect, which is why it is more dangerous and why delayed labels are the central difficulty of production monitoring.

Which drift test is best: PSI, KS or Wasserstein?

No single test wins. PSI is auditable and common in credit risk, but depends on binning and has convention-based thresholds. KS needs no bins but becomes oversensitive at large sample sizes and is weak in the tails. Wasserstein reports shift magnitude and handles tails but needs normalisation. A practical policy mirrors Evidently’s: p-value tests for small samples, effect-size distances for large ones, with thresholds calibrated on healthy historical windows.

What PSI value indicates significant drift?

The common credit-risk convention treats PSI below 0.1 as stable, 0.1 to 0.25 as moderate shift, and above 0.25 as significant shift. These cut-offs are rules of thumb, not statistical thresholds, and they vary with the number of bins and the sample size. Calibrate them against PSI values measured between healthy adjacent windows of your own data, and combine the number with a view of which bins moved.

How do you monitor a model when ground truth labels are delayed?

Combine three tactics: monitor input and prediction drift in real time, use proxy outcomes that arrive faster, and estimate performance without labels using methods such as CBPE for classification or DLE for regression. Also join late labels by request ID to compute true metrics retrospectively. Remember that label-free estimators handle covariate shift but not concept drift, and CBPE requires calibrated probabilities.

How often should I retrain a model after detecting drift?

Drift alone should not trigger automatic retraining. First rule out data pipeline bugs, then check whether estimated or measured performance has actually degraded. A common policy is a scheduled retrain as a floor, a performance-triggered retrain as an accelerator, and drift as a diagnostic. Always validate the candidate on a recent labelled window and run it in shadow before promotion.

How do you detect drift in LLM and RAG applications?

Track embedding distributions with a pinned embedding model using centroid distance or a domain classifier, retrieval similarity scores and no-result rates, and output proxies such as refusal rate, length, groundedness and user feedback. Version any evaluator model, since an LLM judge can itself drift. Expect noisy individual signals and rely on trends across windows rather than single-point thresholds.