Conformal Prediction: Distribution-Free Uncertainty with Coverage Guarantees

A model that says “42.7” is making a claim it cannot support. Neural networks emit confident softmax scores that are routinely miscalibrated, Bayesian posteriors are only as good as their priors, and bootstrap intervals quietly assume the training distribution is the deployment distribution. Meanwhile a maintenance planner, a clinician or a risk engine needs one thing: a set or interval that contains the truth at a rate you chose in advance.

Conformal prediction is the cleanest tool we have for that job. It wraps any trained model, treats it as a black box, and uses a small held-out calibration set to turn point predictions into prediction sets with a finite-sample coverage guarantee. It needs no assumption about the data distribution beyond exchangeability, and it works with gradient-boosted trees, deep nets and LLM pipelines alike.

You will leave with the proof idea in one page, a worked numeric example you can check by hand, runnable Python for split conformal and conformalized quantile regression, and a clear map of where the guarantee breaks.

What this covers: the guarantee and why it holds, split conformal with numbers, conformalized quantile regression, adaptive conformal inference under drift, classification sets, LLM outputs, industrial forecasting, limits, and libraries.

Context and Background

Conformal prediction was developed by Vladimir Vovk, Alex Gammerman and Glenn Shafer in the late 1990s and early 2000s, and consolidated in the book Algorithmic Learning in a Random World. The original “full” or transductive version refits the model for every candidate label, which is statistically elegant and computationally brutal. The practical breakthrough was the inductive, or split, variant: fit once, calibrate once, and reuse the calibration quantile for every future query.

The field then split into two lines. One line, led by Jing Lei, Larry Wasserman and colleagues, framed conformal inference as distribution-free predictive inference for regression. The other, popularised for machine learning practitioners by Anastasios Angelopoulos and Stephen Bates in their A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification (arXiv:2107.07511), emphasised code and calibration sets. If you only read one companion text, read that one.

Why is this relevant now? Because the dominant failure of deployed ML in 2026 is not low accuracy, it is unearned confidence. Teams ship forecasts into digital twins, anomaly detectors into plants, and LLM agents into workflows, and downstream logic needs a defensible notion of “how sure”. We covered the validation side of that problem in our post on digital twin verification, validation and uncertainty quantification; conformal prediction is the model-agnostic layer that makes the uncertainty part auditable.

The standard alternatives each carry a hidden assumption. Gaussian residual intervals assume homoscedastic, light-tailed errors. Deep ensembles and MC dropout give a spread, but nothing ties that spread to a coverage rate. Bayesian methods are calibrated only if the model is well specified. Quantile regression targets the right object but can undercover when the quantile model itself is wrong. Conformal methods do not replace these; they repair them. You can feed any of them in as the base model and get back a set with a guarantee.

The guarantee, stated precisely, is this. Pick a miscoverage level alpha, for example 0.1. Given exchangeable calibration and test data, the prediction set C(X) satisfies the following for a fresh test pair.

P(Y in C(X)) >= 1 – alpha

That statement is marginal, finite-sample and distribution-free. Each of those three adjectives matters, and the rest of this post is about what they buy and what they cost.

How Conformal Prediction Works: The Reference Architecture

The recipe in one paragraph: hold out a calibration set, compute a nonconformity score for each calibration point, take a carefully chosen empirical quantile of those scores, and include in the prediction set every candidate output whose score falls at or below that quantile. For regression with an absolute-error score, the result is the point forecast plus or minus one number.

Figure 1: Split conformal prediction. Training and calibration data are disjoint, and a single quantile of calibration scores converts any model into an interval with a coverage guarantee.

The diagram shows the entire algorithm. Data is partitioned, the model is fit on one part only, scores are computed on the other, and one quantile is taken. Nothing about the model is inspected, which is why the method is described as post-hoc and model-agnostic.

The split conformal algorithm

Split conformal prediction proceeds in five steps.

Split labelled data into a proper training set and a calibration set of size n.
Train any model f on the proper training set.
For each calibration pair (x_i, y_i), compute the score s_i = |y_i – f(x_i)|.
Let k = ceil((n + 1)(1 – alpha)) and set q_hat to the k-th smallest score. If k exceeds n, q_hat is infinite, meaning the honest answer is “no finite interval at this level”.
For a new x, return the interval [f(x) – q_hat, f(x) + q_hat].

The detail that trips up implementers is the (n + 1) in step 4. A naive empirical quantile at level 1 – alpha undercovers slightly because the test point is itself a draw from the same distribution. The finite-sample correction accounts for it, and skipping it is the most common bug in hand-rolled implementations.

Why exchangeability gives the guarantee

Exchangeability means the joint distribution of the data is invariant under permutation. IID data is exchangeable, but exchangeability is weaker, and it is the only assumption the guarantee uses.

Here is the argument. Consider the n calibration scores plus the test score, n + 1 numbers in total. If they are exchangeable, then the rank of the test score among all n + 1 is uniform over positions 1 to n + 1. The test score is at most the k-th smallest calibration score exactly when its overall rank is at most k, which has probability k/(n + 1) when there are no ties. With k = ceil((n + 1)(1 – alpha)) that probability is at least 1 – alpha.

Figure 2: Proof sketch of the coverage guarantee. Uniform rank of the test score under exchangeability yields the lower bound, and the absence of ties yields the upper bound.

The proof also delivers an upper bound. If scores are continuous, so ties have probability zero, coverage is at most 1 – alpha + 1/(n + 1). This is why conformal sets are not wildly conservative: the guarantee is two-sided in expectation, and the excess shrinks as the calibration set grows.

Notice what the proof never touched: the model, the feature dimension, the noise distribution. It used only the symmetry of the scores. That is the sense in which the method is distribution-free. For the formal treatment, see Shafer and Vovk’s tutorial, A Tutorial on Conformal Prediction (Journal of Machine Learning Research, 2008), and the Angelopoulos and Bates text cited above.

The coverage guarantee is a random variable

A subtle point practitioners miss: the guarantee holds on average over the draw of the calibration set. For one fixed calibration set the realised coverage is itself random. Under exchangeability and continuous scores, the conditional coverage given the calibration data follows a Beta distribution with parameters k and n + 1 – k. This distribution is exact, and it tells you how much calibration data you need.

Take alpha = 0.1. With n = 99, k = ceil(100 x 0.9) = 90, and realised coverage is Beta(90, 10): mean 0.90 and a standard deviation of about 0.030. A single deployment can easily land at 0.87 or 0.93. With n = 999, k = 900 and coverage is Beta(900, 100): mean 0.90 and a standard deviation of about 0.0095. So a thousand calibration points gives you coverage within roughly plus or minus two points at 95% confidence over calibration draws. Because these figures follow directly from the Beta formula, they are derived rather than benchmarked.

Two practical rules follow. First, if you need a tight, high-probability guarantee on realised coverage, size the calibration set from the Beta distribution rather than guessing. Second, if you need alpha = 0.01, you need at least 99 calibration points before the quantile is even finite, since k = ceil((n + 1) x 0.99) must not exceed n.

Deeper Analysis: Worked Numbers, Code, and Adaptive Variants

A hand-checkable example

Suppose a model predicts tank temperature, and nine calibration residuals (absolute errors, in degrees C) are: 0.4, 1.1, 0.7, 0.2, 1.6, 0.9, 0.5, 1.3, 0.3. We want alpha = 0.2, that is 80% coverage. Illustrative numbers, chosen for arithmetic clarity.

Sorted: 0.2, 0.3, 0.4, 0.5, 0.7, 0.9, 1.1, 1.3, 1.6. With n = 9, k = ceil(10 x 0.8) = 8. The eighth smallest score is 1.3, so q_hat = 1.3.

If the model predicts 20.0 degrees C for a new input, the interval is [18.7, 21.3]. Check the probability logic: the test score ranks uniformly among 10 values, and the interval covers when its rank is 1 through 8, which is 8/10 = 80%. The naive empirical 80th percentile of nine values would pick roughly the seventh or eighth value depending on the interpolation rule, and many implementations round down to 1.1, which would give 7/10 = 70% coverage. The correction is not cosmetic.

Runnable split conformal in Python

The following uses only NumPy and scikit-learn. It trains a gradient-boosted regressor on heteroscedastic synthetic data, calibrates, and reports empirical coverage.

import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(7)
n_total = 6000
X = rng.uniform(0, 10, size=(n_total, 1))
# noise grows with x: heteroscedastic by construction
y = np.sin(X[:, 0]) * 3 + rng.normal(0, 0.2 + 0.3 * X[:, 0])

X_tr, X_rest, y_tr, y_rest = train_test_split(X, y, test_size=0.5, random_state=0)
X_cal, X_te, y_cal, y_te = train_test_split(X_rest, y_rest, test_size=0.6, random_state=1)

model = GradientBoostingRegressor(random_state=0).fit(X_tr, y_tr)

alpha = 0.1
scores = np.abs(y_cal - model.predict(X_cal))
n = len(scores)
k = int(np.ceil((n + 1) * (1 - alpha)))
q_hat = np.sort(scores)[k - 1] if k <= n else np.inf

pred = model.predict(X_te)
lo, hi = pred - q_hat, pred + q_hat
covered = (y_te >= lo) & (y_te <= hi)
print(f"n_cal={n} k={k} q_hat={q_hat:.3f}")
print(f"marginal coverage = {covered.mean():.3f} (target {1-alpha:.2f})")

# conditional view: coverage by input region
for a, b in [(0, 3), (3, 6), (6, 10)]:
    m = (X_te[:, 0] >= a) & (X_te[:, 0] < b)
    print(f"x in [{a},{b}): coverage = {covered[m].mean():.3f}, width = {2*q_hat:.2f}")

Run this and you will see marginal coverage near 0.90, as promised, but a revealing pattern by region. Because the noise grows with x, the constant-width interval overcovers at small x and undercovers at large x. The guarantee holds on average, and the average hides a structural miss. This motivates the next method.

Conformalized quantile regression

Romano, Patterson and Candes introduced conformalized quantile regression (CQR) at NeurIPS 2019. The idea is to train two quantile regressors, at levels alpha/2 and 1 – alpha/2, which give a lower and upper estimate that adapt to the input. Those raw quantile estimates have no guarantee. CQR then conformalizes them: it computes a score that measures how far the true value falls outside the quantile band, and adds the calibrated correction.

Figure 3: Conformalized quantile regression. Learned quantile bands supply adaptivity, and a conformal correction supplies the coverage guarantee.

The conformity score is s_i = max(lo(x_i) – y_i, y_i – hi(x_i)). It is negative when the point sits inside the band (how deep inside), and positive when outside (how far). Take q_hat as the same corrected quantile of these scores, and return [lo(x) – q_hat, hi(x) + q_hat]. If the quantile models were too narrow, q_hat is positive and the band widens. If they were too wide, q_hat is negative and the band shrinks.

from sklearn.ensemble import GradientBoostingRegressor

lo_m = GradientBoostingRegressor(loss="quantile", alpha=alpha / 2, random_state=0).fit(X_tr, y_tr)
hi_m = GradientBoostingRegressor(loss="quantile", alpha=1 - alpha / 2, random_state=0).fit(X_tr, y_tr)

lo_c, hi_c = lo_m.predict(X_cal), hi_m.predict(X_cal)
s = np.maximum(lo_c - y_cal, y_cal - hi_c)
k = int(np.ceil((len(s) + 1) * (1 - alpha)))
q_cqr = np.sort(s)[k - 1]

lo_t, hi_t = lo_m.predict(X_te) - q_cqr, hi_m.predict(X_te) + q_cqr
cov = (y_te >= lo_t) & (y_te <= hi_t)
print(f"CQR marginal coverage = {cov.mean():.3f}, q_cqr = {q_cqr:.3f}")
for a, b in [(0, 3), (3, 6), (6, 10)]:
    m = (X_te[:, 0] >= a) & (X_te[:, 0] < b)
    print(f"x in [{a},{b}): coverage = {cov[m].mean():.3f}, mean width = {(hi_t-lo_t)[m].mean():.2f}")

You should observe that the mean width grows with x, and per-region coverage is much closer to 0.90 than with the constant-width interval. CQR does not guarantee conditional coverage, but it inherits adaptivity from the quantile models. Where those models are good, intervals are tight and well-distributed. Where they are bad, the conformal step still rescues the marginal guarantee. Intervals are never worse than valid, only less sharp.

Which score should you use? Absolute residuals are simple and fine for homoscedastic problems. Normalised scores, |y – f(x)| / sigma_hat(x), where sigma_hat is a second model for error scale, are a middle path. CQR is the default I reach for with tabular and sensor data. Choice of score changes efficiency (interval width), never validity.

Libraries: MAPIE and crepes

You do not need to hand-roll this in production. MAPIE (Model Agnostic Prediction Interval Estimator) is the most widely used open-source library; its documentation describes a v1 API with classes such as a split conformal regressor and a ConformalizedQuantileRegressor. The API changed between 0.x and 1.x (the older MapieQuantileRegressor appears in 0.9 docs), so pin your version and check the current docs before copying snippets. Henrik Bostrom’s crepes package takes a different path, offering conformal regressors, classifiers and predictive systems with normalisation and Mondrian (class- or group-conditional) categories, described in a COPA 2024 paper, Conformal Prediction in Python with crepes. I have not benchmarked these two libraries against each other, so treat the choice as a matter of API fit: MAPIE for scikit-learn pipelines and time-series helpers, crepes for Mondrian categories and conformal predictive systems.

Adaptive conformal inference under drift

Everything above relies on exchangeability, and time series violate it. Sensor streams drift, seasons turn, and equipment ages. Gibbs and Candes proposed adaptive conformal inference (ACI) at NeurIPS 2021 to keep long-run coverage correct even when the distribution shifts arbitrarily.

ACI treats the miscoverage level as a control variable. At each step t, you build the interval at the current level alpha_t, observe whether the truth fell outside it (err_t = 1 if so), and update:

alpha_(t+1) = alpha_t + gamma x (alpha – err_t)

When errors occur too often, alpha_t shrinks and intervals widen. When there are no misses, alpha_t grows and intervals tighten. The step size gamma trades responsiveness for stability. The authors show a bound on long-run miscoverage that holds for any data sequence: the average error over T steps differs from alpha by at most (max(alpha_1, 1 – alpha_1) + gamma) / (gamma T), which vanishes as T grows. This is an average-over-time guarantee, not a per-step one, and I am quoting the bound from memory of the paper, so verify the constants against the original before citing it formally.

Figure 4: Adaptive conformal inference on a streaming sensor. Each realised miss or hit nudges the working miscoverage level, widening or narrowing the next interval.

def aci_stream(y, point_pred, scores_init, alpha=0.1, gamma=0.01):
    """Online ACI with a rolling score buffer. Returns per-step lo, hi, errs."""
    buf = list(scores_init)
    a_t = alpha
    lo, hi, errs = [], [], []
    for yt, pt in zip(y, point_pred):
        level = min(max(1 - a_t, 0.0), 1.0)
        s = np.sort(buf)
        if level >= 1.0:
            q = np.inf
        else:
            k = min(int(np.ceil((len(s) + 1) * level)), len(s))
            q = s[k - 1]
        lo.append(pt - q); hi.append(pt + q)
        err = float(not (pt - q <= yt <= pt + q))
        errs.append(err)
        a_t = a_t + gamma * (alpha - err)
        buf.append(abs(yt - pt)); buf = buf[-500:]   # rolling window
    return np.array(lo), np.array(hi), np.array(errs)

Two engineering notes. When alpha_t drops to zero or below, the interval is the whole real line, a signal that the model has broken rather than a bug. And gamma around 0.005 to 0.05 is the range discussed in the literature, but the right value depends on how fast your process drifts; tune it on a backtest rather than copying a default. Barber, Candes, Ramdas and Tibshirani’s Conformal Prediction Beyond Exchangeability (Annals of Statistics, 2023) offers a complementary route using weighted quantiles that down-weight old data, with a coverage gap that depends on how much the distribution moved.

For classification, language and industrial uses, the same machinery applies with a different score, which the next section develops.

Classification, LLMs, and Industrial Forecasting

Prediction sets for classifiers

For a classifier, the output is a set of labels, not an interval. The simplest score is s_i = 1 – p_hat(y_i | x_i), the softmax probability assigned to the true class. The prediction set for a new x contains every class whose probability is at least 1 – q_hat. This is the “least ambiguous set-valued classifier” (LAC) approach. It gives the smallest average set size but can return an empty set, and its coverage across classes can be uneven.

Romano, Sesia and Candes introduced Adaptive Prediction Sets (APS) at NeurIPS 2020. The APS score sums the sorted class probabilities down to and including the true class, with a randomisation term to hit exactness. Sets then grow on hard examples and shrink on easy ones, giving better conditional behaviour at the cost of larger sets. Angelopoulos, Bates, Malik and Jordan then proposed Regularized APS (RAPS) at ICLR 2021, which adds a penalty for including low-ranked classes. In ImageNet-scale experiments, they reported much smaller sets than APS; I will not quote their figures here, because you should read them in the paper rather than from me.

The design choice is straightforward. Use LAC when mean set size is the priority and you can tolerate class-conditional unevenness. Use APS or RAPS when you want sets that expand honestly on ambiguous inputs. If you need per-class coverage, calibrate per class (the Mondrian variant), which requires enough calibration examples in the rarest class.

Conformal methods for LLM outputs

LLMs complicate the story because the output space is open-ended and the notion of “correct” is fuzzy. Three practical patterns exist.

The first is conformal selection from a candidate pool for tasks with a checkable answer: multiple-choice question answering, classification by prompt, or retrieval. Use the model’s option probabilities, or a verbalised confidence, as the score and build sets exactly as for classifiers. A set of size one is a confident answer; a larger set is a legitimate “I am not sure” that can trigger escalation.

The second is Conformal Language Modeling by Quach, Fisch, Schuster, Yala, Sohn, Jaakkola and Barzilay (ICLR 2024). Rather than a set of labels, they sample a set of generations and calibrate a stopping rule and a rejection rule, so that the final set contains at least one acceptable response with a user-specified probability. Acceptability is judged by a task-specific admission function, which in practice is a metric or a human label on the calibration set. The authors report experiments on open-domain QA, text summarisation and radiology report generation.

The third is claim-level filtering: decompose an output into atomic claims, score each with a factuality estimator, and keep the claims above a calibrated threshold so that the filtered answer’s error rate is controlled. The guarantee then depends on the factuality scorer’s labels in the calibration set. This is where our earlier discussion of LLM-as-a-judge evaluation, bias and calibration matters directly: if the judge that supplies calibration labels is biased, the conformal guarantee is valid with respect to the judge, not with respect to reality. Conformal prediction faithfully inherits label quality.

A caution on exchangeability here. Calibration prompts must resemble production prompts. A calibration set of curated benchmark questions will not certify behaviour on adversarial user input, and a prompt-template change can break exchangeability silently. Recalibrate whenever the template, model version or retrieval corpus changes.

Industrial sensors and forecasting

Industrial settings are a natural fit because the cost of a wrong point forecast is concrete, and because operators already think in tolerance bands. Typical applications include remaining-useful-life estimates, energy and load forecasts, quality prediction for a batch process, and soft sensors that infer unmeasured variables. For a digital twin, conformal bands around a surrogate model’s output give a defensible signal for “the twin and reality now disagree beyond what calibration says is normal”.

Three adaptations are needed. First, respect time order: calibrate on a block that follows the training block, never a random shuffle, or leakage will make the intervals look too good. Second, use ACI or weighted conformal methods because plant data drifts. Third, handle multi-step horizons separately: the error distribution at a 24-hour horizon differs from a 1-hour horizon, so calibrate a separate q_hat per horizon.

A worked sketch. A load-forecasting model has a 24-step horizon. You hold out the last 4,000 hourly points as calibration, giving about 4,000 residuals per horizon step. At alpha = 0.1 and n = 4,000, k = ceil(4001 x 0.9) = 3,601, and the Beta standard deviation of realised coverage is about sqrt(0.09 / 4001) = 0.0047, so calibration noise is small relative to drift. The bigger risk is seasonal: a calibration block drawn in summer says little about a winter peak. Mondrian-style conditioning on a regime variable (season, shift, product grade) is the practical fix, at the price of fewer points per bucket.

For the forecasting side of this stack, see our guide to time-series forecasting at the edge in production, which covers the model half; conformal prediction supplies the interval half. A lightweight edge deployment is attractive because the conformal layer is just a sorted array and a lookup, so it adds negligible latency and memory.

Choosing a method: a decision matrix

The variants differ in what they cost and what they protect. The table summarises the trade-offs as qualitative judgements drawn from the mechanisms above, not from a benchmark.

Situation	Method	Strength	Main cost
Plenty of data, roughly constant noise	Split conformal, absolute residual	Simplest, easiest to audit	Constant width, weak conditional behaviour
Noise varies with input	CQR or normalised scores	Adaptive width, still valid	Extra models to train and tune
Small dataset	Cross-conformal or jackknife+	Uses all data for training	More fits, weaker or approximate guarantee
Streaming data with drift	ACI or weighted conformal	Long-run coverage under shift	Time-averaged guarantee, tuning of gamma
Multi-class with ambiguity	APS or RAPS	Sets grow on hard inputs	Larger sets than LAC
Rare groups matter	Mondrian partitions	Per-group coverage	Needs enough calibration points per group

A useful reading of this table: validity is cheap and almost every row delivers it, so the real selection criterion is efficiency under your specific failure pattern. Pick the row whose “main cost” you can afford, then verify sharpness and per-slice coverage on a held-out block. If two rows tie, prefer the one with fewer moving parts, because every extra model is another place for exchangeability to be violated by a pipeline bug rather than by nature.

One more consideration is latency and governance. Split conformal adds one sorted array and a lookup at inference time, so it fits on a microcontroller-class gateway if the base model does. Because the calibration quantile is a single stored number per bucket, it is also trivially versioned and auditable: you can attach the calibration dataset hash, the alpha, and the q_hat to every model release, and an assessor can recompute coverage without touching the model internals. That reproducibility is, in my view, an underrated reason to adopt the method in regulated settings.

Trade-offs, Gotchas, and What Goes Wrong

Marginal is not conditional. The guarantee averages over the whole input distribution. A 90% marginal guarantee is compatible with 99% coverage on easy inputs and 60% on a hard subgroup. Barber, Candes, Ramdas and Tibshirani proved that exact distribution-free conditional coverage is impossible for non-trivial intervals in general, so no clever trick removes this. What you can do is partition (Mondrian), normalise scores, or use CQR, and then measure coverage by slice on held-out data.

Exchangeability fails quietly. Covariate shift, label shift, temporal drift, and duplicates between calibration and test sets all break it. Duplicates are the nastiest: if near-copies of calibration rows leak into training, the model is overfit to them and the scores are too small, so the interval is too narrow. Always deduplicate and split by group (machine, patient, site, day) when rows are correlated.

The calibration set is not free. Split conformal sacrifices data for calibration. For small datasets, cross-conformal and jackknife+ methods reuse data more efficiently, with slightly weaker or approximate guarantees; Barber, Candes, Ramdas and Tibshirani’s jackknife+ gives a 1 – 2 alpha guarantee in the worst case. If you can only afford one hundred calibration points, accept that realised coverage will wobble by about three points.

Set size is the real metric. A trivially valid method returns the whole label space. Always report coverage and sharpness (interval width or set size) together, and compare methods at equal coverage. A model that is merely valid is not necessarily useful.

Guarantees do not compose automatically. If you build a downstream decision from several conformal outputs, for example a joint interval for ten sensors, the joint coverage is not 1 – alpha. Use a Bonferroni split or multivariate conformal methods, and expect wider regions.

Label quality sets the ceiling. Noisy labels, a biased LLM judge, or a delayed ground truth shifts the meaning of coverage. In streaming settings with delayed labels, ACI updates arrive late, and the effective gamma must account for the delay.

Practical Recommendations

Start with the simplest thing that is valid and measure from there. For a tabular or sensor regression problem, train your best model, then conformalize it with CQR when you suspect heteroscedasticity, and with absolute residuals otherwise. Hold out a calibration block that matches the deployment ordering, and size it using the Beta formula rather than a rule of thumb.

Treat coverage as a monitored production metric, not a one-off test. Log whether each realised value fell inside its interval, compute a rolling hit rate, and alert when it leaves a band around 1 – alpha. If it does, you have found drift before your users did, and ACI or recalibration is the response. This turns the uncertainty layer into an early-warning instrument for the whole system.

For classification and LLM pipelines, report average set size and the fraction of singleton sets, and route non-singleton sets to a human or a stronger model. That policy converts uncertainty into a cost-aware escalation rule.

A short checklist:

Confirm the unit of exchangeability and split by group, not by row, when data is correlated.
Apply the ceil((n + 1)(1 – alpha)) correction; test it on synthetic data with known coverage.
Size calibration data from the Beta distribution, and ensure n is at least 1/alpha – 1.
Evaluate coverage per slice (regime, subgroup, horizon), not only in aggregate.
Report sharpness alongside coverage.
Recalibrate after any model, prompt template, sensor or corpus change.
Use ACI or weighted conformal for time series, and monitor rolling hit rate.
Pin library versions; MAPIE’s API changed across major versions.

Frequently Asked Questions

What is conformal prediction in simple terms?

Conformal prediction is a wrapper that turns any model’s predictions into sets or intervals that contain the true answer with a chosen probability, such as 90%. It measures the model’s errors on held-out calibration data, picks a corrected quantile of those errors, and uses it to size each prediction set. The guarantee holds for finite samples and any data distribution, provided calibration and test data are exchangeable.

Does conformal prediction work with deep learning and LLMs?

Yes, because it treats the model as a black box and needs only a score computed from its outputs. For classifiers, the score uses softmax probabilities; for regressors, residuals or quantile bands; for LLMs, option probabilities, sampled generations with an acceptability test, or claim-level factuality scores. The guarantee is only as meaningful as the calibration labels and the match between calibration prompts and production traffic.

What is the difference between marginal and conditional coverage?

Marginal coverage means that, averaged over all inputs, the interval contains the truth at least 1 – alpha of the time. Conditional coverage would mean the same holds for every specific input or subgroup. Conformal prediction guarantees only the marginal kind. Exact conditional coverage is impossible in a distribution-free setting, so practitioners approximate it with normalised scores, CQR or Mondrian partitions.

How much calibration data do I need?

You need at least 1/alpha – 1 points for a finite interval, so 9 for alpha = 0.1. For stable realised coverage, use the Beta(k, n + 1 – k) distribution. With alpha = 0.1, n = 99 gives a coverage standard deviation near 0.03, while n = 999 gives about 0.0095. A few hundred to a few thousand points is a common practical range.

What happens when the data distribution drifts?

The exchangeability assumption fails and the standard guarantee no longer applies. Adaptive conformal inference restores long-run average coverage by adjusting the working miscoverage level after each observed hit or miss. Weighted conformal methods down-weight old or dissimilar data instead. Either way, monitor the rolling hit rate and recalibrate when the model, sensor or environment changes.

Which Python library should I use?

MAPIE integrates with scikit-learn estimators and offers regression, classification and time-series conformal methods; check the version-specific docs because the API changed in v1. The crepes package offers conformal regressors, classifiers and predictive systems with Mondrian categories. For learning or tight control, the NumPy implementations in this post are about twenty lines and easy to audit.

Conformal Prediction: Distribution-Free Uncertainty Quantification with Coverage Guarantees for ML (2026)