LLM Evaluation Harness Architecture: lm-eval-harness, HELM, Inspect and Benchmark Contamination (2026)

Two teams evaluate the same open-weights model on the same benchmark and publish numbers that differ by more than ten points. Neither team made a mistake in the usual sense. The gap came from the llm evaluation harness each one used: how it formatted the prompt, which tokens it scored, where it cut the generation, and what it did when the model produced an answer in an unexpected shape. A benchmark score is not a property of a model. It is a property of a model, a harness, a prompt format, a decoding configuration, and a sample of questions.

That matters more in 2026 than it did two years ago. Release decisions, procurement shortlists, and regression gates in CI all lean on these numbers, and the models being scored may have seen the test questions during training. This post opens up the harness as a piece of software: its internal stages, how EleutherAI’s lm-evaluation-harness, Stanford HELM, UK AISI’s Inspect, and OpenAI Evals differ, why scores drift between them, how to detect benchmark contamination, and how to put honest error bars on a comparison. You will leave with a working bootstrap script and a CI design.

What this covers: harness anatomy, a four-framework comparison, the sources of cross-harness score drift, contamination detection, statistical rigor with runnable code, and evals in CI.

Context and Background

Before 2020, evaluating a language model meant running one benchmark’s reference script. GLUE and SuperGLUE shipped their own loaders and metrics, and each paper reimplemented the rest. The GPT-3 paper, with dozens of few-shot tasks evaluated on a single model family, exposed the cost of that fragmentation: nobody outside the authors could reproduce a table without rebuilding the prompt pipeline. EleutherAI’s lm-evaluation-harness emerged to fix this by putting many tasks behind one command line, and Stanford’s HELM (Liang et al., 2022, arXiv:2211.09110) argued for evaluating along many axes, not accuracy alone.

The landscape today has four reference points. lm-evaluation-harness is MIT-licensed, supports Hugging Face transformers, vLLM, SGLang, llama.cpp servers, and hosted APIs, and its releases page lists v0.4.13 as the latest at the time of writing. HELM is an open-source Python framework from Stanford CRFM; its documentation states that the project entered maintenance mode on June 1, 2026, which is worth knowing before you build a new pipeline on it. Inspect, maintained by the UK AI Security Institute and Meridian Labs, is the newest of the four and is oriented around agents and sandboxed tool use. OpenAI’s evals repository remains MIT-licensed, and its README now points users to configuring evals in the OpenAI Dashboard.

All four solve the same core problem. A benchmark is a dataset plus a protocol, and the protocol is where results diverge. If you have read our guide to LLM-as-judge evaluation pipelines, you have seen the downstream half of this story, where a model grades another model’s free-form output. This post covers the upstream half: the deterministic machinery that builds prompts, runs models, and scores outputs, and which therefore sets the ceiling on how much any number can be trusted. For the production side, where the same evaluation signals feed monitoring, see our piece on LLM observability and LLMOps architecture.

One framing helps throughout. Treat every evaluation run as an experiment, which is how Evan Miller’s paper “Adding Error Bars to Evals” (arXiv:2411.00640) puts it: questions are a sample from a larger population, so a score is an estimate with uncertainty. Most of the failure modes below reduce to forgetting that.

Anatomy of an LLM Evaluation Harness

An llm evaluation harness is a pipeline with six stages: a task registry that defines the dataset and protocol, a prompt builder that renders each sample, a request layer that classifies what must be asked of the model, an execution layer with caching, a post-processing layer that extracts an answer, and a metric and aggregation layer that produces a score with an uncertainty estimate. Differences in any stage change the number, which is why two harnesses rarely agree exactly.

Figure 1: Stages of a typical llm evaluation harness. Every box is a place where two harnesses can make different, individually defensible choices.

Figure 1 shows the stages left to right. The task registry holds a declarative description of the benchmark. The prompt builder turns a row of data into text. The request objects tell the backend what kind of scoring is needed. A cache layer keyed on the request avoids paying twice. The backend runs the model. Filters extract an answer from raw output. Metrics compare that answer with the gold label, and aggregation rolls per-sample scores into task, group, and suite results. Finally, the per-sample records are logged so a human can audit any number.

The task registry and prompt templating

In lm-evaluation-harness a task is a YAML file (or Python class) naming a dataset, a split, a prompt template, an output type, a metric list, and a few-shot configuration. Templates are Jinja2. A typical multiple-choice task defines doc_to_text (how to render the question), doc_to_choice (the candidate answers), and doc_to_target (the gold index). The declarative style is the point: a reviewer can read the YAML and know exactly what the model saw, and contributors can add a benchmark without touching the engine.

HELM takes a more object-oriented route. A scenario produces instances in a standard format; an adapter converts instances into a prompt (zero-shot, few-shot, with or without instructions); and a metric scores the completions. Inspect decomposes an evaluation into a Task made of a dataset, a solver, and a scorer, where a solver can be as simple as a single generate() call or as elaborate as a multi-turn agent using tools. The taxonomy differs, but each separates “what are the questions” from “how do we ask” from “how do we grade.”

Prompt templating is where most silent drift enters. Whether the template says Question: or Q:, whether the choices are labelled A. or (A), whether there is a trailing space after Answer: — each is a tokenization decision that moves probabilities. Sclar et al. (arXiv:2310.11324) reported performance differences of up to 76 accuracy points for LLaMA-2-13B from subtle formatting changes in few-shot prompts, and found that format rankings correlate only weakly between models. Their proposed fix, FormatSpread, reports an interval across plausible formats instead of a single number.

Few-shot construction

Few-shot examples are drawn from a pool, often the train or dev split, and concatenated before the test item. Three details matter. First, the sampling seed: if examples are drawn randomly per item, results vary run to run; if drawn once per task, all items share the same demonstrations and variance moves into that one draw. Second, leakage: the evaluated document must never be among its own demonstrations. The lm-evaluation-harness v0.4.13 release notes list a fix for exactly this bug, few-shot prompts that could include the evaluated document, which is a reminder that mature harnesses still ship such defects. Third, ordering: label-position bias in the demonstrations shifts predictions toward over-represented labels.

Log-likelihood versus generation scoring

This is the deepest architectural split. The lm-evaluation-harness documentation lists four output types: loglikelihood, loglikelihood_rolling, multiple_choice, and generate_until. Models that expose no logits or logprobs, such as most hosted chat APIs, support only generate_until.

In log-likelihood scoring, the harness never lets the model write an answer. For each candidate continuation it computes the model’s log-probability of that string given the prompt, then picks the highest. For MMLU-style tasks the candidates are either the letters A–D or the full answer texts. Cost is one forward pass per candidate (often batched with shared prefixes), there is no decoding, and results are deterministic. The weakness is that it measures preference among options, not whether the model can produce the answer, and it is sensitive to option length: raw log-likelihood favors short continuations, which is why harnesses also report length-normalized accuracy (acc_norm), dividing by character or token count.

In generation scoring, the model produces free text up to a stop condition, and a filter extracts the answer for exact match, F1, or a code-execution test. This matches how chat models are actually used and is the only option for reasoning traces. It is also stochastic unless greedy, slower, and exposes the extraction step to failure: if a model answers “The correct option is (C)” and your regex expects a bare “C”, the sample scores zero. Regex extraction bugs are one of the most common sources of unfair underscoring, and the same v0.4.13 notes mention a multiple-choice regex filter fix.

Property	Log-likelihood	Generation
What is scored	Probability of fixed candidates	Text the model writes
Needs logprobs	Yes	No
Deterministic	Yes	Only at temperature 0
Handles chain-of-thought	No	Yes
Typical failure	Length bias, option-format sensitivity	Answer-extraction mismatch, stop-sequence bugs
Cost per item	One pass per candidate	Full decode

Metrics, caching, and aggregation

Metrics are mostly simple: accuracy, normalized accuracy, exact match, F1, BLEU/ROUGE, pass@k for code, or a model-based grade. What differs is aggregation. Suite scores are often a macro-average of task scores, which weights a 100-item task the same as a 10,000-item one, or a micro-average over items, which lets large tasks dominate. Neither is wrong, but the choice must be stated, and it changes rankings. The v0.4.13 notes mention a fix to group standard errors, a reminder that pooled uncertainty is easy to get wrong.

Caching is the quiet engineering workhorse. The lm-evaluation-harness README documents --use_cache <DIR> to skip previously evaluated samples when resuming and --cache_requests to save dataset preprocessing; it also cautions that the sample cache is rank-dependent, so an interrupted multi-GPU run should restart with the same GPU count. A good cache key hashes the full rendered request plus model identity plus decoding parameters. A bad one keys on the task name, and returns stale scores after you edit a prompt.

Comparing lm-eval-harness, HELM, Inspect and OpenAI Evals

The four frameworks overlap but have different centers of gravity. lm-evaluation-harness is the workhorse for base-model and open-weights evaluation with logit access. HELM prioritizes multi-metric transparency and publishes leaderboards. Inspect is built for agentic and tool-using evaluations with sandboxing. OpenAI Evals centers on registry-defined, often model-graded evals tied to the OpenAI platform.

EleutherAI lm-evaluation-harness

The harness is the closest thing to a lingua franca for open-model evaluation. Its README lists backends for Hugging Face transformers (--model hf), vLLM, SGLang, NVIDIA NeMo and Megatron-LM, GPT-NeoX, ONNX Runtime, llama.cpp servers, and hosted APIs including OpenAI, Anthropic, TextSynth, and LiteLLM, plus local OpenAI-compatible servers. The release notes for v0.4.12 add TensorRT-LLM, Megatron-LM, Intel Gaudi, and LiteLLM backends along with tensor-parallel support for transformers models. Hundreds of tasks ship with it, and v0.4.13 adds benchmark suites such as LongProc, Uncheatable Eval, and LegalBench.

Its strength is breadth and scriptability, and it historically powered the Hugging Face Open LLM Leaderboard, which standardized on the harness so that submissions were scored with identical settings. Its limits are a design centered on completion-style models (chat-template handling has had to be bolted on and requires explicit care, covered below) and a task library whose quality varies by contributor. Always pin the version and the task version; tasks carry a version field that increments when the protocol changes.

Stanford HELM

HELM’s distinguishing idea is that accuracy alone is a poor summary. The original paper evaluated models across scenarios under multiple metrics: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. The helm-run, helm-summarize, and helm-server commands run benchmarks, aggregate results, and launch a local web UI where you can click through the exact prompts and completions behind any cell. That inspectability is HELM’s best feature. Its flagship leaderboards today include HELM Capabilities, HELM Safety, and VHELM for vision-language models, per its documentation.

The maintenance-mode status matters for planning. The project’s docs say it entered maintenance mode on June 1, 2026. That does not make existing results wrong, and the published prompts remain a useful reference, but a team starting a new internal pipeline should expect fewer new scenarios and slower model-provider updates.

UK AISI Inspect

Inspect’s documentation describes it as a framework for frontier AI evaluations covering coding, agentic tasks, reasoning, knowledge, behavior, and multimodal understanding. An evaluation is a Task with a Dataset (samples with input and target), a Solver that produces an answer, and a Scorer that grades it with text comparison, model grading, or custom logic. It supports sandboxed execution (Docker, Kubernetes, Modal, Proxmox, Vagrant, and others via an extension API) for running untrusted model-generated code, built-in tools (bash, python, text editing, web search, browsing, computer use), and MCP tools. The docs report more than 200 pre-built evaluations, over 20 model providers, and local inference via Hugging Face, vLLM, and SGLang.

For static multiple-choice benchmarks Inspect is perfectly capable, but its center of gravity is multi-turn agent evaluation where the unit of work is a trajectory, not a completion. Inspect View, its web log viewer, and a VS Code extension make the audit trail first-class.

OpenAI Evals

The openai/evals repository defines evals as YAML registry entries plus JSONL data stored with Git-LFS. Basic and model-graded templates need only data and YAML parameters, with no evaluation code. The README states that custom-code evals are no longer accepted, while model-graded evals with custom YAML remain welcome, and a banner announces that you can now configure and run Evals directly in the OpenAI Dashboard. Treat the open-source repo as a reference for the registry pattern and the hosted platform as the maintained path for OpenAI-model work.

Figure 2: The same model and benchmark under three scoring protocols. Scores are those reported in Hugging Face’s 2023 analysis of LLaMA-65B on MMLU.

Figure 2 previews the next section: the framework you pick is less important than knowing which protocol it implements.

Decision matrix

Need	lm-evaluation-harness	HELM	Inspect	OpenAI Evals
Base model, logprob scoring	Strong	Partial	Limited	Not designed for it
Multi-metric transparency (fairness, efficiency)	Basic	Strong	Via custom scorers	Basic
Agents, tools, sandboxes	Minimal	Limited	Strong	Limited
Leaderboard/reporting UI	Via add-ons	Built in	Inspect View	Dashboard
New-pipeline outlook in 2026	Active	Maintenance mode	Active	Platform-centric

Where I draw the line: use lm-evaluation-harness when you control the model and want log-likelihood or fast generation across many standard tasks; use Inspect when the thing under test acts, uses tools, or needs a sandbox; consult HELM for its published prompts and multi-metric results; use OpenAI Evals mainly when your stack already lives on that platform. The best practice, regardless of tool, is to run at least one standard benchmark through two independent harnesses to catch protocol drift before it reaches a decision.

Why the Same Model Scores Differently Across Harnesses

Hugging Face’s 2023 investigation of the Open LLM Leaderboard remains the cleanest worked example. It compared three MMLU implementations on LLaMA-65B: the original Berkeley implementation scored 0.636 by comparing the probabilities of the four answer letters; HELM scored 0.637 by having the model generate answer text; and the January 2023 version of the EleutherAI harness scored 0.488 by comparing log-likelihoods of full answer sequences, each letter followed by its answer text. Same weights, same questions, a spread of roughly 15 points. Hugging Face also noted that the prompts differed: the original opened with an instruction line naming the subject, HELM added a Question: prefix, and the harness version omitted the subject line and added a Choices header. Those numbers are from the Hugging Face write-up, and the harness has changed since, so do not read 0.488 as a statement about today’s code. The lesson is the mechanism, not the score.

Prompt format and tokenization

A language model sees tokens, not strings. A trailing space before the answer letter can change tokenization so that the scored candidate is A in one harness and A in another, two different tokens with different probabilities. Choice labels, separators, newline conventions, and the presence of a subject header all shift the conditional distribution. Base models are especially sensitive because they were never trained on your template. This is the reason the Sclar et al. result is so large: it is not a corner case but the default behavior of models on out-of-distribution formatting.

Chat templates

Instruction-tuned models were post-trained on a specific conversation format with role tokens and system prompts. Feeding them a raw completion-style prompt is a distribution shift, and feeding them a template with a different default system prompt is another. In lm-evaluation-harness the chat template is something you opt into at the command line; if you forget, you evaluate an instruct model as if it were a base model and typically understate it, while enabling the template changes both the prompt and, for few-shot, whether demonstrations become prior conversation turns. The README fragment I fetched for this post did not describe the flag, so check the current CLI help and docs for your pinned version rather than trusting a blog post, including this one. Where an API model is concerned, the provider applies its own template server-side and you have no control, which is one reason API and local scores for the “same” model can differ.

Stop sequences, max tokens, and extraction

For generation tasks, termination matters as much as the prompt. If the stop sequence is \n\n and the model begins its answer with a blank line, the output is empty. If it is missing and the model continues past the answer into an invented next question, the extractor may grab the wrong letter. A max_gen_toks budget that is too small truncates chain-of-thought mid-reasoning; reasoning models that spend thousands of tokens thinking are scored near zero under a 256-token limit. Then the extraction rule: first capital letter, last capital letter, regex on “answer is (X)”, strict versus flexible match. Each is defensible, and each moves accuracy by points on models whose style does not match the rule.

Few-shot count, sampling, and numerics

Five-shot versus zero-shot is the obvious knob, but the demonstration selection also counts, as discussed above. For sampled decoding, temperature and n change the estimate’s variance. Numerics enter too: batch size, padding side, dtype (bf16 versus fp16), and attention kernels can change logits at the third or fourth decimal, which flips a few borderline items. Individually trivial, these effects are real; you can only see them if you run the same configuration twice and diff the per-sample logs.

A practical consequence: a reported score without its configuration is not reproducible. At minimum record the harness name and commit or version, task name and task version, few-shot count and seed, chat-template flag, system prompt, stop sequences, max generation tokens, decoding parameters, model revision hash, dtype, and backend. If you cannot list those, you cannot say two scores are comparable.

Benchmark Contamination: Detection and Limits

Benchmark contamination means that test items, or close paraphrases of them, appeared in a model’s training data, so a high score may reflect memorization rather than capability. It is hard to prove for closed models because the training corpus is undisclosed, and hard to disprove even for open ones because web-scale deduplication is imperfect. Detection methods fall into four families, each with different assumptions about what you can access.

Figure 3: Four contamination checks and how their outputs combine into a decision to report a clean subset or flag and down-weight results.

N-gram overlap against the training corpus

The oldest method is direct string matching. The GPT-3 paper (Brown et al., 2020, arXiv:2005.14165) described a contamination analysis that searched for overlapping n-grams between benchmark examples and the training data and then reported results on the “clean” subset. The approach is precise when you have the corpus: slide an n-gram window (13 tokens is a common choice in the literature) across each test item and flag any hit. It needs corpus access, so it is available to model builders and to evaluators of fully open models whose data is published, and it misses paraphrased or translated leakage because it only matches surface strings.

Practical tuning: too short an n-gram flags common phrases (“which of the following is”) and wrongly condemns clean items; too long misses near-duplicates. Match on normalized text (lowercased, whitespace-collapsed) and require a high fraction of the item, not a single hit in a long passage.

Canary strings

Canaries are unique markers embedded in benchmark files to make leakage detectable. BIG-bench, for example, includes a canary GUID in its task files with the explicit request that the files not appear in training corpora. If a model can complete or reproduce the canary, the benchmark file was likely in its training data. The method is cheap and offers strong evidence when positive, but a negative result proves little: well-run data pipelines filter on the canary and strip it, so absence of the string says nothing about whether the questions themselves survived. Canaries protect against careless scraping, not adversarial or paraphrased inclusion.

Perplexity and membership inference

When you only have black-box probability access, you can ask whether a text looks “familiar.” Shi et al. (arXiv:2310.16789) proposed Min-K% Prob: take the k percent of tokens with the lowest probability in a passage and average their log-likelihoods; unseen text tends to contain a few unusually low-probability words, while seen text does not. They introduced the WikiMIA benchmark, using data created before and after a model’s training cutoff for ground truth, and reported a 7.4% improvement over prior methods on it. They also applied it to detecting contaminated benchmark examples. The caveat is that it is a statistical signal, not proof: it works better on long passages than on short multiple-choice questions, and the threshold needs calibration against known-clean data from the same domain.

A simpler cousin compares the model’s perplexity on the original benchmark ordering against shuffled or paraphrased variants. If the original ordering is markedly more probable than random permutations of the same items, the benchmark file’s sequence has probably been memorized.

Behavioral and dynamic approaches

Two practical defenses do not require any access to training data. The first is rephrase sensitivity: rewrite each question while preserving meaning, reorder the answer choices, or change numeric values in word problems, and compare accuracy. A large drop on perturbed variants with unchanged difficulty suggests the model learned the surface form. The second is dynamic or time-split benchmarks: build test items from material created after the model’s training cutoff, which is the logic behind WikiMIA and the reason “uncheatable” and live evaluations exist (v0.4.13 of lm-evaluation-harness adds one named Uncheatable Eval to its task list). Their limitation is the moving target: scores are not comparable across refreshes, and the pool of fresh, high-quality questions is finite.

What contamination does not tell you

Overlap is not binary. A model might have seen a test question in a tutorial with its answer, in a forum thread with a wrong answer, or only the topic. Removing flagged items changes the sample size, which widens confidence intervals, so a “clean subset” score must carry its own error bar. Moreover, deliberate benchmark optimization (training on a benchmark’s training split, or on synthetic data modeled on it) is not contamination in the leakage sense but produces the same inflated-by-familiarity effect. Treat any single benchmark number as an upper-bound signal, and prefer a basket of benchmarks with different construction dates and an internal, private holdout built from your own task distribution.

Statistical Rigor: Confidence Intervals, Paired Tests, and Seeds

Most leaderboard rankings quote scores to one decimal and ignore sampling error. Miller’s paper frames evals as experiments and recommends central-limit-theorem standard errors, clustered standard errors where questions share a passage or source, paired comparisons when two models answer the same questions, and power analysis when planning. These are the right defaults, and they take a few lines of code.

Worked numbers

The benchmark sizes below (14,042 MMLU test questions, 198 GPQA Diamond questions) come from the dataset papers and were not re-verified in this run, and the accuracy levels are illustrative, not reported results.

Take accuracy p on n independent questions. The standard error is sqrt(p(1-p)/n). On a large benchmark such as MMLU with 14,042 test questions, a model at p = 0.70 has SE = sqrt(0.21/14042) = 0.0039, a 95% interval of about plus or minus 0.76 points. That is tight, so differences of a point or more on the full test set are usually real, subject to protocol drift.

Now a small benchmark. GPQA Diamond has 198 questions. At p = 0.60, SE = sqrt(0.24/198) = 0.0348, so the 95% interval is about plus or minus 6.8 points. Two models at 60% and 64% on that set are statistically indistinguishable in isolation. Many headline claims about frontier models on small suites fall inside this noise.

Pairing helps a lot. Suppose models A and B answer the same 1,000 questions, A scores 0.72, B scores 0.70, and the disagreement table shows 90 questions only A got right and 70 only B got right. The mean paired difference is (90 – 70)/1000 = 0.02. The per-item variance of the paired difference is (90 + 70)/1000 – 0.02^2 = 0.1596, so SE = sqrt(0.1596/1000) = 0.0126 and z = 1.58, roughly a two-sided p of 0.11: not significant. The naive unpaired SE would be sqrt(0.720.28/1000 + 0.700.30/1000) = 0.0203, 61% larger, because it ignores that easy questions are easy for both models. Pairing removes shared question difficulty from the noise.

Power analysis flips the question: how many items do you need? To detect a 2-point difference with the disagreement rate above (0.16), with 80% power at the 5% level, n is about (1.96 + 0.84)^2 * 0.16 / 0.02^2 = 3,136 items. If your regression suite has 300 items, it can only reliably flag differences of roughly 6 points; smaller regressions will look like noise, and they will slip through.

Figure 4: Evaluation as CI. A smoke suite gates pull requests using a paired confidence interval, a nightly suite stores full results, and a frozen configuration gates releases.

Runnable bootstrap for the accuracy difference

This script takes per-item correctness vectors for two models on the same questions and returns a paired bootstrap confidence interval for the accuracy difference, plus a cluster-aware option for items that share a source passage. Run it on the per-sample logs your harness writes.

import numpy as np

def paired_bootstrap_diff(a, b, n_boot=10_000, alpha=0.05,
                          clusters=None, seed=0):
    """CI for mean(a) - mean(b) on the SAME items.

    a, b: 0/1 arrays of per-item correctness, aligned by item.
    clusters: optional array of group ids (items sharing a passage);
              if given, whole clusters are resampled together.
    """
    a = np.asarray(a, dtype=float)
    b = np.asarray(b, dtype=float)
    assert a.shape == b.shape, "per-item results must be aligned"
    d = a - b                         # paired per-item difference
    rng = np.random.default_rng(seed)

    if clusters is None:
        n = len(d)
        idx = rng.integers(0, n, size=(n_boot, n))
        boots = d[idx].mean(axis=1)
    else:
        clusters = np.asarray(clusters)
        ids = np.unique(clusters)
        per_cluster = [d[clusters == c] for c in ids]
        boots = np.empty(n_boot)
        for i in range(n_boot):
            pick = rng.integers(0, len(ids), size=len(ids))
            boots[i] = np.concatenate([per_cluster[j] for j in pick]).mean()

    lo, hi = np.percentile(boots, [100 * alpha / 2, 100 * (1 - alpha / 2)])
    return d.mean(), (lo, hi)

if __name__ == "__main__":
    # Reproduce the worked example: 1,000 items, 90 A-only, 70 B-only
    both, neither = 650, 190          # 650 both right, 190 both wrong
    a = np.array([1]*both + [1]*90 + [0]*70 + [0]*neither)
    b = np.array([1]*both + [0]*90 + [1]*70 + [0]*neither)
    diff, (lo, hi) = paired_bootstrap_diff(a, b)
    print(f"A - B = {diff:+.3f}, 95% CI [{lo:+.3f}, {hi:+.3f}]")
    # Expect roughly +0.020 with CI near [-0.005, +0.045]; CI includes 0

On the worked example this should return an interval of about plus or minus 0.025 around +0.020, consistent with the analytic SE of 0.0126. Because the bootstrap resamples items with replacement, it makes no normality assumption and extends naturally to F1 or pass@k if you substitute the per-item metric. For a paired p-value on the binary case, McNemar’s test on the 90 versus 70 discordant counts is the exact equivalent.

Seeds and variance sources

There are three independent noise sources, and a thorough report separates them. Question sampling noise is what the confidence interval above captures. Decoding noise arises from temperature above zero: rerun with several seeds and report the mean and the spread across seeds; Miller’s paper discusses reducing this by resampling answers per question. Prompt noise is the FormatSpread problem: run a few templates and report the range. Hiding any of the three behind one number overstates precision. A reasonable rule: if the difference you care about is smaller than the seed-to-seed spread, run more seeds before running more questions.

Putting Evals in CI

Evaluation belongs in the same pipeline as unit tests, with the same discipline: fast checks on every change, thorough checks on a schedule, and a frozen configuration for releases. The design in Figure 4 has four layers.

Smoke suite on pull requests. A few hundred items per critical capability, run with greedy decoding and a pinned harness version, finishing in minutes. The gate is a paired confidence interval on the difference against the main-branch baseline: fail only when the upper bound of the interval for the change falls below your allowed regression, not when the point estimate dips. This avoids both flaky failures and silent slides. Cache aggressively; with a request-hash cache, unchanged prompts against an unchanged model cost nothing.

Nightly full suite. Broader tasks, multiple seeds, and the prompt-format spread. Results are stored with the full configuration, git SHA, model revision, and per-sample logs, because a regression you cannot diff is a regression you cannot debug.

Contamination hygiene. Keep a private, versioned holdout built from your real traffic or domain, never published and never used for tuning. Rotate it. Periodically run a rephrase-sensitivity check on public benchmarks you quote externally.

Release gate. A frozen configuration file defines exactly which tasks, versions, and prompts count. The gate compares a candidate to the incumbent using the same paired procedure, and the report lists the interval, not just the means. Failures page a human; they do not auto-roll back, because one noisy night should not unship a model.

For agentic systems, sandboxing is a pipeline concern too. Inspect’s Docker and Kubernetes sandboxes let each sample run model-generated code in isolation, and its log viewer makes it practical to open a failing trajectory directly from a CI run.

Trade-offs, Gotchas, and What Goes Wrong

Harness pinning versus freshness. Pinning gives reproducibility but freezes bugs. The v0.4.13 notes for lm-evaluation-harness include fixes to few-shot leakage of the evaluated document, a multiple-choice regex filter, and group standard errors, so scores computed on older versions of those tasks may carry the defect. Record the version; when upgrading, rerun a baseline model and publish the delta as a harness change, not a model change.

Log-likelihood on chat models. Chat-tuned models can behave oddly under raw log-likelihood scoring because the post-training distribution is not a language-model prior over arbitrary continuations. Use generation scoring for instruct models whenever the task allows, and compare with the base model’s log-likelihood score only knowing the protocols differ.

Saturated and noisy benchmarks. When top models cluster within a few points, ranking is mostly noise plus formatting. A benchmark with labeling errors puts a ceiling on the score and rewards matching the mistakes. Audit a sample of “failures” by hand; you will often find the gold label is wrong.

Judge-based metrics inherit judge bias. If a model grades free-form outputs, you add position bias, verbosity bias, and self-preference; see our LLM-as-judge pipeline guide for controls. A harness that reports a model-graded score without the judge model, prompt, and agreement rate is not reporting a measurement.

Multiple comparisons. Run 50 tasks and compare two models and some will differ at p < 0.05 by chance. Pre-register the primary metric, or correct for multiplicity, or at least report how many comparisons were made.

Metric-gaming and overfitting to the harness. Teams that tune prompts on the evaluation set, or select the best of many format variants, produce numbers that do not transfer. Choose the template once, on a development split, then freeze it.

Cost and latency blind spots. Accuracy per dollar and tail latency are often the deciding factors in production and are absent from a pure accuracy harness. HELM’s efficiency metric is an example of recording them alongside quality.

Practical Recommendations

Start by deciding what decision the number will inform, because that determines the rigor needed. A research claim that model A beats model B needs paired intervals, multiple seeds, and contamination checks. A CI regression alarm needs a stable protocol and enough power to catch the regressions you care about. A procurement shortlist needs your own holdout more than any public leaderboard.

Pick one primary harness and pin it, then validate it against a second independent implementation on at least one benchmark to catch protocol drift. Prefer generation scoring for chat models and log-likelihood for base-model research, and say which one you used. Store per-sample logs for every run, since every investigation of a surprising number starts there. Quote intervals, not bare means, and compare models with paired methods.

Treat HELM’s maintenance-mode status as a planning input: reuse its scenarios and published prompts as references, but anchor new infrastructure on actively developed tooling. Choose Inspect when an agent or sandbox is involved.

Checklist before you publish or gate on an eval number:

Harness name, version or commit, task version, and model revision recorded
Prompt template, few-shot count and seed, chat-template and system-prompt settings recorded
Stop sequences, max generation tokens, and extraction rule recorded
Per-sample outputs saved and a sample of failures audited by hand
95% confidence interval reported, paired when comparing models
Power checked: is n large enough for the difference you claim?
Contamination considered: overlap check, canary probe, or rephrase test
A private holdout from your real task distribution included
Results reproduced on a second harness for at least one benchmark

Frequently Asked Questions

What is an LLM evaluation harness?

An llm evaluation harness is software that runs a model through a benchmark in a standardized way. It loads the dataset, renders prompts with few-shot examples, sends requests to the model, extracts answers from outputs, computes metrics such as accuracy or exact match, and aggregates results with uncertainty estimates. Examples include EleutherAI’s lm-evaluation-harness, Stanford HELM, UK AISI’s Inspect, and OpenAI Evals. Its protocol choices strongly affect the final score.

Why do different harnesses give different scores for the same model?

Because a score depends on the protocol, not just the weights. Prompt wording, answer-choice formatting, few-shot examples, chat-template use, stop sequences, token limits, answer-extraction rules, and whether scoring uses log-likelihood or generation all differ. Hugging Face found that LLaMA-65B scored 0.636, 0.637, and 0.488 on MMLU under three implementations. Compare scores only when the full configuration matches, or report your own harness results for every model.

How do I detect benchmark contamination?

Combine methods. If you have the training corpus, run n-gram overlap checks and report results on the clean subset. Probe for canary strings such as BIG-bench’s GUID. With black-box access, try Min-K% Prob membership inference or compare accuracy on rephrased and reordered variants. Use time-split benchmarks built after the training cutoff. No single test is conclusive, so treat results as evidence and keep a private holdout for decisions.

Which should I use: lm-evaluation-harness, HELM, or Inspect?

Use lm-evaluation-harness for broad, fast evaluation of base and open-weights models with logit access. Use Inspect when you evaluate agents, tool use, or code that needs sandboxing. HELM is valuable for its multi-metric design and inspectable prompts, though its docs state it entered maintenance mode on June 1, 2026. Many teams use two frameworks and cross-check one benchmark to catch protocol drift.

How many benchmark questions do I need to compare two models?

It depends on the effect size and how often the models disagree. With paired comparison and a 16% disagreement rate, detecting a 2-point accuracy difference at 80% power and 5% significance needs roughly 3,100 items. A 300-item suite only reliably detects differences around 6 points. Small benchmarks such as the 198-question GPQA Diamond have 95% intervals near plus or minus 7 points at 60% accuracy.

Should I put evals in CI?

Yes, with tiers. Run a small, fast, deterministic smoke suite on pull requests and gate on a paired confidence interval versus the baseline, not the raw difference. Run the full suite nightly with several seeds and store per-sample logs. Freeze a configuration for release gates. Pin the harness version so a tooling update is not mistaken for a model regression.