Agent Benchmarks in 2026: SWE-bench Verified, GAIA, and tau-bench

A model that scores 95% on one leaderboard and 45% on another has not gotten stupider between the two — the benchmarks are measuring different things, under different rules, with different amounts of scaffolding wrapped around the same weights. AI agent benchmarks are the single most cited and most misread numbers in the field. A headline like “our agent hits 92% on SWE-bench” tells you almost nothing until you know which split it ran, which harness applied the patch, how many steps and tool calls it was allowed, and whether the number reflects the raw model or an elaborate agentic wrapper doing most of the work.

This post takes the three benchmarks that actually drive purchasing and research decisions in 2026 — SWE-bench Verified, GAIA, and tau-bench — and dissects what each one measures, how grading works down to the string-match level, where contamination and leakage creep in, and why two leaderboard rows are rarely comparable. Then it shows how to run an honest internal evaluation of your own agent.

What this covers: the grading mechanics of each benchmark, the scaffold-versus-model confound, contamination and repo-state leakage, the pass^k reliability metric, a results comparison table with real 2026 numbers, and a practical recipe for an internal agent eval you can trust.

Context and Background

Benchmarks for language models used to be static question-answer sets: MMLU, GPQA, HellaSwag. You feed the model a prompt, read one token or one span back, and grade it. That worked while models were essentially text completers. It stopped being sufficient the moment “agents” — systems that plan, call tools, browse, execute code, and take multi-turn actions against a live environment — became the product being sold.

An agent benchmark has to grade behavior over a trajectory, not a single answer. That introduces a whole new surface of confounds. The same base model produces wildly different scores depending on the scaffold (the planning loop, retry logic, tool wiring), the step budget, and how the environment is reset between runs. The evaluation harness is now part of the system under test, and that is precisely where most misreadings originate.

The three benchmarks here were chosen because they anchor three distinct capability axes. SWE-bench Verified measures software engineering: can the agent fix a real bug so the repository’s tests pass? GAIA measures general assistant competence: can it answer real-world questions that need browsing, tool use, and multimodal reading? tau-bench measures reliable tool-agent-user interaction: can it follow a policy while transacting against a database over a multi-turn conversation, repeatably? For the broader discipline of scoring model outputs, see our guide to LLM evaluation and LLM-as-judge. The original GAIA design goals are laid out in the GAIA paper on arXiv.

How SWE-bench Verified Actually Grades a Patch

SWE-bench Verified is an execution benchmark. It does not ask a model to describe a fix — it hands the agent a real GitHub issue plus a snapshot of the repository at the commit before the fix landed, and it accepts an answer only if the agent’s patch applies cleanly and makes the project’s test suite pass. The metric is binary per task and averaged across the set: resolved or not resolved.

Figure 1: The SWE-bench Verified resolution pipeline — issue plus repo state in, candidate patch out, then a two-gate check on patch application and test execution.

Figure 1 traces the flow. The agent reads the issue and the code, emits a diff, and the harness applies it. If the patch fails to apply, the task is an automatic fail. If it applies, the harness runs two test groups: the FAIL_TO_PASS tests (which were failing before the fix and must now pass) and the PASS_TO_PASS tests (which passed before and must still pass, guarding against regressions). Only when every test in both groups is green does the task count as resolved.

The “Verified” split exists because the original was noisy

Plain SWE-bench, released in 2023, drew from 2,294 issues across 12 popular Python repositories. Practitioners quickly found that many tasks were underspecified or had broken or flaky tests, so a “resolved” result did not cleanly mean “solved the bug.” In 2024 OpenAI, working with the SWE-bench authors, released SWE-bench Verified: a 500-task subset that human software engineers manually validated as well-specified and correctly testable. That is the split almost every 2026 leaderboard reports. When someone says “SWE-bench,” in a coding context they almost always mean Verified.

Scaffold versus model is the confound that ruins comparisons

Here is the trap. A SWE-bench score is a property of agent plus harness plus model, not of the model alone. The scaffold decides how the issue and code are presented, how many turns the agent gets, whether it can run tests iteratively before submitting, how retrieval over the repository works, and how failures are retried. Swap a bare model into a stronger scaffold and the number jumps double digits without touching the weights.

This is why the official SWE-bench leaderboards separate submissions and require harness disclosure, and why a June 2026 analysis by DigitalApplied argued that most of the recent gains reflect scaffolding improvements rather than raw model capability. When you read “88.6%” versus “95.5%” on two rows, the delta may be almost entirely scaffold. Numbers are only comparable within an identical harness.

To make the confound concrete: consider two submissions of the same model. Submission A wraps it in a scaffold that retrieves the ten most relevant files via embedding search, lets the agent run the failing tests up to five times and edit between runs, and caps the run at 40 tool calls. Submission B gives the model the raw issue and the file tree, one shot, no test execution before submit. Same weights, and A can post a resolution rate 15–25 points higher than B. Neither number is “the model’s SWE-bench score” — there is no such thing. The score is a joint property of the whole pipeline, and the only defensible comparison holds every part of that pipeline fixed while varying one factor.

Contamination and repo-state leakage

Because the tasks come from public GitHub history, the fixes are in the training data of any model trained on a recent web crawl. In early 2026 OpenAI’s Frontier Evals team stopped reporting SWE-bench Verified after an internal audit found that frontier models could reproduce gold-patch solutions from just the task ID — a fingerprint of contamination — and that a meaningful fraction of tasks had flawed tests. Reporting from CodeSOTA on the SWE-bench contamination debate put roughly 60% of one problematic-task sample as unsolvable-as-written.

There is a second, subtler leak: repository-state loopholes. If the harness does not scrub the future state of the repo, an agent can read the fix directly out of git metadata — later commits, branch names that describe the fix, tags, or the reflog with future commit messages. The SWE-bench maintainers tracked this under a public issue on repo-state loopholes during agentic evaluation and the mitigation is to strip reflogs, branches, origins, and tags and to hide test.patch artifacts so the agent cannot simply copy the expected test changes. If your internal harness reuses SWE-bench tooling, verify these scrubs are on — otherwise your numbers are inflated by leakage you did not intend.

GAIA and tau-bench: Assistants and Reliable Transactions

SWE-bench lives entirely inside a code repository. The other two benchmarks push into messier territory: open-ended assistant tasks and policy-bound customer transactions. Their grading philosophies differ sharply, and understanding both is what lets you read a general-agent leaderboard without being fooled.

GAIA grades a single string, but the work is everything before it

GAIA — “General AI Assistants” — is a set of 466 real-world questions designed so that a human finds them conceptually simple but a system must chain reasoning, web browsing, tool use, and multimodal reading to answer. Questions are split into three difficulty levels by the number and variety of steps required: Level 1 is short tool chains, Level 3 demands long multi-step plans with several tools and modalities. The authors keep 300 test answers private and release a 166-question development split with ground truth.

Figure 2: A GAIA trajectory — plan, fan out across browsing, code, and multimodal tools, synthesize, then normalize and quasi-exact-match against a short gold answer.

Figure 2 shows why GAIA is deceptively hard. The grading itself is trivial: the answer is a short string — a number, a name, a date — checked by quasi-exact match after light normalization for case, units, and formatting. There is no partial credit. Either the final string matches the gold answer or the task is wrong. All the difficulty is in the trajectory: the model has to actually browse the live web, read a spreadsheet or image, run a calculation, and land on the exact token the graders expect. A single wrong hop anywhere in the chain produces a clean, unforgiving failure.

That exact-match design is a feature — it makes grading automated, fast, and factual, with no LLM judge in the loop and no rubric drift. But it also means GAIA rewards agents tuned to emit answers in the graders’ expected format, and it penalizes correct reasoning that phrases the answer differently. When you see a top GAIA number, remember two things. First, top entries are usually orchestrated multi-tool agents or ensembles, not a single model call, so the score belongs to a system. Second, “bare model” GAIA scores (no tools) are far lower — frontier models sit in roughly the 40–50% range without scaffolding, while heavily scaffolded systems on the community leaderboards report figures north of 90%.

The difficulty gradient is worth dwelling on because it explains the shape of most GAIA leaderboards. Level 1 questions typically need a short chain — one or two tool calls, maybe a single web lookup — and strong agents clear them at high rates. Level 2 adds branching: several tools, some intermediate computation, a modality switch. Level 3 is where systems separate, because a long plan multiplies the chance of a single fatal misstep. Since grading is all-or-nothing per question, an agent that is 90% reliable per hop but needs eight hops lands near 0.9 to the eighth power — around 43% — on the hardest tier. That compounding is the real adversary in GAIA, not any single hard question, and it is why browsing quality and error recovery matter more than raw model IQ.

tau-bench measures whether the agent behaves the same way twice

tau-bench (τ-bench), from Sierra, targets a capability the other two ignore: reliable, policy-compliant interaction with a real user over multiple turns. It simulates a customer-service agent in two domains — retail and airline — each with a realistic database, a set of domain API tools, and a written policy document the agent must obey. A language model plays the user; the agent under test must gather intent, respect the policy, and execute the right tool calls.

Figure 3: A tau-bench episode — the simulated user drives a conversation, the agent calls domain tools that mutate a database, and the grader compares the final database state to an annotated goal state.

Figure 3 shows the loop. Crucially, tau-bench does not grade the conversation transcript. It grades the final world state: after the episode, the harness compares the database plus any required outputs against an annotated goal state. Did the reservation actually get cancelled? Did the refund land with the right amount, and only if policy allowed it? This state-based grading is why tau-bench is a good proxy for production agents that take real actions — you cannot bluff your way to a pass with a confident-sounding but wrong transcript. The benchmark’s design is described in Sierra’s write-up on tau-bench and agent evaluation and the τ-bench paper.

pass^k: the metric that exposes flaky agents

tau-bench’s most important contribution is the pass^k metric. Standard code benchmarks report pass@k — the probability that at least one of k attempts succeeds — which flatters agents by counting a lucky run. tau-bench inverts it: pass^k is the probability that all k independent attempts succeed on the same task, averaged across tasks. Because independent runs are roughly i.i.d., pass^k decays like p^k. An agent with 85% pass^1 might drop toward the low-to-mid 60s at pass^4, because a one-in-seven per-attempt failure rate compounds fast across four tries.

That is the number that matters for production. A customer-service agent that resolves a ticket 85% of the time still mishandles roughly one in seven identical customers — unacceptable for anything touching money or bookings. pass^k makes reliability, not peak capability, the headline. It is the honest counter to a leaderboard culture that celebrates best-of-N. For a fuller argument on why headline scores mislead and reliability should lead, the community guide on reading agent leaderboards honestly is a good companion.

Policy adherence is the other axis tau-bench forces into the open, and it is where many agents quietly fail. Each domain ships a written policy — the airline agent, for example, may only waive a change fee under specific documented conditions, and must refuse otherwise. An agent that helpfully grants the waiver to satisfy the simulated user has produced a wrong final state even though the customer would be delighted. tau-bench grades that as a failure because the database now diverges from the policy-correct gold state. This penalizes sycophancy directly, which is exactly right for production: an over-accommodating agent that bends the rules is a compliance and revenue risk, not a feature. The task-fix work in the τ³ update further tightened these domains after the community found ambiguous tasks where the “correct” behavior was itself under-specified — a reminder that even reliability benchmarks need their own verification passes before their numbers can be trusted.

A 2026 Results Comparison — and Why the Numbers Fight Each Other

Here is a snapshot of representative 2026 leaderboard figures. Read the caveats column before the score column — that is the whole point of this section.

Benchmark	What it measures	Grading	Representative 2026 top score	Critical caveat
SWE-bench Verified	Fix a real GitHub bug so tests pass	Binary per task: patch applies AND FAIL_TO_PASS + PASS_TO_PASS green	Frontier agents reported ~88–95% on community trackers	Score = model + scaffold + harness; contamination and repo-state leakage inflate; only comparable within one harness
GAIA	Real-world assistant tasks needing browsing, tools, multimodality	Quasi-exact string match, no partial credit	Scaffolded systems reported >90%; bare frontier models ~40–50%	Top entries are orchestrated agents or ensembles, not one model; format-sensitive
tau-bench (retail/airline)	Policy-bound, multi-turn tool-agent-user transactions	Final database/world state vs annotated goal	Leading agents ~85–89% pass^1 on public snapshots	pass^1 hides reliability; pass^k drops sharply; policy adherence, not peak skill, is the real test

The exact numbers move week to week and differ by tracker, so treat these as ranges, not verdicts. Community leaderboards such as BenchLM.ai’s SWE-bench Verified table and tau-bench table, and the official SWE-bench site, report the live figures; Epoch AI maintains an independent SWE-bench Verified tracker with methodology notes.

Figure 4: An illustrative comparison of score bands across the three benchmarks. Exact values shift by tracker and harness — the shape of the story (scaffold gap on GAIA, reliability cliff on tau-bench pass^k) is the durable part.

Figure 4 sketches the score bands rather than exact points, because exact points are the least stable thing here. Three structural reasons make cross-row comparison unsafe, and they are worth internalizing.

First, harness and tool access differ. A SWE-bench agent that can run the test suite iteratively before submitting will beat one that submits blind, same model. A GAIA agent with a good browser and a code interpreter beats one with search-only. The tool stack is part of the score.

Second, the model-versus-agent boundary is blurry. GAIA’s leading entries are ensembles that route between models and specialized sub-agents. Attributing a 90%+ GAIA score to a single base model is simply wrong — the orchestration did much of the work. SWE-bench has the same problem in milder form.

Third, step and cost budgets are rarely held constant. An agent allowed 50 tool calls and unlimited retries will out-score the same agent capped at 10. If a leaderboard does not publish max-steps, cost, and latency alongside the accuracy, the accuracy is only half a result. The Holistic Agent Leaderboard work on arXiv argues exactly this: agent evaluation needs cost and reliability reported next to raw accuracy, or the ranking is misleading.

Trade-offs, Gotchas, and What Goes Wrong

The failure modes cluster into four families, and every one of them will bite an internal eval that is not defended against it.

Contamination is the default, not the exception. Any benchmark drawn from public data is in the training corpus. SWE-bench’s gold patches, GAIA’s question style, tau-bench’s public tasks — all potentially memorized. A high score can reflect recall rather than reasoning. Reported figures of solution leakage on successful SWE-bench passes have run in the 30%+ range once you account for the model copying fix text out of issue comments. The defense is held-out, private tasks you author yourself, refreshed over time.

Grading brittleness cuts both ways. GAIA’s exact match fails correct-but-differently-phrased answers; that is a false negative. LLM-as-judge grading, used in many custom evals, produces false positives — a plausible wrong answer that the judge model rates as correct, plus position and verbosity biases where the judge favors longer or first-listed responses. If you lean on an LLM judge, calibrate it against human labels and hold out a judged-versus-human agreement number. Our LLM-as-judge deep dive catalogues these biases in detail.

Reliability hides behind pass@1. A single-run number is the most flattering and least useful statistic. Production behavior is a pass^k story. If you only ever report pass@1, you are advertising your agent’s best day, not its typical one.

Security and policy adherence are usually unmeasured. None of these three benchmarks is a security test. An agent can ace tau-bench’s happy-path tasks and still be trivially manipulable by an adversarial user or a poisoned tool output. Benchmark accuracy says nothing about resistance to prompt injection — for that, treat evaluation and red-teaming as separate tracks, as covered in our piece on agentic AI security and prompt injection. A benchmark-topping agent with an open injection surface is a liability, not an asset.

The meta-gotcha: optimizing to a public benchmark is Goodhart’s law in action. Once a number becomes a target, teams tune scaffolds to that number, and the metric stops measuring the underlying capability. That is why the serious labs rotate to fresh, private suites — and why your internal eval should never be a mirror of the public leaderboard.

Practical Recommendations

An honest internal agent eval is not hard, but it requires discipline the public leaderboards do not enforce on you. Build it around held-out tasks you own, fixed seeds, reliability metrics, and a cost budget reported next to every accuracy number.

Start by authoring your own tasks that resemble your production workload, not the benchmark. Keep them private and refresh a slice each quarter to blunt contamination. Grade on end state where you can — the tau-bench philosophy of checking the final world state beats grading a transcript. Where you must use an LLM judge, pin the judge model and prompt, and validate it against a human-labeled sample before you trust it. Always report reliability as pass^k, not just pass@1, and publish the step budget, tool set, cost per task, and latency alongside accuracy so a future reader can reproduce and compare.

Checklist for a trustworthy internal eval:

• Held-out, self-authored tasks that mirror your real workload; refresh a slice each quarter.
• Fixed seeds and a pinned harness (model version, scaffold, tool set, max steps) recorded with every run.
• State-based grading where feasible; a calibrated LLM judge only where it is not, validated against human labels.
• pass^k, not pass@1 — report k≥4 so the reliability cliff is visible.
• Cost and latency budgeted and printed next to accuracy; an agent that is 2 points better at 5x the cost is rarely a win.
• A separate security/red-team track for prompt injection and adversarial tool output — accuracy is not safety.

Frequently Asked Questions

What is the difference between SWE-bench and SWE-bench Verified?

Plain SWE-bench is the original 2,294-issue set drawn from 12 Python repositories, and it contained underspecified or flaky-test tasks that made “resolved” ambiguous. SWE-bench Verified is a 500-task subset that human software engineers manually confirmed as well-specified and correctly testable. Verified is the split almost every 2026 leaderboard reports, so when a vendor cites a “SWE-bench” coding score, they nearly always mean Verified. The grading is identical — patch must apply and all designated tests must pass.

Why do the same models score so differently across leaderboards?

Because an agent score is a property of the model plus its scaffold, harness, tool access, and step budget — not the model alone. A stronger scaffold, iterative test running, a better browser, or a larger max-steps cap can move a score by double digits without changing the weights. GAIA’s top entries are also multi-model ensembles, so their scores belong to a system, not one model. Numbers are only comparable within an identical harness, which is why cross-leaderboard comparison is usually invalid.

What does pass^k measure and why does it matter?

pass^k is the probability that all k independent attempts on the same task succeed, averaged across tasks — the reliability inverse of pass@k, which counts a single lucky success. Because runs are roughly independent, pass^k decays like p^k, so an 85% pass^1 agent can fall well below 70% at pass^4. It matters because production agents face the same task repeatedly; a one-in-seven per-attempt failure rate is unacceptable for transactions touching money, bookings, or safety. pass^k surfaces that fragility that a sing