Agent Evaluation: Trajectory Evals, SWE-bench, tau-bench and pass^k (2026)

An agent that solves a task once is a demo. An agent that solves it every time is a product. Most teams discover the gap in production: the support bot that passed your ten hand-picked test conversations quietly fails one customer in five, and nobody can tell you which step went wrong because the only thing you recorded was the final message.

Agent evaluation is harder than evaluating a chat model because the object under test is not an answer, it is a multi-step process that calls tools, mutates state, talks to a user and may take a different path on every run. A single pass/fail score hides variance, hides unsafe shortcuts and hides the cost of getting there. Benchmarks such as SWE-bench and tau-bench exist because the field learned this the hard way, and the reliability metric pass^k exists because averages lie.

This post gives you a working mental model and a harness. You will leave knowing which metrics measure what, how pass@k and pass^k diverge with worked numbers, how the major benchmarks grade, where they break, and how to build a regression suite you can run on every prompt or model change.

What this covers: why agent evals differ from single-turn evals, outcome versus trajectory metrics, the benchmark landscape, pass@k versus pass^k math, environments and user simulators, grading with state diffs versus LLM judges, cost, a runnable Python harness, and failure modes.

Context and Background

Single-turn evaluation has a comfortable shape: a fixed prompt goes in, a string comes out, and a reference answer, a metric or a judge model scores it. Multiple-choice suites and reference-based metrics grew up in that world. Even LLM-as-a-judge evaluation mostly assumes you are scoring one output against a rubric.

Agents break every part of that shape. The input is a goal, not a prompt. The output is a sequence of decisions: which tool to call, with which arguments, in which order, and when to stop and ask the human. The environment changes as a result of those decisions, so the same agent can face different states on different runs. And the model itself is stochastic, so run-to-run variance is a first-class property rather than noise to be averaged away.

Three consequences follow. First, you must define success in terms of the world after the run, such as a database row, a repository test result or a file on disk, because the transcript can claim success that never happened. Second, you must measure consistency, because a customer-facing agent that works 80% of the time fails constantly at scale. Third, you must observe the path, because two agents with the same final state can differ wildly in cost, risk and fragility.

The research community converged on this over 2023 to 2025. SWE-bench (Jimenez et al., 2023) grounded coding agents in real GitHub issues graded by unit tests. WebArena and OSWorld grounded web and desktop agents in executable environments. GAIA tested general assistants with questions that have unambiguous short answers. tau-bench, from Sierra, added simulated users and a reliability metric. Each fixed something the previous generation missed, and each has since developed its own failure modes, which we cover below.

Operationally, evaluation is also tied to your observability stack. Traces captured with OpenTelemetry GenAI semantic conventions are the raw material for trajectory evals, and the governance questions around long-running governed AI agents are exactly the ones an eval suite should encode as tests.

A Reference Architecture for Agent Evaluation

Direct answer: agent evaluation runs an agent inside a reproducible environment, records the full trajectory, and then scores two things separately: the outcome (did the world end up in the correct state) and the process (were the steps valid, safe and efficient). Repeating each task k times turns a single score into a reliability estimate such as pass^k.

Figure 1: Agent evaluation architecture. A task set drives the agent inside a sandbox, the trace store captures every step, and separate graders score outcome, trajectory and cost.

Figure 1 shows the pieces. A task specification carries the goal, any user persona, the initial environment state and the expected final state or a checker. A sandbox provides tools and mutable state, ideally resettable to a snapshot. The agent under test is a black box: a model, a prompt, a tool set and an orchestration loop. Every model call and tool call is written to a trace store. Graders then read the trace and the final state, never the agent’s own claims.

Outcome metrics versus trajectory metrics

Outcome metrics ask whether the goal was achieved. They are cheap to explain and hard to game when computed from environment state. Typical forms: unit tests pass (SWE-bench), database state matches the expected state (tau-bench), the file contains the expected content (OSWorld-style checkers), or a short answer matches exactly (GAIA).

Trajectory metrics ask how the agent got there. Common ones are tool-call validity (schema-correct arguments), tool selection accuracy against a reference path, redundant call rate, step count, retries after error, policy violations, and whether the agent asked for confirmation before an irreversible action. These matter because outcome-only scoring rewards lucky paths. An agent that deletes and recreates a record may reach the right final state while violating a rule you care about.

The right stance is not to pick one. Use outcome as the gate and trajectory as the diagnostic. When outcome fails, trajectory tells you where. When outcome passes but trajectory is poor, you have found a latent regression before it becomes an outcome failure.

Why reference trajectories are a trap

It is tempting to define a golden sequence of tool calls and score similarity. Resist it for open-ended tasks. Many tasks admit several valid orderings, and a strict match punishes correct creativity while rewarding memorized scripts. Prefer invariants: “the refund tool is never called before the order lookup”, “no write call occurs before the user confirms”, “total calls under 12”. Invariants are checkable on any trajectory and survive prompt changes. Keep golden paths only for narrow, deterministic workflows where order truly matters.

Components and ownership

A practical split of responsibilities keeps the suite maintainable. The task authors own specifications and checkers. The platform team owns the sandbox images, determinism controls and the trace schema. The agent team owns the agent and reads the reports. When one person owns all three, the checker slowly drifts toward whatever the agent already does, which is the quiet way a regression suite stops catching regressions.

The Benchmark Landscape: What Each One Actually Measures

Public benchmarks are useful for two things: calibrating your intuition about what is hard, and giving you a template for building your own. They are poor substitutes for a private suite on your tasks. Here is what each major benchmark grades and the caveats you should carry with it. Check each project’s leaderboard directly before quoting scores, because they move monthly.

Figure 2: Agent benchmarks grouped by how they decide success. Executable checks dominate, which is why environment fidelity matters more than the judge.

SWE-bench and SWE-bench Verified

SWE-bench turns real GitHub issues from popular Python repositories into tasks. The agent receives the repository at a commit and the issue text, and must produce a patch. Grading applies the patch and runs the project’s tests: “fail-to-pass” tests that the fix should make pass, and “pass-to-pass” tests that must not break. That is a clean, execution-based outcome metric.

SWE-bench Verified is a 500-task subset that human annotators screened for well-specified issues and valid tests, released with OpenAI’s help in 2024. It became the default headline number for coding agents. In February 2026, OpenAI announced it had stopped reporting SWE-bench Verified scores. Its audit of 138 problems that its o3 model did not consistently solve found that 59.4% had material test or description problems, including tests so narrow that they rejected functionally correct fixes. OpenAI also reported that every frontier model it tested could reproduce the original human fix or verbatim problem details for some tasks, which points to training-data contamination. It recommended reporting the public split of SWE-bench Pro instead, while noting that no public benchmark is immune.

The lesson is general. A benchmark with a fixed public task set decays: tasks leak into training data, flawed tests get discovered, and the score saturates into a measure of exposure rather than capability. Treat any single public number as a noisy upper bound.

tau-bench and tau2-bench

tau-bench (written τ-bench), from Sierra’s research team, evaluates agents in customer-service style domains, originally retail and airline. Each episode has three parties: an LLM-simulated user with a hidden goal, an agent with a policy document and API tools, and a database. The agent converses, calls tools and may change records. Grading compares the final database state against the expected state, so neither a human nor a judge model has to read the conversation.

Its signature contribution is pass^k. Sierra reported that the 12 agents it tested, built with function calling or ReAct on proprietary and open models, struggled, with the best (a GPT-4o agent) averaging under 50% pass^1 across the two domains. On retail, that GPT-4o agent fell to roughly 25% at pass^8, a drop of about 60% from its pass^1 score. Those numbers are from the original 2024 release; newer models score much higher, so use them as an illustration of the pass^k effect, not as a current ranking.

tau2-bench (Barres et al., 2025) extends this with a dual-control setting in which the user can also act on the shared environment, with a telecom domain added, so the agent must guide a user through steps rather than only act itself. Verify the current domain list and task counts in the repository before citing them.

OSWorld, WebArena and GAIA

OSWorld evaluates multimodal agents on real computer environments. The original paper contains 369 tasks on Ubuntu, with a further 43 adapted Windows tasks for supplementary analysis. Grading is execution-based: each task has a custom script that inspects the final environment state. At publication humans achieved about 72.4% while the best model reached 12.24%, a gap that has narrowed considerably since; check the live leaderboard.

WebArena places agents in self-hosted websites (shopping, forums, code hosting, content management) and checks outcomes programmatically, such as page content or database state. Its value is a reproducible web you can reset, which avoids the drift of the live internet. GAIA is different in kind: questions require browsing, tools and reasoning, but the answers are short and unambiguous so scoring is an exact-style match. The GAIA authors reported a large human versus GPT-4-with-plugins gap at launch (human accuracy in the low 90s percent versus roughly 15% for the model, from my reading of the paper; I could not re-verify those figures this run). Short-answer grading is cheap and robust but says nothing about side effects.

Choosing benchmarks for your agent

Match the benchmark to the agent’s risk profile. A coding agent should be sanity-checked on SWE-bench-style tasks but primarily judged on your own repositories, with your own tests. A transactional agent that edits records should be modelled on tau-bench: simulated users, a policy document and database-state grading. A browser or desktop agent needs an OSWorld or WebArena style resettable environment. In every case the transferable idea is the same: executable checks on environment state beat reading transcripts.

pass@k Versus pass^k: The Reliability Math

Both metrics repeat a task k times. They answer opposite questions.

pass@k asks: is at least one of k attempts correct? It measures capability under retry, and it matches workflows where a human or a verifier picks the winner, such as code generation with unit tests as a filter.

pass^k asks: are all k attempts correct? It measures reliability, and it matches workflows where every run is delivered to a user with no safety net, such as a support agent handling refunds.

If a task has a per-attempt success probability p and attempts are independent:

pass@k = 1 − (1 − p)^k
pass^k = p^k

With p = 0.8, pass@1 and pass^1 are both 0.80. At k = 8, pass@8 is 1 − 0.2^8, about 0.99999, while pass^8 is 0.8^8, about 0.168. The same agent looks nearly perfect or badly unreliable depending on which metric you choose to print. With p = 0.9, pass^8 is about 0.43. With p = 0.95, pass^8 is about 0.66. These are straightforward arithmetic illustrations, not measured results.

Figure 3: From n trials per task to unbiased pass@k and pass^k estimates, averaged across tasks.

Estimating from n trials per task

In practice p differs per task, and you run n ≥ k trials. If c of the n trials succeed on a task, the unbiased estimators are:

pass@k = 1 − C(n − c, k) / C(n, k)
pass^k = C(c, k) / C(n, k)

where C is the binomial coefficient. Average across tasks. The pass@k estimator is the one popularized by the Codex paper (Chen et al., 2021); the pass^k form is its mirror, and tau-bench defines pass^k as the probability that all k i.i.d. trials of a task succeed, averaged over tasks.

Notice what the average hides. Because the estimator averages p_i^k across tasks, it is not equal to (average p)^k. Suppose half your tasks have p = 1.0 and half have p = 0.6. Mean p is 0.8. The true pass^8 is 0.5 × 1 + 0.5 × 0.6^8, about 0.508, far higher than 0.8^8 = 0.168. Reliability is concentrated in the tasks the agent already masters. The practical reading: pass^k is a map to your flaky tasks, not just a scoreboard. Sort tasks by c/n and fix the middle of the distribution.

How many trials do you need?

Standard error shrinks with both tasks and trials, but tasks matter more for generalization. A rough illustration: with 100 tasks and n = 5 trials, you can estimate pass^3 reasonably, but the confidence interval on any single task is wide. Use bootstrap resampling over tasks for the interval on the aggregate, and report it. A 3-point difference between two prompt versions on 100 tasks is frequently inside the noise. If you cannot afford more trials, run paired comparisons: same tasks, same seeds where supported, and test the per-task difference.

Temperature, seeds and what variance you are measuring

Setting temperature to zero reduces but does not eliminate variance: batched inference, floating point non-associativity and tool nondeterminism can still change paths. More importantly, a temperature-zero run measures a variance you will not see in production if production samples at higher temperature, or the reverse. Evaluate at the settings you ship. When the vendor offers a seed parameter, record it and treat it as best-effort, not a guarantee.

Also separate agent variance from environment variance. If the user simulator or a tool is flaky, pass^k penalizes the agent for the environment’s randomness. Measure environment-only noise by replaying a recorded, fixed agent trajectory a few times and checking that the grader returns the same verdict.

Building the Harness: Environments, Simulators and Graders

Environment determinism

A good evaluation environment is resettable, hermetic and fast. Resettable means every trial starts from an identical snapshot: a container image plus a database dump, or an in-memory fixture. Hermetic means no live network dependency, because a third-party API that changes or rate-limits will contaminate your scores. Fast means a trial costs seconds of infrastructure, not minutes, so that you can afford n = 5 trials on hundreds of tasks.

For tool-heavy agents, build tool mocks that implement the real contract: the same schema, the same error shapes, realistic latency, and state. A mock that always returns success teaches you nothing about error recovery. Inject faults deliberately: a timeout on the second call, a 429, an empty search result, a malformed field. Agent robustness is largely about what happens after the first failure, and you will never see it with happy-path mocks.

User simulators

Conversational agents need someone to talk to. A user simulator is an LLM given a persona, a hidden goal and instructions on when to reveal information, when to be vague, and when to end the conversation. tau-bench popularized this: the simulator holds the instruction (for example, “return the two items from order W123 and refund to the original card”) and the agent must elicit and act on it.

Simulators are powerful and fragile. They may leak information too eagerly, drift off script, accept bad answers politely, or fail the task for reasons unrelated to the agent. Mitigate by validating the simulator itself: run a scripted, known-good agent and confirm it passes; run a known-bad agent and confirm it fails. Pin the simulator model version, because swapping it changes your baseline just as much as swapping the agent. Where fidelity matters, keep a small set of human-written conversation replays as an anchor.

Figure 4: One evaluation episode. The simulator drives the conversation, the agent calls tools that mutate the database, and the grader compares final state to the expected state.

Grading: state diffs first, judges second

The grader hierarchy, from most to least trustworthy:

Deterministic state checks. Compare the database, filesystem or repository to an expected state or run an assertion. Fast, free, reproducible.
Programmatic trajectory invariants. Assertions over the trace, such as “no write tool before user confirmation”, “no more than N tool calls”, “arguments validate against the schema”.
Rubric-based LLM judge. Needed for qualities you cannot assert: tone, whether an explanation was faithful to the tool outputs, whether a refusal was appropriate. Use binary or narrow-scale rubrics, and calibrate the judge against human labels as covered in the guide to LLM-as-a-judge evaluation linked above.
Human review. A small, regularly sampled slice to detect grader drift.

State diffs need care. Compare only fields that matter, ignore timestamps and generated IDs, and define equivalence classes where several states are acceptable. Over-strict checkers reproduce the SWE-bench Verified problem in miniature: tests that reject correct behavior. Over-loose checkers pass broken agents. Review a sample of failures and passes by hand whenever you write a new checker; most checker bugs show up within twenty minutes of reading real traces.

Cost and latency belong in the score

An agent that succeeds after 40 tool calls and 300,000 input tokens is not equivalent to one that succeeds in 6 calls. Record per-trial: total input and output tokens, number of model calls, number of tool calls, wall-clock time and estimated dollar cost. Report success rate alongside median and p95 cost per successful task. A more useful single figure is cost per resolved task: total spend across all trials divided by the number of successes, since failed attempts still cost money.

An illustrative calculation: agent A succeeds 70% of the time at $0.12 per trial, agent B succeeds 80% at $0.40 per trial. Cost per resolved task is about $0.171 for A and $0.50 for B. If a failure falls back to a human at $4.00, expected cost per task is 0.12 + 0.30 × 4 = $1.32 for A and 0.40 + 0.20 × 4 = $1.20 for B. Which one wins depends entirely on the fallback cost. The numbers here are invented to show the structure of the decision, not measured values.

Regression harness design

The evaluation you run once at model selection is a benchmark. The one you run on every change is a regression suite, and it needs a different design.

Use tiers. A smoke tier of 20 to 50 high-signal tasks runs on every pull request in minutes and blocks obvious breakage. A full tier of hundreds of tasks with n = 3 to 5 trials runs nightly or before release and produces pass^k with intervals. A canary tier replays sampled production traces, redacted, against the new version offline. Store results by git commit, prompt hash, model identifier and tool schema version so any score can be reproduced.

Gate on paired differences, not absolute thresholds. “pass^3 dropped by more than 4 points with the lower bound of the interval below zero” is a decision; “must be above 70%” is brittle as the suite evolves. Tag tasks by capability (retrieval, multi-step planning, error recovery, policy adherence) so a regression report reads as “policy adherence fell” rather than “score fell”.

A Runnable Python Harness

The following minimal harness runs each task n times, grades by environment state, records cost and computes pass@k and pass^k with the unbiased estimators. It uses a stub agent so you can run it as is, then swap in your own agent callable.

import random
from math import comb
from dataclasses import dataclass, field
from statistics import median

@dataclass
class Task:
    id: str
    goal: str
    initial_db: dict
    expected_db: dict          # fields that must match at the end
    max_tool_calls: int = 12

@dataclass
class Trial:
    task_id: str
    passed: bool
    tool_calls: int
    tokens: int
    violations: list = field(default_factory=list)

def stub_agent(task, db, rng):
    """Replace with a real agent. Must mutate db via tools and return stats."""
    calls, tokens = 0, 0
    for key, value in task.expected_db.items():
        calls += 1
        tokens += 800
        if rng.random() < 0.85:          # per-step success probability
            db[key] = value
    return {"tool_calls": calls, "tokens": tokens, "log": []}

def grade(task, db, stats):
    violations = []
    if stats["tool_calls"] > task.max_tool_calls:
        violations.append("too_many_tool_calls")
    outcome_ok = all(db.get(k) == v for k, v in task.expected_db.items())
    return outcome_ok and not violations, violations

def run_trial(task, agent, seed):
    rng = random.Random(seed)
    db = dict(task.initial_db)           # reset to snapshot
    stats = agent(task, db, rng)
    ok, violations = grade(task, db, stats)
    return Trial(task.id, ok, stats["tool_calls"], stats["tokens"], violations)

def pass_at_k(n, c, k):
    return 1.0 if n - c < k else 1.0 - comb(n - c, k) / comb(n, k)

def pass_hat_k(n, c, k):
    return comb(c, k) / comb(n, k) if c >= k else 0.0

def evaluate(tasks, agent, n=8, ks=(1, 2, 4, 8)):
    per_task, all_trials = {}, []
    for t in tasks:
        trials = [run_trial(t, agent, seed=hash((t.id, i)) & 0xFFFF)
                  for i in range(n)]
        per_task[t.id] = sum(x.passed for x in trials)
        all_trials += trials
    report = {}
    for k in ks:
        report[f"pass@{k}"] = sum(pass_at_k(n, c, k) for c in per_task.values()) / len(tasks)
        report[f"pass^{k}"] = sum(pass_hat_k(n, c, k) for c in per_task.values()) / len(tasks)
    wins = [x for x in all_trials if x.passed]
    report["median_tokens_per_success"] = median(x.tokens for x in wins) if wins else None
    report["tokens_per_resolved"] = (sum(x.tokens for x in all_trials) / len(wins)) if wins else None
    return report, per_task

if __name__ == "__main__":
    tasks = [Task(f"t{i}", "update record", {}, {"a": 1, "b": 2, "c": 3}) for i in range(30)]
    report, per_task = evaluate(tasks, stub_agent)
    for name, value in report.items():
        print(name, round(value, 3) if isinstance(value, float) else value)

With a per-step success of 0.85 across three steps, per-trial success is about 0.61, so you should see pass@8 near 1.0 and pass^8 near 0.02 on this stub. That is the reliability gap in forty lines. To make it production-grade, add: a bootstrap confidence interval over tasks, trace persistence keyed by commit and prompt hash, trajectory invariants as pluggable checks, a pairwise comparison mode against a stored baseline, and parallel execution with per-trial timeouts.

Wiring in real traces

If your agent already emits OpenTelemetry spans, you can grade from the exported trace instead of instrumenting the agent twice. Group spans by trace ID, extract tool names and arguments from the span attributes defined in the GenAI conventions, and feed them to the same invariant checkers. Evaluate offline against the same trace schema you use in production, and the eval suite doubles as a regression test for your instrumentation.

Trade-offs, Gotchas, and What Goes Wrong

Benchmark overfitting. Teams tune prompts against a fixed public suite until scores rise and real-world behavior does not. Hold out a private split you never tune against, and rotate tasks. The SWE-bench Verified episode shows the end state of this dynamic at industry scale: scores went up, signal went down.

Contamination. If tasks or solutions are on the public web, models may have seen them. Prefer tasks written after a model’s training cutoff, privately authored tasks, or freshly mined repository issues. Treat suspiciously high scores on public sets as a prompt to investigate, not to celebrate.

Flaky tools and environments. A 2% environment failure rate caps your pass^k and misleads. Track environment errors separately from agent failures and exclude infrastructure errors from the numerator and denominator, but report their rate.

Judge bias. LLM judges favor longer answers, their own model family’s style and confident tone. If a judge scores trajectories, randomize order in pairwise comparisons, use a different model family from the agent where possible and calibrate against human labels.

Simulator leakage and sycophancy. A user simulator that volunteers the answer makes tasks easier than reality. A simulator that agrees with every agent claim hides hallucination. Include adversarial personas: impatient, vague, contradicting, policy-violating.

Reward hacking by the agent. Coding agents have been observed editing tests or special-casing to satisfy a checker rather than fixing the bug. Protect the grader: run tests from a clean checkout of the test files, forbid writes to test directories, and diff for suspicious changes.

Averaging away the tail. A mean score of 85% can coexist with a handful of catastrophic failures, such as deleting data or leaking PII. Track a separate safety suite with zero-tolerance gates rather than folding it into the average.

Cost blow-ups. Evaluating with n = 8 on 500 tasks with a frontier model can be expensive. Budget it: estimate tokens per trial from a pilot, multiply by tasks and trials, and use the tiering above so the expensive run happens rarely.

Practical Recommendations

Start small and executable. Pick 30 real tasks from your logs, write a state-based checker for each, and run them n = 5 times. Look at the pass^k curve before anything else. If pass^1 is high but pass^5 collapses, your problem is variance, not capability, and the fixes are different: tighter tool schemas, explicit confirmation steps, lower temperature, retrieval of policy text rather than memory, and fewer degrees of freedom per step.

Make failures legible. Store full traces, tag each failed trial with a root cause category (wrong tool, wrong argument, premature stop, policy violation, environment error, simulator error), and review twenty failures per week. The distribution of categories tells you what to build next.

Gate releases on paired comparisons with intervals, and when you change the model, the prompt, the tool schema or the simulator, change one at a time so the delta is attributable. Feature-flag the agent version so evaluation results and production exposure are linked; the patterns in feature flag architecture with OpenFeature apply directly to progressive rollout of an agent configuration after it clears the suite.

Checklist:

[ ] Every task has a deterministic, state-based checker reviewed against real traces
[ ] Environment resets from a snapshot and is hermetic
[ ] Each task runs n >= 5 trials; report pass@1 and pass^k
[ ] Trajectory invariants cover safety and policy rules
[ ] Cost per resolved task is reported next to success rate
[ ] User simulator validated with known-good and known-bad agents
[ ] Private held-out split never used for tuning
[ ] Regression gate uses paired differences with intervals
[ ] Environment errors are tracked separately from agent failures

Frequently Asked Questions

What is the difference between pass@k and pass^k?

pass@k is the probability that at least one of k attempts succeeds, so it rewards retries and measures capability. pass^k is the probability that all k attempts succeed, so it measures reliability. For a per-attempt success rate of 0.8, pass@8 is nearly 1.0 while pass^8 is about 0.17. Use pass@k when a verifier or human selects among attempts, and pass^k when every run reaches a user unchecked. Always state which one you are reporting, because they can differ enormously for the same agent.

Is SWE-bench Verified still a good benchmark?

It remains a useful historical reference, but its signal has weakened. In February 2026 OpenAI said it had stopped reporting SWE-bench Verified because of flawed tests and evidence of training-data contamination, and it recommended SWE-bench Pro. Use Verified results as a rough ceiling, prefer newer or privately built task sets, and above all measure your agent on tasks from your own repositories with your own tests before trusting any public number.

What is tau-bench and why does it matter?

tau-bench is a benchmark from Sierra that evaluates agents in customer-service domains, originally retail and airline, using an LLM-simulated user, policy documents and API tools. It grades by comparing the final database state to the expected state, and it introduced pass^k to expose inconsistency. It matters because it modeled the real deployment shape, a conversation with a human who has hidden goals, and showed that agents which look strong on one trial often fail repeated trials.

Should I grade agents with an LLM judge or with code?

Use code first. State comparisons and trace invariants are deterministic, cheap and free of judge bias. Add an LLM judge only for qualities you cannot assert, such as tone or faithfulness of an explanation to tool outputs, and constrain it with narrow rubrics. Calibrate the judge against human labels on a sample and re-check periodically. If a single behavior can be tested by code, write the code; judges should cover the remainder, not the core outcome.

How many trials and tasks do I need?

There is no universal number. A practical starting point is 30 to 100 tasks with 3 to 5 trials each, enough to see the shape of the pass^k curve and catch large regressions. Small differences need more data or paired comparisons with bootstrap intervals. Add tasks before adding trials when you care about generalization, and add trials when you care about reliability of specific critical workflows. Report intervals so readers can judge significance.

How do I evaluate an agent trajectory, not just the final answer?

Record every model call and tool call as a trace, then run checks over it: argument validity against the tool schema, ordering invariants such as confirming before writing, step and token budgets, redundant calls, and recovery after errors. Avoid requiring an exact golden sequence for open-ended tasks, since many orderings are valid. Combine trajectory checks with a state-based outcome gate, so you learn both whether the task succeeded and how fragile the path was.

LLM Agent Evaluation: Trajectory Evals, SWE-bench, tau-bench and pass^k Reliability (2026)