AI Agent Trajectory Evaluation Patterns for 2026

Your agent passed every task-success check and still shipped a regression nobody caught. It reached the right final state, but it called a deletion tool three times, looped on a paginated API, and burned four extra model turns getting there. The AI agent trajectory evaluation patterns in this post exist because the final answer is only half the signal: the path an agent takes is where cost, safety and latency regressions hide. A green outcome dashboard can sit on top of a decision process that is quietly rotting.

This matters more in late 2026 than it did a year ago. Agents now hold write access to ticketing systems, databases and payment rails, and a model upgrade or a one-line tool-description edit can change behavior without changing a single end result. You will leave with a layered harness design, a scoring vocabulary you can gate on, a capture method based on OpenTelemetry, statistics for deciding whether a regression is real, and a Python scorer you can adapt.

What this covers: the outcome-versus-trajectory split, reference-based and reference-free scoring, deterministic checks versus LLM judges and their known biases, trace capture, golden sets, reliability across repeated trials, CI gates, and the failure modes of the whole approach.

Context and Background

Single-shot LLM evaluation is a mostly solved problem. You have a prompt, a response and a scoring function: exact match, overlap metrics, or an LLM grader. Multi-step agents broke that model. An agent plans, calls tools, reads results, replans and recovers from errors across many turns. The unit of work is no longer a string. It is a trajectory: an ordered sequence of model calls, tool calls, arguments, observations and the final state they produce.

Most teams began with whatever was easiest, which meant checking the final outcome. The benchmark that made this respectable was tau-bench from Yao, Shinn, Razavi and Narasimhan (arXiv:2406.12045). It simulates a conversation between an LLM-played user and an agent that has domain API tools and policy guidelines, then scores the database state at the end of the conversation against an annotated goal state. That is outcome evaluation done well: cheap, faithful and deterministic. The same paper also exposed its limit. The authors introduced the pass^k metric to measure consistency across repeated trials, and reported that function-calling agents such as GPT-4o succeeded on fewer than 50% of tasks, with pass^8 below 25% in the retail domain. Those figures date from 2024 models, so treat them as a demonstration of the method rather than a current leaderboard. The lesson that survives is that a single run hides variance.

Two industry threads converged afterwards. The first is observability. The OpenTelemetry GenAI semantic conventions define span types for model calls, agent invocations and tool execution, so a trajectory can be recorded as a structured trace instead of ad hoc logs. Our walkthrough of OpenTelemetry GenAI semantic conventions for LLM agent tracing covers the capture side in depth. The second thread is research on grading process rather than product. The Agent-as-a-Judge paper by Zhuge and colleagues (arXiv:2410.10934) argues that current methods either judge only final outcomes or demand heavy manual effort, and proposes agentic evaluators that give feedback across the whole task-solving process. TRAIL (arXiv:2505.08638) went further and built 148 human-annotated agent traces with a formal error taxonomy. Its headline result is sobering: the best model tested, Gemini-2.5-pro, scored 11% at localizing the errors. If frontier models struggle to diagnose a bad trace, you should not expect a lazy “rate this trajectory 1 to 10” prompt to do it for you.

Tooling followed. LangChain’s open-source AgentEvals package offers trajectory matching modes (strict, unordered, subset and superset) plus an LLM-as-judge trajectory evaluator that needs no reference. Google’s Vertex AI evaluation service documents a family of trajectory metrics (exact match, in-order match, any-order match, precision and recall over tool calls). The vocabulary is converging, which is useful, because it means the patterns below transfer between stacks.

The thesis of this post is that trajectory evaluation is not a product you buy but a discipline you assemble: deterministic checks where the world is deterministic, judged checks where it is not, and a statistical gate that forces both to agree before code ships. For the single-response version of the judging problem, see our LLM-as-judge evaluation pipeline guide, which this post extends to multi-step paths.

The Reference Harness: Outcome Plus Trajectory, Gated in CI

AI agent trajectory evaluation is the practice of scoring the full ordered sequence of an agent’s steps (tool selections, arguments, observations and error recovery) rather than only its final answer. It measures whether the path was correct, efficient and safe, and it catches cost and risk regressions that outcome-only checks cannot see. Run it as one pipeline that ends in a hard CI gate.

Figure 1: The reference harness. Instrument the agent, store spans, score outcome and trajectory separately against a golden set, aggregate, and gate the pipeline.

The harness has five stages. The agent under test emits OpenTelemetry spans for every model call and tool invocation. Those spans land in a trajectory store. Two scorers read the same trace: an outcome scorer that checks the final state and a trajectory scorer that grades the path. Both consult a golden set, which holds expected tool sequences, assertions and rubrics. An aggregator rolls per-run scores into metrics, compares them to a baseline and emits a verdict plus a regression report. The gate is the point. An evaluation that blocks nothing is decoration.

The original claim here is that outcome and trajectory are different measurements with different failure modes. Outcome answers “did it work?” Trajectory answers “would I trust how it worked, at scale, on a bad day?” You need both numbers, stored independently, and the disagreements between them are the most valuable output of the system.

Outcome scoring is assertions, not vibes

Make outcome scoring as deterministic as the task allows. For a database agent, assert the final rows. For a coding agent, run the tests the change was meant to fix. For a support agent, compare the resulting state of the order system against the goal state, exactly as tau-bench does. The output is a binary success bit per run, aggregated into a task success rate.

The common trap is the soft outcome check that quietly becomes a judge call. If you catch yourself asking an LLM “did the agent answer correctly?” for a task where a concrete assertion exists, you have traded a stable signal for a noisy one to save twenty minutes of test authoring. Pay the twenty minutes. Reserve judged outcomes for open-ended deliverables such as a written summary or a plan, and even then anchor the judge with a rubric and a reference answer.

Trajectory scoring decomposes the path

Trajectory scoring grades four largely separable properties of the path:

Tool selection: did the agent pick the right tool for each decision point?
Argument correctness: were the arguments well formed and semantically right?
Redundancy: did it repeat calls, loop, or fetch what it already had?
Recovery: when a tool failed or returned something unexpected, did it adapt or flail?

Keep these four separate in storage, not only in scoring. When a regression fires, you want to know which dimension moved. A drop in tool selection points at a prompt or tool-description change. A drop in argument correctness usually points at a schema or formatting issue. A redundancy spike means the agent is looping. A recovery decline means it stopped adapting to errors. A single blended path-quality number hides all of that diagnostic signal, and getting that signal is the whole reason to invest.

Where outcome and trajectory disagree

Consider a support agent asked to refund an order. The refund posts, so the outcome scorer returns 1. The trajectory shows the agent called lookup_customer four times with slightly different arguments before finding the record, then issued the refund without the policy check that the golden path requires. Outcome is green and trajectory is alarming. Two weeks later the missing policy check becomes an incident, and the only reason you could see it coming is that you captured the path.

The reverse case matters too. A run that fails with a clean trajectory is often an environment problem: a flaky tool, a stale fixture, an unreachable sandbox. You debug that very differently from a reasoning failure. Collapsing both into one number throws away exactly the information that tells you where to look. The 2×2 below is worth printing for the team.

	Good trajectory	Poor trajectory
Outcome pass	Healthy. Track cost and latency.	Latent regression. Fix before it becomes an incident.
Outcome fail	Suspect environment, tool or golden-set error.	Genuine agent failure. Debug reasoning and prompts.

Reference-based versus reference-free scoring

There are two families of trajectory scorers. Reference-based scoring compares the observed path to a golden trajectory, the sequence of calls a competent operator would make. It yields crisp, cheap metrics such as tool-call precision, recall and F1, and step accuracy. It is exact when valid paths are few and brittle when they are many. Reference-free scoring grades the path against a rubric with no golden path, tolerating diversity at the price of cost and noise, because it leans on an LLM judge.

The choice is per task, not per harness. “Look up an order and issue a refund” has essentially one correct tool sequence, so a reference is exact and cheap. “Research this vendor and summarize the risk” has dozens of valid routes, and forcing it onto a golden path produces false regressions whenever the agent picks a different but fine route. The practical rule is to author golden paths for low-entropy tasks and grade high-entropy tasks with rubrics. Tag each task with its scoring family so the aggregator knows which metrics are meaningful for it. Averaging tool-call F1 across both families produces a number that means nothing.

There is also a hybrid that works well in regulated settings: reference-based assertions on the subset of calls that must always appear, such as an authorization check or an audit-log write, with rubric scoring on everything around them. This catches the safety-critical steps deterministically without forcing the whole trajectory into a rigid template.

Existing libraries map neatly onto this. AgentEvals’ four match modes are really four different questions. Strict asks whether the same calls happened in the same order. Unordered asks whether the same calls happened at all. Subset asks whether the agent stayed inside the reference toolset, which is a least-privilege check. Superset asks whether the key reference calls appeared while tolerating extras, which is the natural fit for the mandatory-steps hybrid. Choosing the mode per task family is a design decision, not a configuration detail.

Capturing Trajectories as OpenTelemetry Spans

You cannot evaluate a trajectory you did not record faithfully. The 2026 answer is to record it as OpenTelemetry spans, using the GenAI semantic conventions maintained in the open-telemetry/semantic-conventions-genai repository. These conventions are still marked Development status in the current specification, so attribute names can change between releases. Pin your instrumentation library version and keep a thin adapter between raw spans and your scorers.

Figure 2: A trajectory as a span tree. The agent invocation parents model calls and tool executions, and the exported tree is replayable into the scorers.

The spec defines a gen_ai.operation.name attribute with values that matter for trajectories: invoke_agent for an agent run, chat for a model call, plan for a planning phase, and execute_tool for a tool invocation. An execute_tool span carries gen_ai.tool.name (required), and optionally gen_ai.tool.call.id and gen_ai.tool.type. The result is a tree you can walk: the agent span is the root, model calls and tool executions are children in temporal order, and the order of execute_tool children is, in effect, your tool-call sequence.

The opt-in trap

The detail that bites teams is that gen_ai.tool.call.arguments and gen_ai.tool.call.result are Opt-In attributes in the spec, as are gen_ai.input.messages and gen_ai.system_instructions. They are off by default for good reasons: arguments and results can contain customer data, credentials and personal information. A trace without them is useless for argument-correctness scoring, because “called search_orders” says nothing about whether it was called with the right customer ID.

Resolve this with environment separation, not wishful thinking. In the CI evaluation environment, where fixtures are synthetic, enable full content capture. In production, keep content capture off or route it to a restricted store with its own retention and access policy, and sample selectively. This also determines which evaluations you can run on production traces. Without arguments you can still compute tool-sequence metrics, redundancy, turn count, latency and token cost. You cannot compute argument accuracy or judge reasoning quality.

What capture buys beyond evaluation

Standard spans pay off in three ways. First, the same traces drive production debugging, so the eval harness and the on-call dashboard read identical data. Second, you can run evaluation offline by replaying recorded traces against new scorers, which needs no live tool calls and lets you add a new metric retroactively across last month’s traffic. Third, the conventions are vendor neutral, so you are not locked to one tracing backend. Our piece on long-running governed AI agents describes the same telemetry backbone from the governance side, so evaluation reuses production observability instead of adding a parallel logging path.

One more practical rule: record the environment, not only the agent. Store the tool-fixture version, the model identifier and version, the prompt hash, the temperature and seed if any, and the golden-set version in resource or span attributes. When a score moves, the first question is always “what changed?”, and a trace that cannot answer it forces an archaeology project.

Scoring and Judging Steps

Once you have steps, decide per step how to score them. The routing below sends each step to the cheapest defensible scorer.

Figure 3: Step-level scoring pipeline. Deterministic checks first, rubric-grounded judging with bias controls for the irreducibly fuzzy steps, and calibration against human labels before aggregation.

Exact match where you can, judge where you must

For a tool call you can usually check correctness without a model. Did the agent call search_orders where the golden path expects it? Do the arguments validate against the tool’s JSON schema and match expected values? That is comparator territory: fast, free and stable. Reach for an LLM judge only when quality is genuinely subjective: was this clarifying question reasonable, was this summary faithful, was the replan sensible given the error.

Argument matching deserves nuance, because strict equality is too brittle. A timestamp, a generated ID or a paraphrased query differs run to run without being wrong. Define per-argument comparators: exact for enums and identifiers, set membership for categories, numeric tolerance for thresholds, and semantic similarity only for free-text fields. AgentEvals exposes the same idea through its tool_args_match_mode setting (exact, ignore, subset, superset) and per-tool overrides that accept a list of fields that must match exactly or a custom comparator. Whatever you use, the comparator config doubles as documentation of what “correct arguments” means for each tool.

A metric vocabulary to gate on

Fix the vocabulary before you build the aggregator. Each row below is a candidate gate.

Metric	Measures	How computed	Catches
Task success rate	Outcome correctness	Share of runs passing deterministic assertions	Wrong final result
Tool-call precision	No spurious calls	Correct calls over observed calls	Extra or destructive calls
Tool-call recall	No missing calls	Expected calls found over expected	Skipped checks
Argument accuracy	Well-formed inputs	Per-argument comparators	Malformed or wrong parameters
Redundancy rate	Wasted work	Repeated calls over total calls	Loops, re-fetching
Recovery score	Error adaptation	Rubric judge on failed-call handling	Flailing after tool errors
Turn count (p50, p95)	Path length	Number of agent steps	Latency and cost creep
Cost per task	Spend	Summed token and tool cost	Quiet budget regressions
Pass^k	Reliability	Probability all k trials pass	Flaky behavior

You will not gate on all of them for every task family. Naming them forces an explicit decision about which matter where, and that decision is the substance of an evaluation strategy.

A compact trajectory scorer

The scorer below encodes the routing and the outcome/trajectory split. It is a pattern sketch, not a library: extract_steps, align, args_match and judge are yours to implement against your span adapter.

def score_trajectory(trace, golden, judge, rubric):
    steps = extract_steps(trace)                  # from execute_tool and chat spans
    expected = golden.tool_calls                  # list of (tool, args)
    observed = [(s.tool, s.args) for s in steps if s.is_tool_call]

    # Reference-based, deterministic metrics
    obs_names = [t for t, _ in observed]
    exp_names = [t for t, _ in expected]
    hits = sum(1 for n in set(obs_names) if n in exp_names)
    precision = hits / max(len(set(obs_names)), 1)
    recall = hits / max(len(set(exp_names)), 1)
    arg_acc = mean(args_match(o, e) for o, e in align(observed, expected))
    redundancy = count_repeats(observed) / max(len(observed), 1)

    # Mandatory steps are hard assertions, never averaged away
    mandatory_ok = all(m in obs_names for m in golden.mandatory_tools)

    # Reference-free, judged steps (only the fuzzy ones)
    judged = [judge.score(step=s, rubric=rubric.recovery,
                          anchors=rubric.few_shot,
                          controls=["shuffle_positions", "length_normalize"])
              for s in steps if s.needs_judge]
    recovery = mean(j.score for j in judged) / 5 if judged else 1.0

    path_quality = weighted_mean(
        {"precision": precision, "recall": recall, "arg_acc": arg_acc,
         "no_redundancy": 1 - redundancy, "recovery": recovery},
        weights=rubric.weights)

    outcome = assert_final_state(trace, golden.assertions)   # 0 or 1
    return {"outcome": outcome, "mandatory_ok": mandatory_ok,
            "path_quality": path_quality, "precision": precision,
            "recall": recall, "arg_acc": arg_acc, "turns": len(steps),
            "cost": trace.total_cost}

The shape matters more than the syntax. Deterministic metrics are computed directly, fuzzy steps go to a grounded judge, mandatory steps are a separate boolean so a perfect score elsewhere cannot hide a skipped authorization check, and outcome stays its own field so the aggregator can detect disagreement.

Aggregation is its own design decision

Do not average outcome and path quality into one headline number. Keep task success rate and median path quality as separate gates, and add explicit gates on cost and turn count. A useful report shows, per task family: success rate, the trajectory sub-scores, p50 and p95 turn count, cost per task, and the delta against the previous baseline. The gate fails if any guarded metric regresses beyond its threshold. That is what turns a chart into an enforceable contract.

Own the trajectory data even if you rent the scoring interface. A harness worth trusting is mostly a set of contracts (capture format, scorer interface, gate semantics) rather than a specific vendor. It must let you swap judges, version golden sets and inspect any failing trajectory down to the span. If a tool hides the raw trace, you will end up reverse-engineering scores from a dashboard.

Grounding the Judge: Rubrics, Anchors, and Known Biases

An ungrounded LLM judge is a random number generator with good manners. Grounding means three things: an explicit rubric with score definitions, few-shot anchors showing real examples at several score levels, and a forced rationale before the score. A workable rubric for the recovery dimension:

RECOVERY (1-5): When a tool call failed or returned an error,
did the agent adapt sensibly?
5 = Diagnosed the error, chose a correct alternative, made progress.
3 = Retried the same call once, then adapted.
1 = Looped on the failing call or ignored the error and proceeded.
Output JSON: {"rationale": "...", "score": <int>}
Anchors: [error, reauth, retry = 5] [same call x3, same args = 1]

Rationale-first ordering is not cosmetic. It makes the judge commit to reasoning before emitting a number, which tends to reduce arbitrary scores, though you should confirm that on your own data rather than assume it.

Anchors do the heavy lifting. A prose rubric tells the judge what 5 and 1 mean in words, while an anchor shows it with a real trajectory snippet and its assigned score. Without anchors, two judge models, or one model two versions apart, interpret “sensible progress” differently, and your scores stop being comparable over time. Three or four anchors spanning the scale pin it. Authoring them is the most underrated task in the whole harness, and the best source is real failures from your own traces.

Documented judge biases and the controls that follow

The foundational LLM-as-judge paper by Zheng and colleagues (arXiv:2306.05685) reported that strong judges such as GPT-4 reach over 80% agreement with human preferences, comparable to human-human agreement, and also named the limitations: position bias, verbosity bias, self-enhancement bias and limited reasoning ability. Each has a trajectory-specific form.

Position bias appears when comparing two trajectories side by side. Run each comparison twice with the order swapped, and count a win only when both orderings agree.
Verbosity bias rewards long reasoning traces and chatty intermediate messages. Normalize for length, or judge the actions and observations while hiding the model’s narration.
Self-enhancement bias means a model may prefer outputs from its own family. Use a judge from a different family than the agent, or ensemble two judges and review disagreements.
Limited reasoning shows up on long trajectories. Do not hand a 60-step trace to a judge and ask for one score. Judge step windows, or have a deterministic pass extract the suspicious steps first.

TRAIL’s result is the reason for that last point. Localizing errors across a long trace is hard even for strong long-context models, so the more you can decompose the trajectory before it reaches a judge, the more reliable the judge becomes. Agent-as-a-Judge takes the opposite approach, giving the evaluator its own tools to inspect artifacts. That is promising for code-generation tasks, but it adds a second agent whose own trajectory you now have to trust.

Calibrate against humans, then keep calibrating

A judge is an instrument, and instruments need calibration. Label 100 to 200 steps by hand across your task families, run the judge, and measure agreement with a chance-corrected statistic such as Cohen’s kappa instead of raw percent agreement. Raw agreement flatters a judge on skewed data: if 90% of steps are fine, a judge that always says “fine” scores 90%. Rerun the calibration whenever you change the judge model, the rubric, or the agent’s tool set, and store the judge version next to every score so old results stay interpretable.

Golden Sets, Reliability, and the CI Gate

A harness is only as good as the tasks it runs. A golden set is a versioned collection of tasks, each with a starting fixture, an outcome assertion, a scoring family tag, and where appropriate a golden trajectory or rubric. Build it from three sources: hand-authored tasks for the behaviors you must never break, production traces that exposed real failures, and synthetic variations that probe edge cases such as tool errors, empty results and ambiguous requests.

Building the set from production

The highest-value tasks come from incidents. Every time an agent misbehaves in production, convert the trace into a regression task: freeze the fixture, write the assertion that would have caught it, and add the mandatory step that was skipped. Over a year this produces a set weighted toward your real failure modes instead of the failures you imagined. Keep a deliberate share of simple happy-path tasks too, because they are your canary for global breakage such as a broken tool schema.

Version the set like code. Each task carries an identifier, a version and an owner. When a golden path changes because a tool was legitimately redesigned, the change goes through review, and the baseline is recomputed on the old and new sets so the effect of the set change is separated from the effect of the agent change. A moving golden set with no versioning is how teams end up chasing regressions that were really test edits.

One run is an anecdote: measure reliability

Agents are stochastic. The tau-bench authors made this concrete with pass^k, the probability that an agent succeeds on all of k independent trials of the same task. If a task is attempted n times with c successes, an unbiased estimate of the per-task value is the number of ways to choose k successes from c divided by the number of ways to choose k trials from n, averaged over tasks. The practical reading is simpler: an agent that succeeds 90% of the time per attempt succeeds on eight consecutive attempts only about 43% of the time (0.9 to the power of 8). A customer-facing workflow that must work every time needs a pass^k gate, not a pass@1 gate.

The same logic applies to trajectory metrics. Run each task several times, for example 3 to 5 trials in CI, and record the distribution of turn count, cost and path quality, not just the mean. A p95 turn count that doubles while the median stays flat is a real regression, a rare looping path that the average hides. Trial counts are a budget decision: a suite of 150 tasks at 5 trials with an average of 12 model calls per run is 9,000 model calls per CI run, so price it before you commit and consider a small smoke suite on every pull request with the full suite nightly.

Deciding whether a difference is real

Because both the agent and the judge are noisy, you need a rule for when a change is a regression and not wobble. Treat success rate as a proportion and compute a confidence interval. With 150 tasks at 5 trials you have 750 runs, but they are not independent because trials of the same task correlate, so resample at the task level (a bootstrap over tasks) rather than treating runs as independent draws. As a rough illustration, a baseline success rate of 85% measured on 150 tasks has a 95% interval of roughly plus or minus 6 points, so a one-point dip means nothing and a ten-point dip is a signal. These numbers are illustrative arithmetic, not a benchmark, and you should compute intervals on your own data.

Two refinements help. Use paired comparisons: run baseline and candidate on the same tasks and look at the per-task differences, which removes between-task variance and tightens the interval considerably. And set asymmetric gates: be strict on safety-relevant metrics such as mandatory-step compliance and destructive-call counts, where any regression blocks, and tolerant on soft metrics such as redundancy, where a small drift is acceptable.

Figure 4: The CI gate loop. Multiple trials per task, paired baseline comparison with an interval, blocking on guarded metrics, triage by failing dimension, and production traces feeding the golden set.

Gate design that survives contact with a team

A gate that fires constantly gets disabled, and a gate that never fires is theater. Three habits keep it credible. First, every failure must come with evidence: the gate output links the failing traces, the dimension that moved and the delta, so triage takes minutes. Second, thresholds are written down with a rationale and reviewed, not tuned after the fact to make a build pass. Third, there is a defined override path with an owner and an expiry, so an emergency fix does not require deleting the gate.

Finally, close the loop. Triage by dimension, fix the cause (prompt, tool description, schema or golden path), and promote interesting production traces into the golden set. A regression that was caught only by a human reading a trace should become a deterministic assertion afterward, so the same failure can never again depend on someone’s attention.

Operating the Harness: Cost, Latency, and Online Evaluation

Offline CI evaluation answers “is this version safe to ship?” Online evaluation answers “is production behaving like the tests said it would?” You need both, and they use the same scorers on different data.

For online use, sample production traces and score the cheap, deterministic metrics on all of them: turn count, tool-call sequence against mandatory-step rules, redundancy, latency and cost. Score the expensive judged metrics on a small stratified sample, weighting toward low-confidence runs, long runs and runs where the user retried or escalated. Alert on distribution shifts in these metrics rather than on absolute values, since a new user population can move them legitimately.

Judge cost deserves a budget. If a judge call costs a fraction of a cent to a few cents depending on model and trace length (check your provider’s current price list, since this changes often), a nightly run that judges every step of every trial can quietly dwarf the cost of the agent itself. Control it by pushing steps to deterministic checks, judging windows instead of full traces, caching judge results keyed on the hash of the step and rubric version, and using a cheaper judge for triage with a stronger judge reserved for disputed cases.

Latency matters for developer experience. A suite that takes ninety minutes will be skipped. Parallelize across tasks, keep a fast smoke subset of perhaps twenty tasks covering each tool and each scoring family, and run the full suite on merge to main or nightly. Stub slow, deterministic tools with recorded responses so the suite measures the agent rather than your staging environment.

Trade-offs, Gotchas, and What Goes Wrong

Trajectory evaluation is not free, and it fails in predictable ways.

Over-fitting to the golden path. If a reference-based metric punishes any deviation, engineers learn to make the agent mimic the golden path instead of solving the task, and you lose the agent’s ability to find better routes. Reserve strict matching for low-entropy tasks and for mandatory steps. Review a sample of “regressions” where the outcome passed and the path differed, because some of them are improvements.

Judge drift. A hosted judge model can change behavior when the provider updates it, shifting every score without any change on your side. Pin model versions where the provider allows, store the judge version with each score, and keep a frozen calibration set that you rerun to detect drift.

Goodhart effects. Once the agent is optimized against a rubric, scores rise faster than real quality. Hold out a private slice of tasks that never informs prompt tuning, and refresh it periodically.

Non-determinism in the environment. Live tools return different data on different days. Fixtures and recorded responses make runs reproducible, but they can also drift from production reality. Refresh fixtures on a schedule and track the age of each.

Trace leakage. Capturing arguments and results means capturing data. Treat the trajectory store as sensitive: scrub or synthesize inputs, restrict access, and set retention. This is the same tension as the opt-in attributes described earlier, and it should be decided deliberately, not discovered in an audit.

Instrumentation instability. The GenAI conventions are in Development status, and agent frameworks emit them with varying completeness. Some frameworks omit tool arguments or merge spans. Verify with a handful of known trajectories that your adapter reconstructs the sequence you expect, and add a test that fails when the span shape changes.

Evaluating the evaluator. A harness that has never caught a planted bug has unknown sensitivity. Periodically inject a known regression, such as a tool description edited to be ambiguous or a prompt that drops a policy check, and confirm that the gate fires on the right dimension. Treat that as a unit test for your evaluation system.

Multi-agent systems. When several agents delegate to each other, a single trajectory becomes a graph. Score each agent’s sub-trajectory and the handoffs between them, and be careful about blame: a downstream agent that fails because an upstream agent passed bad context should not be penalized for it. The span hierarchy gives you the structure, but the attribution logic is yours to write.

Practical Recommendations

Start small and make the first version blocking, because a non-blocking evaluation does not change behavior. A sensible rollout runs in four steps over a few weeks.

In the first week, instrument with OpenTelemetry GenAI spans, enable content capture in the CI environment only, and verify that you can reconstruct tool-call sequences from stored traces. Then write 30 to 50 golden tasks from real incidents and critical workflows, each with a deterministic outcome assertion and a list of mandatory tool calls. Next, add the deterministic trajectory metrics (precision, recall, redundancy, turn count, cost) and gate on mandatory steps and cost immediately, since those need no judge. Only then add judged dimensions such as recovery, with a written rubric, four or more anchors, and a 100-step human calibration set.

Use this checklist as a review gate:

[ ] Outcome scoring uses assertions wherever a concrete check exists.
[ ] Trajectory sub-scores (selection, arguments, redundancy, recovery) are stored separately.
[ ] Each task is tagged reference-based or rubric-based, and metrics are never mixed across the two.
[ ] Mandatory steps are hard boolean assertions.
[ ] Argument comparators are defined per tool.
[ ] Judges use a rubric, anchors, rationale-first output, order swapping and a different model family from the agent.
[ ] Judge agreement with humans is measured with a chance-corrected statistic and rerun on every judge change.
[ ] Each task runs several trials, and gates use task-level paired intervals.
[ ] Judge and agent versions are stored with every score.
[ ] Thresholds are documented, overrides expire, and failures link to traces.
[ ] Production incidents are promoted into the golden set.
[ ] A planted regression periodically proves the gate still fires.

The overall principle is to spend deterministic effort first and judged effort last. Most of the value of agent trajectory evaluation comes from capturing the path and asserting a handful of facts about it. The elaborate judging layer matters, but only after the cheap, stable checks are in place.

Frequently Asked Questions

What is AI agent trajectory evaluation?

It is the practice of scoring the entire sequence of an agent’s steps, including model calls, tool selections, arguments, observations and recovery from errors, instead of only its final output. It answers whether the agent took a correct, efficient and safe path. Teams use it to catch regressions in cost, latency, safety and reliability that outcome-only checks miss, and to diagnose which part of the decision process changed.

How is trajectory evaluation different from outcome evaluation?

Outcome evaluation checks the end state, such as whether the right database record exists. Trajectory evaluation checks how the agent got there: which tools it called, with what arguments, how many times, and how it handled failures. The two can disagree. A run can pass on outcome while skipping a required policy check, which is invisible to outcome scoring and is exactly what trajectory scoring is designed to expose.

Should I use an LLM judge or exact match for agent steps?

Use exact match, schema validation and per-argument comparators wherever correctness can be checked deterministically, because they are cheap and stable. Use an LLM judge only for subjective qualities such as whether a replan was sensible or a summary faithful. When you do, apply a rubric, few-shot anchors, rationale-first output and bias controls, and calibrate the judge against a human-labeled sample.

How do I capture an agent trajectory?

Instrument the agent with OpenTelemetry using the GenAI semantic conventions, which define invoke_agent, chat and execute_tool operations. Enable the opt-in attributes for tool arguments and results in a controlled evaluation environment, since they are off by default for privacy. Export the span tree to a store, and replay it into your scorers. Note the conventions are in Development status, so pin versions.

How many trials and tasks do I need?

It depends on the effect you must detect, but a single run per task is almost never enough because agents are stochastic. Several trials per task, scored with a paired, task-level comparison against the baseline, gives a usable signal. A few dozen well-chosen tasks per task family is a reasonable start. Compute intervals on your own data instead of copying a number from a blog.

What should block a merge?

Gate hard on safety and compliance metrics: skipped mandatory steps, destructive or out-of-scope tool calls, and drops in task success beyond the confidence interval. Gate on cost and p95 turn count with explicit thresholds. Treat soft metrics such as redundancy and judged recovery scores as warnings that trigger review first. Every gate failure should link to the failing traces and the dimension that moved.

AI Agent Trajectory Evaluation: 2026 Patterns