Pass rate only records whether an AI agent reached the expected final state. It misses wrong tool calls, inefficient paths, and policy violations along the way. Agent trajectory evaluation scores each tool call, intermediate state, and recovery step, so teams can gate releases on agent behavior, not just outcomes.
Pass rate answers an important question: Did the agent complete the task? For agentic AI, that question is no longer enough.
An AI agent can reach the correct final state after selecting the wrong tool, making unnecessary calls, violating an intermediate policy, or recovering from an error unsafely. A terminal evaluator may still record the run as a pass because it only sees the endpoint. This matters as agents take on longer, tool-driven workflows. Each tool call, state transition, and recovery step creates another opportunity for execution to diverge from the intended behavior.
TRAJECT-Bench addresses this gap by evaluating tool selection, argument correctness, and dependency ordering alongside final accuracy. This creates a problem for ML engineers moving agents into production. Agents do not produce only an answer. They produce a sequence of decisions, tool calls, observations, state changes, retries, and actions before reaching that answer.
This article explains where pass rate falls short, what agent trajectory evaluation measures instead, and how teams can use trajectory-level signals in release gates and regression pipelines.
What Pass Rate Actually Measures and Why It's Blind Mid-Trajectory
Pass rate, or task success rate, is typically a binary outcome metric: an agent gets a 1 if it reaches the expected final state and a 0 if it doesn’t. This makes it useful for answering whether a task was completed. The problem is that when used alone, the metric collapses the execution path into a final result.
Consider an agent that needs to authenticate, retrieve customer data, and submit it through an API. If it makes the correct final API call but first makes a wrong tool call and fails authentication twice, a terminal evaluator can still mark the run as a pass. The pass rate records the destination, while ignoring how the agent got there.
This creates three important blind spots:
- Tool call errors: A successful outcome can hide incorrect tool selection, invalid arguments, or calls made in the wrong order. TRAJECT-Bench shows this approach by evaluating tool selection, argument correctness, and dependency ordering alongside final accuracy.
- Inefficient decision paths: An agent may complete a task through unnecessary retries, redundant tool calls, or excessive steps. The endpoint remains correct, but the trajectory can consume more time, tokens, API calls, or compute than the task requires.
- Policy violations: An agent can reach an approved final state after taking an action that should have been blocked. A 2026 review of 15 major agent benchmarks found that 0 of 15 incorporated security or safety into their scoring, while 13 of 15 relied exclusively on binary success measures. The review also identifies planning coherence, resource efficiency, and safety as dimensions that binary metrics can obscure.
Pass rate also hides instability across repeated runs. A 75% per-trial success rate, for example, does not tell an engineering team whether failures are concentrated in particular trajectories or arise unpredictably across runs. Pass@k can further mask this distinction because it counts a task as successful when at least one of k attempts succeeds. This distinction matters when evaluating an agent that must perform reliably in a single customer interaction rather than succeed eventually across multiple samples.
For production agents, the evaluation target therefore needs to move from “Did it finish?” to “Did it execute the task correctly, efficiently, and within constraints?” That requires inspecting the trajectory between the initial state and final outcome.
What Trajectory-Level Evaluation Actually Measures
Traditional chat evaluation usually maps one input to one output and one score. Agent evaluation has a different structure. The evaluator needs to inspect the full interaction: the goal, the agent’s tool call, the tool result, the agent’s next action, the resulting state, and the final answer. This makes evaluation possible at both the step level and outcome level.
The core agent evaluation metrics therefore extend beyond final task success:
- Tool call accuracy: Did the agent select the correct tool and provide valid arguments? A trajectory can be evaluated against expected tool usage, ordering, and parameters. Google Cloud’s Vertex AI evaluation API now includes trajectory-specific metrics such as trajectory exact match, in-order match, precision, recall, and single-tool-use measures.
- Intermediate and state-level review: After each action, the evaluator checks whether the environment is in the expected state. For example, if an agent updates an account, the evaluation should verify that the correct account changed before allowing the next step to proceed. This exposes errors that may disappear by the time the final state is evaluated.
- Error recovery scoring: A trajectory should capture whether the agent recognizes a failed action, diagnoses the error, and takes an appropriate corrective step. Two agents can reach the same final state while showing very different recovery behavior. That difference matters when failures can propagate across later tool calls.
This shift also changes how teams assess consistency. Recent reliability research shows that output-level success can conceal failures in the agent’s intermediate behavior, strengthening the case for examining trajectories rather than relying exclusively on terminal outcomes.
For complex workflows, automated metrics can identify specific trajectory properties, but human review remains valuable for judging whether an action was contextually appropriate, whether recovery was sensible, and whether the overall sequence followed the intended behavior. This creates a more useful evaluation layer between task completion and production reliability.
Building an AI Agent Evaluation Framework for Regression Testing
Agent trajectory evaluation becomes most useful when it is part of the development loop rather than an occasional audit. Each change to a model, prompt, tool schema, retrieval layer, or orchestration policy can alter how an agent reaches the same outcome. A regression framework should therefore compare execution behavior as well as task results.
Build a Trace-Level Evaluation Set
Start with a Trace-Level Evaluation Set that represents important production workflows. Each scenario should specify the expected outcome, allowed or required tools, expected state transitions, policy constraints, known failure conditions, and acceptable recovery behavior.
The evaluator can then capture the complete execution trace, including tool calls, observations, state changes, retries, and the final response. This gives teams a stable reference against which future agent versions can be evaluated.
Compare Agent Versions at the Trajectory Level
When introducing a new model, prompt, tool definition, retrieval system, or orchestration policy, compare its trajectories with the previous version. A regression may appear as additional tool calls, different tool selection, retry loops, missed intermediate checks, weaker recovery, or new policy violations.
The final answer can remain identical while the underlying execution becomes less reliable or more expensive. Trajectory comparison makes these changes visible before they become production issues.
Put Trajectory Metrics Into Release Gates
The final step is to turn trajectory signals into explicit release criteria. For example, a release can require:
- Task success remains above the required threshold
- Critical tool calls meet accuracy requirements
- No high-severity policy violations occur
- Recovery performance meets the minimum threshold
- Trajectory length and cost remain within the defined budget
This turns agent evaluation from a benchmark report into an engineering control. Teams can detect regressions at the step level, block unsafe changes, and maintain a consistent behavioral standard as agents evolve.
How iMerit Operationalizes Agent Trajectory Evaluation
Agent trajectory evaluation requires more than checking whether an agent reaches the expected outcome. The evaluation needs to examine the actions, tool calls, intermediate states, and recovery decisions that lead to that outcome. iMerit’s agent evaluation workflows are designed to evaluate these behaviors across complete execution traces, using structured scenarios and rubrics to assess agent performance.
This supports several layers of trajectory analysis:
- Tool-call accuracy: Verify whether the agent selected the right tool, formed valid calls, handled failures, and validated tool outputs.
- Intermediate-state review: Check whether the agent maintained the expected state throughout the workflow rather than reaching the right endpoint by an invalid path.
- Error and recovery analysis: Identify failed calls, retry loops, dead ends, and whether the agent recovered appropriately.
- Policy and workflow adherence: Evaluate actions against application-specific constraints, escalation rules, and safety requirements.
- Structured trajectory scoring: Convert reviewer judgments into labels, scores, and tags that can be used for benchmarking and regression analysis.
- Version comparison: Compare behavior when models, prompts, tools, policies, or connectors change.
- High-volume evaluation: Run repeatable scenario-based reviews with structured outputs and operational reporting.
Ango Hub in the Production Evaluation Workflow
Ango Hub provides the evaluation workflow for reviewing traces, managing scenario queues, applying rubrics, and producing structured evaluation outputs. Its dashboards and APIs/SDKs can also connect trajectory review with existing evaluation workflows, supporting repeatable, high-volume assessment.
That makes trajectory-level evaluation usable within ongoing development and release workflows. Teams can review individual traces, aggregate failure patterns, compare versions, and feed structured evaluation results into their broader testing systems.
The practical distinction is simple: iMerit helps teams evaluate not only whether an agent passed, but how it behaved on the way to passing.
Building a Production-Ready Agentic AI Evaluation Loop
A production evaluation loop should make trajectory review part of the agent development lifecycle. The process can remain straightforward:
1. Define the Expected Trajectory
For each critical workflow, document the expected tools, state transitions, constraints, recovery behavior, and final outcome. This creates the reference trajectory against which agent behavior can be evaluated.
2. Capture Complete Traces
Record the execution path, including tool calls and arguments, tool responses, observations, state changes, errors, retries, and the final result. Without these intermediate events, teams cannot reliably diagnose where a trajectory failed.
3. Score at Multiple Levels
Evaluate individual actions, the overall trajectory, and the final outcome separately. This helps distinguish a correct result produced through a compliant trajectory from one reached through unnecessary calls, incorrect actions, or unsafe recovery.
4. Inject Failure Scenarios
Test how the agent behaves when tools time out, return invalid responses, become unavailable, or receive ambiguous inputs. Include policy conflicts and other conditions that could force the agent to recover or escalate.
5. Gate Releases and Monitor Regressions
Run new agent versions against a fixed evaluation set and require critical trajectory metrics to remain within defined thresholds. Track changes in tool accuracy, recovery, policy adherence, trajectory length, and cost alongside task success.
The evaluation set should also evolve with the agent. Every meaningful failure discovered in production should become a candidate evaluation scenario. Over time, this creates a regression suite based on observed failure modes rather than only the workflows engineers expected the agent to encounter.
Conclusion
A production evaluation loop should make trajectory review part of the agent development lifecycle. The process can remain straightforward:
Treat pass rate as one signal, not the release decision. Before shipping an agent, evaluate the full trajectory to verify that it uses the right tools, maintains valid intermediate states, handles failures correctly, and stays within defined policy and efficiency limits. This gives ML teams a clearer view of whether an agent is reliable in execution, not just successful at the endpoint.
Key Takeaways
- Pass rate measures outcomes, not execution quality. An agent can complete a task while making incorrect tool calls, taking inefficient paths, or violating constraints along the way.
- Agent trajectory evaluation examines the complete execution path. It can evaluate tool selection, arguments, intermediate states, errors, recovery, and final outcomes together.
- Release gates need trajectory-level signals. Teams can combine task success with thresholds for tool accuracy, policy adherence, recovery, and trajectory efficiency.
- Regression testing should compare agent behavior, not just final answers. Changes to models, prompts, tools, or orchestration can introduce execution regressions even when pass rates remain stable.
- Evaluation should evolve with the agent. Production failures and newly discovered edge cases should feed back into evaluation scenarios and regression suites.
