Verdict
Run both, but at different frequencies - and know the cost asymmetry before you wire the pipeline. Outcome-level evals are your CI gate: cheap, stable across valid alternative paths, and the only metric that maps to what users experience. Step-level trajectory evals are your debugging and regression microscope: when an outcome eval fails, only step-level scoring tells you which tool call, argument, or ordering decision broke - but running LLM-judged step-level metrics on every trajectory costs roughly 30-60x more judge spend per test case than outcome-only judging at mid-2026 pricing (our math below, computable from OpenAI's and Anthropic's published per-token prices). The workable pattern: deterministic trajectory matching (LangChain's open-source agentevals strict/unordered/subset/superset modes - no judge, no per-call cost) plus outcome judging in CI, with full LLM-judged step-level suites reserved for pre-release runs and failure triage. On framework support, the marketing does not match the docs: DeepEval, Arize Phoenix, LangSmith, and Galileo ship real trajectory-level scoring today, while OpenAI Evals remains outcome-shaped unless you build custom trace graders yourself.
The support matrix, verified against each project's own docs: DeepEval (Confident AI) ships agentic metrics that score inside the trajectory - ToolCorrectnessMetric (did the agent call the right tools), ArgumentCorrectnessMetric (with the right arguments), and TaskCompletionMetric (confident-ai.com docs). LangChain's agentevals package (github.com/langchain-ai/agentevals) ships four deterministic trajectory-match modes - strict (exact messages and tool calls in order), unordered (same tool calls, any order), subset (no tools beyond the reference), superset (at least the reference tools) - plus create_trajectory_llm_as_judge for rubric-based path grading, and LangSmith documents trajectory evals over the full execution path including tool calls and reasoning steps (docs.langchain.com/langsmith/trajectory-evals). Arize Phoenix is OpenTelemetry-native and documents agent trajectory evals over logged spans; Galileo documents agentic evaluations including tool selection quality. OpenAI Evals, by contrast, is benchmark-and-final-answer shaped - 2026 framework roundups (mlflow.org's top-5 comparison, futureagi.com's framework surveys) consistently place it in the outcome-oriented column, with trajectory scoring possible only via custom graders. The number nobody publishes is the cost delta, so we computed it from current list prices: assume a 15-step trajectory, 4 step-level metrics, and a judge call averaging ~3,000 input + 400 output tokens. Step-level judging is ~60 judge calls per trajectory: about $0.24 per trajectory with GPT-5.4 mini at $0.75/$4.50 per million input/output tokens (OpenAI's developer pricing page, July 2026) or about $0.30 with Claude Haiku 4.5 at $1/$5 (Anthropic pricing docs, July 2026). Outcome-only judging is 1-2 calls: about $0.004-$0.008 per trajectory on GPT-5.4 mini. A 200-case suite at 3 trials per case lands near $145 (GPT-5.4 mini) to $180 (Haiku 4.5) per full step-level run versus roughly $2.50-$5 outcome-only - a 30-60x multiplier that compounds with every retry, every model candidate, and every nightly run. That multiplier, not philosophy, is why granularity should be a tiered decision.
Decision Table
| Criterion | Edge | Explanation |
|---|---|---|
| Failure localization and debuggability | Step-level trajectory evals (tool-call correctness, argument correctness, step order, intermediate state) Best edge | An outcome eval tells you the agent failed; a step-level eval tells you it failed because step 7 called search_orders with a customer name instead of an ID. DeepEval's ToolCorrectnessMetric and ArgumentCorrectnessMetric exist precisely for this decomposition (confident-ai.com agent-eval docs), and Phoenix's OTel-native trajectory evals attach scores to individual spans so regressions bisect to a specific tool call. For multi-step agents, outcome-only suites produce the worst debugging loop in the discipline: a red flag on a 40-step run and no idea which step to fix. |
| Cost per eval run | Outcome-level evals (score only the final answer or task completion) Best edge | The multiplier is roughly the number of judged steps times the number of step-level metrics. Our worked example - 15 steps, 4 metrics, ~3,000-token judge inputs - costs ~$0.24 per trajectory with GPT-5.4 mini ($0.75/$4.50 per million tokens, OpenAI developer pricing) and ~$0.30 with Claude Haiku 4.5 ($1/$5, Anthropic pricing docs), versus ~$0.004-$0.008 for one or two outcome judgments. At 200 cases and 3 trials that is ~$145-$180 per step-level run versus $2.50-$5 outcome-only. Teams running evals on every PR feel this immediately: the 30-60x multiplier is the difference between evals-in-CI and evals-once-a-week. |
| Robustness to valid alternative paths | Outcome-level evals (score only the final answer or task completion) Best edge | Agents legitimately solve the same task via different tool sequences, and step-level reference matching punishes that. agentevals' own README concedes this by shipping four progressively looser match modes - strict ordering is the exception, not the default recommendation, because exact-sequence matching flags correct runs as failures whenever the model finds another valid route. Outcome evals are indifferent to the route by construction. If your task has many valid trajectories (research, debugging, open-ended synthesis), a strict step-level suite generates false regressions on every model upgrade; score the outcome, and use subset/superset matching only to ban specific known-bad detours. |
| Framework support out of the box | Step-level trajectory evals (tool-call correctness, argument correctness, step order, intermediate state) Best edge | The step-level side is better tooled than its reputation suggests: DeepEval (ToolCorrectness, ArgumentCorrectness, TaskCompletion metrics), agentevals (four deterministic match modes plus LLM-as-judge trajectory grading), LangSmith (trajectory evaluators over the full run), Phoenix (span-level trajectory evals on OpenTelemetry traces), and Galileo (agentic metrics including tool selection quality) all document trajectory scoring today. The outcome-only camp needs no special tooling - any eval harness scores a final answer - but the notable laggard is OpenAI Evals, which 2026 framework surveys (mlflow.org, futureagi.com) consistently classify as outcome-shaped with trajectory support only via custom graders. If your stack is OpenAI Evals and you want step-level scoring, you are building it yourself. |
| Determinism and reproducibility of scores | Tie | A third approach beats both LLM-judged variants here: deterministic trajectory matching. agentevals' strict/unordered/subset/superset modes compare tool-call sequences against references with zero judge calls - same input, same score, every run, at no marginal cost. LLM-judged scoring, whether of steps or outcomes, inherits judge variance: rubric-based trajectory judges can flip scores across runs and drift across judge-model versions, which is poison for a CI gate that must not flake. The defensible stack: deterministic matchers and programmatic checks (schema validation, state assertions) as the reproducible gate, LLM judges - at either granularity - as advisory signal with multiple trials and score averaging. |
| Alignment with what users actually experience | Outcome-level evals (score only the final answer or task completion) Best edge | Users experience outcomes; nobody ships a trajectory. A suite can score 95% on tool-call correctness while the product fails - right calls, right order, wrong final synthesis - and step-level metrics reward process conformity even when the process is not the point. Outcome evals are also the only granularity comparable across architectures: swap a ReAct loop for a plan-and-execute graph and every step-level reference breaks, while task-success rates remain comparable. Report outcome metrics as the headline; treat step metrics as diagnostics, never as the success criterion. |
Choose Step-level trajectory evals (tool-call correctness, argument correctness, step order, intermediate state) if...
- Debugging and regression triage on multi-step agents, where 'which step broke' is the question that matters.
- High-stakes tool use (payments, destructive operations, compliance actions) where a bad intermediate call is itself the failure, regardless of final outcome.
- Teams with eval infrastructure already on DeepEval, LangSmith, Phoenix, or Galileo - the step-level metrics are built and documented; the marginal cost is judge spend, not engineering.
- Model-upgrade bake-offs where you need to know whether a new model wins by better routing or just better recovery from the same mistakes.
Choose Outcome-level evals (score only the final answer or task completion) if...
- CI gates and per-PR checks, where cost, speed, and zero flake dominate - outcome checks plus deterministic matchers keep the gate under pennies per run.
- Open-ended tasks with many valid trajectories, where step references would punish legitimate solutions.
- Cross-architecture comparisons (loop vs graph vs multi-agent) where trajectory references cannot transfer but task-success rates can.
- Early-stage agents: before the happy path is stable, step-level references churn weekly - outcome evals give a stable north star while the trajectory design settles.
Decision rules
- 01CI gate: deterministic trajectory matching (agentevals subset/superset modes) plus programmatic outcome checks. Zero judge cost, zero flake; save LLM judges for what code cannot check.
- 02Nightly / pre-release: full step-level LLM-judged suite. Budget it explicitly - at current pricing (GPT-5.4 mini $0.75/$4.50, Haiku 4.5 $1/$5 per million tokens) a 200-case, 15-step, 4-metric run costs roughly $145-$180 per pass, ~30-60x the outcome-only equivalent.
- 03On any outcome-eval failure, auto-trigger step-level scoring for that trajectory only - you get localization when you need it without paying the multiplier on every green run.
- 04If your tasks admit many valid solution paths, never gate on strict step matching - use outcome evals plus subset matching to ban known-bad tool detours; strict mode will false-fail every model upgrade.
- 05If you are on OpenAI Evals and need trajectory granularity, plan for custom grader work or add a purpose-built layer (DeepEval, agentevals, Phoenix) - the 2026 framework surveys agree it does not ship step-level trajectory scoring out of the box.
- 06Cap judge input size: step-level cost scales with tokens-per-judge-call as much as call count. Feed the judge the step plus a distilled state summary, not the full transcript per step, or the 3,000-token assumption in our math quietly becomes 30,000.
Migration Notes
- Moving from outcome-only to tiered evals: start by logging full trajectories (OpenTelemetry spans via Phoenix, or LangSmith runs) without scoring them - you cannot add step-level evals later if today's runs only persist final answers.
- Build step references from your best passing runs, not from spec: record a known-good trajectory, prune incidental calls, and use it as the subset/superset reference rather than hand-authoring ideal paths.
- Recompute the cost model when judge prices move - the 30-60x multiplier is arithmetic on July 2026 list prices (GPT-5.4 mini $0.75/$4.50; Haiku 4.5 $1/$5), and a judge-price change or a cheaper distilled judge shifts where the tiering line belongs. OpenAI has already delisted older mini models once this cycle; re-check the pricing page before trusting any cached figure.