In this article, we will take an evaluation model designed for prompt -> response systems and stretch it over software that reasons, calls tools, mutates state, writes memory, retries failures, and operates for dozens or hundreds of steps.

As you will quickly realize, this is an infrastructure problem that should compound as you operate more agents in production.

Let’s dive into it.

Agent eval is a system.

A single-turn LLM eval is compact: an input, an output, a reference or rubric, and a score.

You can store the whole experiment in a CSV with four columns: prompt, response, expected, and score.

Agentic systems add several dimensions at once, and each one changes what you have to store:

• Unit of work: prompt and response -> episode or rollout
• Main evidence: final text -> output, trace, artifacts, and state delta
• Failure location: usually the answer -> any step in the trajectory
• Environment: mostly static context -> mutable files, databases, APIs, tools, and memory
• Reproducibility: prompt, model, and seed -> model, harness, state, tools, clocks, packages, and permissions
• Safety failure: bad text -> bad action
• Cost: tokens -> tokens, tool calls, containers, latency, and human review

The important shift is the unit of evaluation.

You are no longer scoring a response but evaluating a bounded attempt to change a world.

The attempt begins with an initial state where agent receives an instruction, operates through a harness, invokes tools, observes results, and eventually hits a terminal condition.

The run leaves behind several artifacts:

• A final answer or generated deliverable
• A trajectory of messages, tool calls, and observations
• A final environment state
• A delta between the initial and final state
• Cost, latency, retry, and error data
• Possibly modified memory that will influence later runs

Each attempt can named as a trial, episode or rollout.

The labels differ slightly, but the implication is the same.

Your minimum useful record looks more like this:

A score without this chain of evidence is difficult to reproduce and even harder to debug.

The five surfaces of an agent eval

There are five surfaces:

Output // Trace // Memory // Environment // Mechanistic Interpretability

Most product teams using hosted model APIs cannot inspect model activations, circuits, or internal features, so mechanistic interpretability is usually outside the practical stack.

That leaves four surfaces every agent team should instrument.

1. Output

The output is still important. Did the patch pass tests? Did the report reconcile with source data? Did the generated UI match the acceptance criteria? Did the agent actually answer the user’s question?

Output graders fall into familiar categories:

• Deterministic checks for code, math, schemas, files, and database rows
• Rubric-based checks for quality, completeness, or style
• Human review for high-value or ambiguous cases
• LLM judges when deterministic verification is insufficient

Start with deterministic checks whenever possible. They are cheaper, easier to debug, and less likely to drift when you change the judge model.

2. Trace

The trace explains how the output happened. For each step, capture at least:

• Tool name
• Arguments or an arguments hash
• Observation or an observation hash
• Start time and latency
• Cost and token usage
• Permission boundary
• Error and retry information
• State-delta pointer
• Model, prompt, harness, and tool versions

This lets you ask questions that final-answer graders cannot answer:

• Did the agent inspect the relevant files before editing?
• Did it call a privileged tool unnecessarily?
• Did it retry the same failed command until the budget expired?
• Did it fabricate an answer after a tool returned no results?
• Did it validate its change?
• Did the action taken match the explanation given to the user?

Do not grade traces against one “golden” sequence unless the workflow genuinely requires a fixed protocol. A capable agent may find a valid path you did not anticipate. The stronger approach is to define process invariants:

• No secrets written to logs
• No changes outside the allowed repository path
• No destructive database operations
• At least one validation step before completion
• No unsupported claim after an empty retrieval result
• No repeated identical tool call beyond a retry limit

These rules constrain unsafe or wasteful behavior without forcing the agent to imitate a single trajectory.

3. Memory

Memory is part of the executable system. This includes conversation history, summaries, scratchpads, project instructions, AGENTS.md, CLAUDE.md, skills, vector-store entries, cached retrievals, and long-term user preferences.

A bad memory write can outlive the failed trial that created it. The agent may incorrectly summarize a temporary workaround as a permanent architecture rule. It may store one user’s preference as a global default. It may persist stale credentials, incorrect assumptions, or prompt-injection content.

So memory needs its own evaluation questions:

• What did the agent read?
• What did it write?
• Was the write justified?
• Did it improve future trials?
• Can the same task run with clean, stale, disabled, and polluted memory?

If disabling memory barely changes performance, your expensive memory layer may be theater. If stale memory causes a major collapse, you have found a production risk that an output-only eval would miss.

4. Environment

Most useful agents are state-transition systems. They modify repositories, databases, browser sessions, calendars, tickets, spreadsheets, deployments, and filesystems. The environment is not incidental context. It is part of the answer.

A serious evaluator therefore needs snapshots and diffs:

initial_state --agent actions--> final_state
      \_____________________________/
               state delta

For a coding agent, the delta may include:

• Files created, modified, or deleted
• Git diff and repository status
• Test results
• Build artifacts
• Package changes
• Environment variables touched
• Commands executed with elevated privileges

For a support agent, it may include database rows, refund status, ticket fields, and outbound messages. For a browser agent, it may include cookies, local storage, visited URLs, downloaded files, and DOM state.

The environment must also be restorable. If two trials share mutable state, the second trial is no longer measuring the same task. It is measuring the first trial plus whatever state the first agent left behind. That is an uncontrolled experiment.

The architecture: control plane versus data plane

A useful way to organize the stack is to separate what decides the experiment from what executes it, Control Plane vs Data Plane.

Control plane

The control plane contains:

• Task suites and task distributions
• Success contracts
• Graders, rubrics, and human-review policy
• Model, prompt, tool, and harness configurations
• Perturbation and ablation definitions
• Aggregation, confidence intervals, and slices
• Regression tracking
• Release gates

Data plane

The data plane contains:

• Model endpoint
• Agent scaffold or harness
• Runtime and sandbox
• Tools and permissions
• Initial state and checkpoints
• Memory
• Trace capture
• Outputs, logs, and artifacts
• State deltas

Agents are coupled systems.

A score can move because the model changed, the system prompt changed, a tool schema changed, a package upgraded, a browser profile changed, memory was populated differently, or a verifier became more lenient.

If all those dimensions are bundled together, the number may move while your understanding remains flat.

A clean experiment changes one factor and holds the rest fixed which gives you two especially useful test types.

Perturbation tests

Keep the task constant but alter the path available to the agent:

• Randomly fail a tool
• Increase tool latency
• Reduce the turn budget
• Revoke a permission
• Add stale documentation
• Return an empty retrieval result
• Disable network access halfway through the run

This tests whether the agent is robust or merely successful on the happy path.

Ablation tests

Remove one system component:

• Long-term memory
• Browser access
• Semantic search
• Sub-agents
• A planning prompt
• A repository map
• A specialized tool

If you measure which task slices change, that will tell you whether a component earns its operational cost.

Quick start with Harbor

The ideas above become much easier to understand when you implement one task.

Harbor, created by the Terminal-Bench team, is a useful reference implementation because it treats an eval as a task, an isolated environment, an agent run, a verifier, and a stored trajectory.

It is also the official harness for Terminal-Bench 2.0.

The setup below follows Harbor’s current repository and task format.

Prerequisites

You need:

• Docker installed and running for local container execution
• uv or a Python environment
• An API key for the model provider you plan to test

Install Harbor:

uv tool install harbor

Verify the CLI and inspect available integrations

harbor --help harbor dataset list

Before testing a real agent, run an oracle solution against Terminal-Bench. An oracle executes the task’s known-good solution and verifies that the environment and grader are valid.

harbor run \
--dataset terminal-bench@2.0 \
--agent oracle \
--n-concurrent 2

This step is more important than it looks. If the oracle fails, you do not have an agent failure. You have a broken task, environment, solution, or verifier.

Run a real coding agent

Harbor includes adapters for several coding agents. The exact set evolves, so check harbor run --help on your installed version. A typical run looks like this:

export ANTHROPIC_API_KEY="your-key"
harbor run \
--dataset terminal-bench@2.0 \
--agent claude-code \
--model anthropic/claude-opus-4-1 \
--n-concurrent 4

The command creates a job directory containing job configuration, trial-level results, agent logs, trajectories, verifier output, and artifacts. Launch the local viewer:

harbor view jobs

Now you can inspect individual trials instead of staring at an average. That is where agent eval work actually begins.

Build your first stateful task

The official Harbor tutorial uses a small SSH-key task. For a developer team, a repository repair task is a better mental model. Create the scaffold:

harbor task init fix-order-total

The generated directory follows this shape:

fix-order-total/ ├── instruction.md ├── task.toml ├── environment/ │ ├── Dockerfile │ └── order_total.py ├── solution/ │ └── solve.sh └── tests/ ├── test.sh └── test_order_total.py

Define the task contract

instruction.md is what the agent sees.

Fix order total calculation

Repair /app/order_total.py. The function must apply the percentage discount to the subtotal before tax, then add tax to the discounted amount. Preserve the public function signature. Do not add network dependencies.

Notice what is missing: an exact sequence of commands. We define the desired outcome and important boundaries. We do not require the agent to follow one golden path.

Configure the environment

task.toml controls timeouts, resources, and metadata.

schema_version = "1.3"
[task]
name = "acme/fix-order-total"
description = "Repair a deterministic pricing function"
[metadata]
category = "software-engineering"
keywords = ["python", "debugging", "stateful-eval"]
[agent]
timeout_sec = 300
[verifier]
timeout_sec = 120
[environment]
workdir = "/app"
build_timeout_sec = 600
cpus = 1
memory_mb = 1024

Create the broken implementation in environment/order_total.py:

def order_total(subtotal: float, discount: float, tax: float) -> float:
  """Return the final order total.
  discount and tax are decimal rates, such as 0.10 for 10%.
  """
  discounted = subtotal - discount  # Bug: subtracts the rate as currency
  return round(discounted + subtotal * tax, 2)  # Bug: taxes full subtotal

Create the container in environment/Dockerfile:

FROM python:3.12-slim WORKDIR /app COPY order_total.py /app/order_total.py

This is the world the agent receives. Every trial should start from a fresh copy of it.

Add an oracle solution

The oracle proves the task is solvable. Here is solution/solve.sh:

#!/usr/bin/env bash
set -euo pipefail
cat > /app/order_total.py <<'PY'
def order_total(subtotal: float, discount: float, tax: float) -> float:
  discounted = subtotal * (1 - discount)
  return round(discounted * (1 + tax), 2)
PY

Make it executable:

chmod +x fix-order-total/solution/solve.sh

Write a deterministic verifier

tests/test_order_total.py:

import importlib.util
import unittest
from pathlib import Path
MODULE_PATH = Path("/app/order_total.py")

def load_function():
  spec = importlib.util.spec_from_file_location("order_total", MODULE_PATH)
  module = importlib.util.module_from_spec(spec)
  assert spec.loader is not None
  spec.loader.exec_module(module)
  return module.order_total

class OrderTotalTests(unittest.TestCase):
  def test_discount_is_applied_before_tax(self):
      order_total = load_function()
      self.assertEqual(order_total(100.0, 0.10, 0.20), 108.0)
  def test_zero_discount(self):
      order_total = load_function()
      self.assertEqual(order_total(80.0, 0.0, 0.05), 84.0)
  def test_signature_stays_compatible(self):
      order_total = load_function()
      self.assertEqual(order_total.__code__.co_argcount, 3)

if __name__ == "__main__":
  unittest.main()

tests/test.sh converts the verifier result into Harbor’s reward file:

#!/usr/bin/env bash
set -euo pipefail
mkdir -p /logs/verifier
if python /tests/test_order_total.py; then
echo 1 > /logs/verifier/reward.txt
else
echo 0 > /logs/verifier/reward.txt
fi

This grader checks the outcome rather than the agent’s prose. The agent may claim it fixed the bug. The reward is still zero unless the final filesystem state passes the tests.

Validate the task before benchmarking models

Run the oracle:

harbor run -p fix-order-total -a oracle

Then run a real agent:

harbor run \
-p fix-order-total \
-a claude-code \
-m anthropic/claude-opus-4-1

Open the viewer:

harbor view jobs

Inspect three things:

1. Outcome: Did the tests pass?
2. Trace: Did the agent inspect the implementation, edit it, and validate it?
3. State delta: Did it modify only the intended file?

That third check is easy to omit. A passing patch that also edits unrelated configuration is not equivalent to a minimal passing patch.

Add multiple reward dimensions

Binary pass/fail is a good starting point, not a complete destination. Harbor’s verifier can emit a JSON object instead of one scalar:

{
"correctness": 1.0,
"scope_control": 1.0,
"efficiency": 0.8
}

This lets you separate questions that would otherwise collapse into one ambiguous number. For a coding agent, a practical scorecard might be:

• Correctness: hidden tests pass
• Scope control: no unrelated files changed
• Safety: no secrets, destructive commands, or forbidden paths
• Efficiency: tool calls, tokens, wall time, repeated failures
• Process: validation happened before completion
• Quality: human or calibrated model review of maintainability

Do not let an LLM judge override a failed deterministic correctness check. A well-written explanation cannot compensate for broken code. Model-based graders are better used for dimensions that code cannot reliably capture, such as readability or whether a report is well structured.

Reliability is not Pass@1

Agents are nondeterministic.

A model can solve the same task once and fail it on the next attempt because it chooses different tools, takes a different branch, receives a different simulated-user response, or accumulates context differently.

This is why multiple trials per task should have distinction between Pass@K and the stricter Pass^K used in the τ-bench family.

If an agent has an 80% independent chance of success on one run:

• The chance it succeeds at least once in four attempts is 99.84%.
• The chance it succeeds all four times is only 40.96%.

A demo rewards the first number but aproduction workflow lives closer to the second.

For customer support, deployment automation, finance, or any workflow where users expect consistency, “it worked once” is almost meaningless. Run multiple attempts, report the distribution, and slice failures by task type and failure mode.

A useful release report contains more than an average:

• Success rate by task slice
• All-attempt reliability for critical tasks
• Cost and latency distributions
• Tool-error recovery rate
• Unsafe-action count
• Tasks that flipped from pass to fail
• Tasks that flipped from fail to pass
• Confidence intervals or uncertainty estimates

The question is not just, “Did the score go up?” It is:

Which component changed, which tasks flipped, and what evidence explains the flip?

Replay and branching turn failures into engineering data

Long-horizon failures are expensive to reproduce from the beginning. You want checkpoints that contain enough state to inspect or continue the rollout:

• Filesystem or database snapshot
• Conversation and tool observations
• Model and harness versions
• Prompt and configuration hash
• Memory state
• Package versions
• Clock and random-seed information where available
• Pending jobs or external fixtures

Then distinguish three operations:

• Resume: Continue the same attempt with the same configuration.
• Replay: Re-execute or inspect a known trajectory from known state.
• Branch: Start from a checkpoint while changing one factor.

Branching is where the eval stack becomes an experimentation system. From one failure checkpoint, run:

• Same state, new model: did the model upgrade fix the failure?
• Same model, new prompt: did the instruction change matter?
• Same task, memory disabled: is memory helping or poisoning the run?
• Same state, tool unavailable: does the agent recover safely?
• Same task, lower budget: are the extra steps buying quality?
• Same output, new grader: did the agent fail, or did the scorer fail?

This is far more actionable than rerunning the whole benchmark and comparing two aggregate percentages.

Keep tasks separate from the harness

One of the strongest design principles in Lee’s article is to decouple what you measure from how you run it. The task should define:

• Initial state
• Instruction
• Success contract
• Constraints
• Verifier

The harness should define:

• Model
• Tools
• Agent loop
• Prompting strategy
• Context management
• Memory
• Runtime
• Budgets

This separation lets you run the same task with:

• A plain model baseline
• A tool-using agent
• A new scaffold
• A different model
• Memory enabled or disabled
• Local or cloud sandboxes

OLMo Eval uses this same idea: a task can remain fixed while the harness changes between plain, tool-using, and scaffolded execution. Without this separation, a “model comparison” can quietly become a comparison of two completely different systems.

Production failures should become eval tasks

Offline evals and production monitoring should not be separate universes. When a production failure occurs:

1. Capture the trace and relevant state delta.
2. Redact secrets and user data.
3. Reconstruct a minimal initial state.
4. Write a deterministic success contract where possible.
5. Confirm the oracle or known-good behavior.
6. Add the case to the regression suite.
7. Tag it by failure mode and task slice.

This creates a compounding loop:

Production incident
    -> reproducible task
    -> regression suite
    -> release gate
    -> fewer repeated incidents

A benchmark that never absorbs production failures gradually stops representing the product. A benchmark that absorbs every failure without curation becomes noisy and expensive. The evaluation team’s job is to preserve representative, solvable, high-signal failures.

Common ways teams accumulate evaluation debt

The debt usually starts with reasonable shortcuts. A developer writes a notebook. Tasks live in a CSV. Judge prompts are copied into a shared document. The model points to whichever API alias is current. Tool schemas change without versioning. Production failures arrive as screenshots in Slack.

Everything still runs. Then the next model upgrade arrives. The score improves, but customers report regressions. The team cannot reproduce them because the evaluation used different memory, timeouts, permissions, package versions, prompts, browser state, and tool behavior.

Watch for these warning signs:

• Only the final response is stored
• Tasks share mutable databases or filesystems
• The oracle solution is missing
• Judge prompts are not versioned
• Tool schemas are not hashed or recorded
• Model aliases are used instead of fixed versions
• One attempt per task is treated as reliability
• Aggregate score is reported without slices
• Production incidents are not promoted into regression tasks
• The eval harness is materially different from production
• A model and harness change ship in the same experiment

At that point, the dashboard is measuring a moving target with incomplete instrumentation.

A minimum viable agent eval stack

You do not need a huge platform team to start. A small engineering team can build a credible first version with:

• Tasks: 10 to 20 realistic cases from actual workflows
• Environment: a fresh Docker container or isolated database fixture per trial
• Oracle: a known-good solution or verified reference outcome
• Verifiers: deterministic checks first, model or human review only where needed
• Traces: structured tool calls, observations, timing, cost, and errors
• State: initial snapshot, final snapshot, and diff
• Repetition: multiple attempts for important tasks
• Analysis: per-task results, slices, flips, cost, and failure labels
• Release gate: block on critical regressions and safety violations
• Feedback loop: promote representative production failures into the suite

Start small. Begin with roughly 10–20 manually curated tasks rather than collecting a massive dataset upfront. That is enough to catch obvious failures and expose weaknesses in the harness itself. The task suite should then grow from real failures, not from random prompt generation.