Production Agent Evals Are Not SWE-bench: Trace Promotion, Pass^k, and AgentOps Gates

Every agent team I talk to has an eval story, and it usually goes like this. They benchmarked the model on launch. The number was good. Then something changed, a prompt, a tool, a model alias, and quality regressed for a week before a customer noticed. When they went to check, the eval was still green, because the eval was measuring what the model could do on a public benchmark, not what the agent does on their traffic.

SWE-bench is a fine instrument for its purpose: comparing models on a fixed set of repository tasks. It is not an instrument for your agent, because your agent's failures come from your tools, your data, your users and your harness, none of which are in the benchmark. Anthropic's 2026 report names verification as the new bottleneck. This is what the bottleneck looks like from inside a team, and how to build the thing that clears it.

Four kinds of evidence, and what each one cannot tell you

Before building anything, name the layers. Teams conflate them, and the conflation is where the false confidence comes from.

Label every result with its layer. A green replay suite and a green live set are different claims, and a dashboard that shows one number is hiding which claim it is making.

Trace promotion: where the eval set comes from

The eval set nobody maintains is the one someone wrote at launch. The eval set that stays useful is the one fed by production. The mechanism is promotion.

Every run produces a trace: inputs, tool calls, outputs, cost, the model and harness versions. When a run fails, or a human corrects it, or a customer complains, that trace is a candidate case. Promotion is the review step that turns the candidate into a case with an expected outcome and a grader.

{
"case_id": "refund-policy-0412",
"promoted_from": "run_01J9Q…",
"promoted_on": "2026-10-01",
"promoted_by": "m.cakmak",
"split": "dev",
"task": {
  "input": "Customer asks for a refund on an order delivered 41 days ago.",
  "fixtures": ["tenant-acme", "order-88213"]
},
"expected": {
  "outcome": "no refund; offers store credit; escalates if customer disputes",
  "must_not": ["issue a refund", "quote a policy window other than 30 days"]
},
"graders": [
  { "type": "code", "name": "no_refund_side_effect" },
  { "type": "code", "name": "policy_window_is_30_days" },
  { "type": "judge", "name": "tone_and_escalation", "calibrated_on": "judge-cal-v3" }
],
"failure_class": "policy_misread",
"severity": "high"
}

Three rules keep promotion honest.

Every case has a failure class. Policy misread, tool misuse, wrong tenant, gave up early, hallucinated a field. The class is what lets you see that eight of your last ten failures are the same bug.

Code graders before judge graders. If a property can be checked by code, a refund was or was not issued, check it with code. Reserve the judge for the genuinely fuzzy property, and calibrate it.

Promotion is a human decision. The agent can propose a case. It cannot add one to the protected set. That rule sounds bureaucratic until the day the agent learns to propose cases it already passes.

Pass@k is not pass^k

Here is the distinction that changes how you read every eval report.

pass@k: the agent succeeded on at least one of k attempts. It measures capability. It is what leaderboards report.

pass^k: the agent succeeded on every one of k attempts. It measures reliability. It is what a customer experiences.

An agent with pass@5 of 90% and pass^5 of 40% can do the task and cannot be trusted with it. Best-of-many is not single-attempt reliability, and a product ships on the single attempt. Report both, separately, and gate on pass^k for the task classes where a failure costs something.

from statistics import mean

def pass_at_k(results):  # results: list of lists of bools, one inner list per case
  return mean(any(r) for r in results)

def pass_pow_k(results):
  return mean(all(r) for r in results)

# 3 attempts per case, 40 cases
runs = run_eval_set("refund-policy", attempts=3, model="…", harness="…")
print("pass@3 ", pass_at_k(runs))   # can it do it?
print("pass^3 ", pass_pow_k(runs))  # does it always do it?

Counts, denominators, and the two-point trap

A 30-case set is good for finding bugs. It is not good for resolving a two-percentage-point difference between two runs, because one binary result moves the score by 3.33 points. If your release gate says "no more than 2 points below baseline", a single flaky case fails or passes the release on its own.

The minimum evaluation literacy for a gate is short.

• Report exact numerators and denominators, the population sampled, and the model, harness and environment revisions.
• Separate success on one attempt, on at least one of several, and on every attempt.
• Repeated trials of one case are not new cases. Group related cases; do not pretend they are independent.
• Choose thresholds on a development split, then assess a frozen candidate on an untouched test split. If the test split influenced tuning, reclassify it.
• Report severe error slices even when the average is high. A rare unauthorised action must not disappear inside a 96% success rate.
• Treat "insufficient evidence" as a valid conclusion. A gate that cannot say "we do not know" will say "pass" instead.

For the comparison itself, keep task pairing: run baseline and candidate on the same cases, and count flips in each direction rather than comparing two averages. Ten cases that went from pass to fail, offset by ten that went the other way, is a different situation from a stable set, and two averages will not tell you which one you have.

Protecting the eval from the agent

Agents optimise for what they are measured on. That is the feature. It is also why an eval the agent can see is an eval the agent will learn.

The gate

Everything above exists so that one decision can be made mechanically: may this change ship?

# agentops/gates/refund-agent.yaml
gate: refund-agent-release
applies_to: [prompt, tools, policy, model, harness]   # any of these changing triggers the gate

eval_set: refund-policy@v14          # frozen; promotion creates v15, it does not edit v14
attempts_per_case: 3

require:
replay_contract_suite: pass                     # layer 1, always
pass_pow_k:
  min: 0.85
  slices:
    severity_high: 1.00                          # zero tolerance on the severe slice
pass_at_k:
  min: 0.95
flips_to_fail_vs_baseline: { max: 2, must_review: true }
cost_per_accepted_outcome: { max_increase_pct: 15 }
judge_calibration: judge-cal-v3                  # the judge must be the calibrated one

record:
- model_version_exact
- harness_version
- prompt_hash
- policy_version
- eval_set_version

on_fail: block_release
on_insufficient_evidence: block_release           # 'we do not know' is not 'pass'

Two lines carry most of the value. The severe slice at 1.00: an average hides the one unauthorised refund, so the slice does not get averaged. And on_insufficient_evidence: if the run did not complete, the model was not the pinned one, or the set is too small to resolve the threshold, the gate blocks rather than guesses.

A promotion, worked through

Abstractions about promotion hide the decisions. Here is one trace becoming one case, with the choices called out.

A support agent handled a refund request on an order delivered 41 days ago. The policy window is 30 days. The agent issued the refund. A human caught it in the daily review, reversed it, and flagged the run.

Deciding whether it is a case. Not every bad run deserves a permanent case. This one does because the failure is a policy misread, it is repeatable, and the cost of a repeat is money out the door. A one-off tool timeout would be a case for the replay suite, not the live set.

Writing the expected outcome as behaviour, not as text. "The agent should refuse" is not checkable. "No refund side effect; store credit offered; escalation if the customer disputes" is three checkable properties, two of them by code.

Choosing graders in the right order. no_refund_side_effect is a code grader: it inspects the tool calls in the trace and fails if a refund tool was called. policy_window_is_30_days is a code grader: it checks that any policy figure in the reply is thirty. tone_and_escalation is the only judge, because tone is fuzzy, and it is calibrated against a labelled set before it is allowed to vote.

Assigning the failure class and severity. policy_misread, severity high. In a month, if eight of ten new cases carry the same class, that is a prompt or a retrieval problem, not eight bugs.

Picking the split. Development. The case is visible to whoever tunes the agent. A near-duplicate, same policy but a 35-day order, goes into the protected holdout, and the similarity check at promotion time is what stops both landing in the same split by accident.

Recording provenance. Promoted from run 01J9Q…, by a named person, on a date. When the case is wrong, and some will be, the trail says who to ask.

The grader code is not exotic:

def no_refund_side_effect(trace) -> bool:
  return not any(call.tool == "issue_refund" for call in trace.tool_calls)

def policy_window_is_30_days(trace) -> bool:
  figures = re.findall(r"(\d+)[- ]day", trace.final_reply)
  return all(f == "30" for f in figures)

# Judges are called only for what code cannot check, and only after calibration.
def tone_and_escalation(trace, judge=CALIBRATED_JUDGE["judge-cal-v3"]) -> bool:
  return judge.passes(trace.final_reply, rubric="polite; offers store credit; escalates if disputed")

Twenty minutes of work, and the agent can never silently regress on that policy again.

Calibrating a judge in an afternoon

A judge that has not been calibrated is a random number generator with a good vocabulary. Calibration is not a research project; it is an afternoon, and it has four steps.

Label a set by hand. Fifty examples of the property the judge will assess, labelled pass or fail by someone who knows the domain, with a short reason each. Include the hard cases: polite but wrong, correct but rude, correct answer with a fabricated citation.

Measure agreement. Run the judge on the fifty and count agreement with the labels, separately for passes and for fails. A judge that agrees on 96 percent of passes and 40 percent of fails has a high false-pass rate, which is the dangerous direction; report the two numbers, never their average.

Test for the two known biases. Order sensitivity: present pairs in both orders and count verdicts that flip. Persuasion: include outputs that argue for their own correctness and check whether the judge rates them higher than equally correct outputs that do not. A judge that can be argued with is a judge the agent will learn to argue with.

Version it. The calibration set, the rubric, the model and the results get a version, judge-cal-v3 in the gate above. Any change to the judge's model, prompt or rubric invalidates the calibration, and the gate references the version so that a stale judge cannot quietly vote.

Where a property can be checked by code, none of this is needed. That is the strongest argument for code graders first: they do not have biases to calibrate.

Cost per accepted outcome

The number that ties evals to the business is not accuracy. It is what an accepted outcome costs, all in.

Cost per accepted outcome equals the model, tool and execution spend for every attempt, plus the review and correction time attributable to those attempts, divided by the outcomes that were actually accepted. The failed attempts are in the numerator. The review time is in the numerator. Only accepted work is in the denominator.

Three things that formula makes visible that a per-run cost does not. An agent that succeeds on the third attempt costs three attempts. An agent whose output takes forty minutes to review costs forty minutes of an engineer. And a change that lifts pass@3 while lowering pass^3 raises the cost per accepted outcome even though the leaderboard number went up, because more runs are being retried.

Keep human time, machine time and calendar delay as separate columns and do not double count overlapping work: an engineer reviewing one packet while another agent runs is not spending that time twice. The gate above uses a fifteen percent maximum increase in cost per accepted outcome as a release criterion; the percentage is yours to set, but the metric should be this one, because it is the only one that goes up when the agent gets worse in either of the two ways that matter.

Gate anti-patterns

Gates fail in patterned ways. These are the ones we have seen or built.

The average that hides the incident. A 96 percent pass rate with a single unauthorised refund inside it. Slices exist so that the severe class is scored on its own, and the severe class's threshold is 100 percent. If your gate has one number, it has this bug.

The threshold tuned on the test set. The team adjusts the prompt until the holdout passes, at which point the holdout is a development set with a misleading name. Tune on development, validate on validation, and let the test set be touched only by the gate. If it influenced tuning, reclassify it and build a new one.

The gate that passes on silence. The eval job crashed, or ran on the wrong model, or ran three cases instead of forty, and the gate saw no failures and passed. on_insufficient_evidence: block_release exists because "we do not know" must not compile to "pass".

The two-point rule on thirty cases. Covered above, and still the most common. A regression threshold finer than one case's weight is a coin flip with a dashboard.

The judge that grades its own homework. The same model, prompt or provider that generated the output is used to judge it, unversioned, with no calibration. It will like its own work. Use code where you can, calibrate what you cannot, and never let a judge change without re-calibrating.

The grader in the agent's reach. The eval files live in the repository the coding agent edits, so the agent can weaken an assertion in the same change that fails it. Separate ownership, separate review, and a diff rule that flags any agent-authored change to a grader.

Where the public benchmarks fit

None of this is an argument against SWE-bench Verified or Terminal-Bench. It is an argument about what they are for.

Public benchmarks compare models and harnesses on fixed task sets under controlled conditions. That is exactly what the empirical harness study used them for: 176 matched settings across four models and two benchmarks, which is how it could say that bash-only lowered cost for capable models and that planning helped weaker ones. Matched settings are the point; the same tasks under different configurations, so the difference is attributable.

Your production gate borrows the method and changes the population. Same discipline, paired comparisons, exact counts, versions pinned, uncertainty reported. Different tasks, yours, promoted from your traces. Use the public benchmarks to choose a model. Use your gate to decide whether this change to this agent may ship. Confusing the two is how a green leaderboard and a red customer happen in the same week.

After the gate

The gate is where verification stops being a bottleneck and becomes a routine. It is not the end of the evidence. Shadow mode, then a limited pilot with owners, cohort limits and rollback triggers, then production observation with the same metrics: accepted outcomes, review effort, rework, escaped defects, unauthorised-action attempts, cost per accepted outcome including failed attempts.

And every failure that reaches production goes back through promotion. That loop, trace to case to gate, is the whole system. SWE-bench never sees your traffic. Your eval set sees nothing else.