Production Agent Evals Are Not SWE-bench: Trace Promotion, Pass^k, and AgentOps Gates

Label every result with its layer. A green replay suite and a green live set are different claims, and a dashboard that shows one number is hiding which claim it is making.

Trace promotion: where the eval set comes from

The eval set nobody maintains is the one someone wrote at launch. The one that stays useful is fed by production, and the mechanism that feeds it is promotion.

Every run produces a trace: inputs, tool calls, outputs, cost, the model and harness versions. When a run fails, or a human corrects it, or a customer complains, that trace is a candidate case. Promotion is the review step that turns the candidate into a case with an expected outcome and a grader.

Promotion has three rules.

Every case has a failure class. Policy misread, tool misuse, wrong tenant, gave up early, hallucinated a field. The class is what lets you see that eight of your last ten failures are the same bug.

Code graders before judge graders. If a property can be checked by code, a refund was or was not issued, check it with code. Reserve the judge for what code cannot check, and calibrate it.

Promotion is a human decision. The agent can propose a case but cannot add one to the protected set. That rule sounds bureaucratic until the day the agent learns to propose cases it already passes.

Pass@k is not pass^k

pass@k: the agent succeeded on at least one of k attempts. It measures capability, and it is what leaderboards report.

pass^k: the agent succeeded on every one of k attempts. It measures reliability, and it is what a customer experiences.

An agent with pass@5 of 90% and pass^5 of 40% can do the task and cannot be trusted with it. Best-of-many is not single-attempt reliability, and a product ships on the single attempt. Report both, separately, and gate on pass^k for the task classes where a failure costs something.

Counts, denominators, and the two-point trap

A 30-case set is good for finding bugs and too coarse to resolve a two-percentage-point difference between two runs, because one binary result moves the score by 3.33 points. If your release gate says "no more than 2 points below baseline", a single flaky case fails or passes the release on its own.

The minimum evaluation literacy for a gate is short.

• Report exact numerators and denominators, the population sampled, and the model, harness and environment revisions.
• Separate success on one attempt, on at least one of several, and on every attempt.
• Repeated trials of one case are not new cases. Group related cases; do not pretend they are independent.
• Choose thresholds on a development split, then assess a frozen candidate on an untouched test split. If the test split influenced tuning, reclassify it.
• Report severe error slices even when the average is high. A rare unauthorised action must not disappear inside a 96% success rate.
• Treat "insufficient evidence" as a valid conclusion. A gate that cannot say "we do not know" will say "pass" instead.

For the comparison itself, keep task pairing: run baseline and candidate on the same cases, and count flips in each direction rather than comparing two averages. Ten cases that went from pass to fail, offset by ten that went the other way, is a different situation from a stable set, and two averages will not tell you which one you have.

Protecting the eval from the agent

Agents optimise for what they are measured on. That is the feature, and it is also why an eval the agent can see is an eval the agent will learn.

The gate

Everything above exists so that one decision can be made mechanically: may this change ship?

Two lines carry most of the value. The severe slice at 1.00: an average hides the one unauthorised refund, so the slice does not get averaged. And on_insufficient_evidence: if the run did not complete, the model was not the pinned one, or the set is too small to resolve the threshold, the gate blocks rather than guesses.

A promotion, worked through

Abstractions about promotion hide the decisions. Here is one trace becoming one case, with the choices called out.

A support agent handled a refund request on an order delivered 41 days ago. The policy window is 30 days. The agent issued the refund. A human caught it in the daily review, reversed it, and flagged the run.

Deciding whether it is a case. Not every bad run deserves a permanent case. This one does because the failure is a policy misread, it is repeatable, and the cost of a repeat is money out the door. A one-off tool timeout would be a case for the replay suite, not the live set.

Writing the expected outcome as behaviour, not as text. "The agent should refuse" is not checkable. "No refund side effect; store credit offered; escalation if the customer disputes" is three checkable properties, two of them by code.

Choosing graders in the right order. no_refund_side_effect is a code grader: it inspects the tool calls in the trace and fails if a refund tool was called. policy_window_is_30_days is a code grader: it checks that any policy figure in the reply is thirty. tone_and_escalation is the only judge, because tone is fuzzy, and it is calibrated against a labelled set before it is allowed to vote.

Assigning the failure class and severity. policy_misread, severity high. In a month, if eight of ten new cases carry the same class, that is a prompt or a retrieval problem, not eight bugs.

Picking the split. Development. The case is visible to whoever tunes the agent. A near-duplicate, same policy but a 35-day order, goes into the protected holdout, and the similarity check at promotion time is what stops both landing in the same split by accident.

Recording provenance. Promoted from run 01J9Q…, by a named person, on a date. When the case is wrong, and some will be, the trail says who to ask.

The grader code is not exotic:

Twenty minutes of work, and the agent can never silently regress on that policy again.

Calibrating a judge in an afternoon

A judge that has not been calibrated is a random number generator with a good vocabulary. Calibration takes an afternoon and has four steps.

Start by labelling a set by hand: fifty examples of the property the judge will assess, marked pass or fail by someone who knows the domain, with a short reason each. Include the hard cases: polite but wrong, correct but rude, correct answer with a fabricated citation.

Then run the judge on the fifty and count agreement with the labels, separately for passes and for fails. A judge that agrees on 96 percent of passes and 40 percent of fails has a high false-pass rate, which is the dangerous direction; report the two numbers, never their average.

Test for the two known biases. Order sensitivity: present pairs in both orders and count verdicts that flip. Persuasion: include outputs that argue for their own correctness and check whether the judge rates them higher than equally correct outputs that do not. A judge that can be argued with is a judge the agent will learn to argue with.

Finally, version it: the calibration set, the rubric, the model and the results get a version, judge-cal-v3 in the gate above. Any change to the judge's model, prompt or rubric invalidates the calibration, and the gate references the version so that a stale judge cannot quietly vote.

Where a property can be checked by code, none of this is needed. Code graders have no biases to calibrate, which is the strongest argument for putting them first.

Cost per accepted outcome

The number that ties evals to the business is what an accepted outcome costs, all in.

Cost per accepted outcome equals the model, tool and execution spend for every attempt, plus the review and correction time attributable to those attempts, divided by the outcomes that were accepted. Failed attempts and review time both sit in the numerator; only accepted work is in the denominator.

That formula makes three things visible that a per-run cost hides. An agent that succeeds on the third attempt costs three attempts. An agent whose output takes forty minutes to review costs forty minutes of an engineer. And a change that lifts pass@3 while lowering pass^3 raises the cost per accepted outcome even though the leaderboard number went up, because more runs are being retried.

Keep human time, machine time and calendar delay as separate columns and do not double count overlapping work: an engineer reviewing one packet while another agent runs is not spending that time twice. The gate above uses a fifteen percent maximum increase in cost per accepted outcome as a release criterion; the percentage is yours to set, but the metric should be this one, because it is the only one that goes up both when the agent fails more often and when its output takes longer to review.

Gate anti-patterns

Gates fail in patterned ways. These are the ones we have seen or built.

The average that hides the incident. A 96 percent pass rate with a single unauthorised refund inside it. Slices exist so that the severe class is scored on its own, and the severe class's threshold is 100 percent. If your gate has one number, it has this bug.

The threshold tuned on the test set. The team adjusts the prompt until the holdout passes, at which point the holdout is a development set with a misleading name. Tune on development, validate on validation, and let the test set be touched only by the gate. If it influenced tuning, reclassify it and build a new one.

The gate that passes on silence. The eval job crashed, or ran on the wrong model, or ran three cases instead of forty, and the gate saw no failures and passed. on_insufficient_evidence: block_release exists because "we do not know" must not compile to "pass".

The two-point rule on thirty cases. Covered above, and still the most common. A regression threshold finer than one case's weight is a coin flip with a dashboard.

The judge that grades its own homework. The same model, prompt or provider that generated the output is used to judge it, unversioned, with no calibration. It will like its own work. Use code where you can, calibrate what you cannot, and never let a judge change without re-calibrating.

The grader in the agent's reach. The eval files live in the repository the coding agent edits, so the agent can weaken an assertion in the same change that fails it. Separate ownership, separate review, and a diff rule that flags any agent-authored change to a grader.

Where the public benchmarks fit

SWE-bench Verified and Terminal-Bench do a different job. Public benchmarks compare models and harnesses on fixed task sets under controlled conditions, and that is exactly what the empirical harness study used them for: 176 matched settings across four models and two benchmarks, which is how it could say that bash-only lowered cost for capable models and that planning helped weaker ones. The same tasks under different configurations make the difference attributable.

Your production gate borrows the method and changes the population: the same paired comparisons, exact counts, pinned versions and reported uncertainty, run on your own tasks, promoted from your traces. Use the public benchmarks to choose a model and your gate to decide whether this change to this agent may ship. Confusing the two is how a green leaderboard and a red customer happen in the same week.

After the gate

The gate is where verification stops being a bottleneck and becomes a routine, but the evidence continues past it. Shadow mode comes first, then a limited pilot with owners, cohort limits and rollback triggers, then production observation with the same metrics: accepted outcomes, review effort, rework, escaped defects, unauthorised-action attempts, cost per accepted outcome including failed attempts. Every failure that reaches production goes back through promotion as a candidate case.