LLM evals in CI checklist
The failure mode of evals-in-CI is not missing regressions - it is gates so slow, flaky, or expensive that the team deletes them within a month. The design that survives is tiered and statistical, and the practitioner guides converge on the same shape: a small curated smoke suite on every relevant PR finishing in minutes - Confident AI's DeepEval guidance starts at 25-50 human-labeled cases and warns against scaling past 100 until your criteria stabilize - with the full suite run nightly against main (FutureAGI's 2026 GitHub Actions workflow is the clearest statement: nightly full rubric sweeps on the versioned dataset, delta-gated against a trailing 7-day baseline). Because LLM outputs are nondeterministic, single-run pass/fail is structurally flaky - run multiple trials per case (3-5 is the common working number), gate on mean scores or pass rates with variance as its own signal, and use baseline-relative thresholds with a tolerance band rather than absolute cutoffs. The tooling is commodity and framework-neutral: the open-source promptfoo GitHub Action runs a before/after comparison when watched prompt files change and posts results to the PR (promptfoo.dev docs), DeepEval turns metric thresholds into ordinary pytest assertions, and Braintrust's eval-action runs braintrust eval and posts the experiment summary as a PR comment - its README's full example shows gating the trigger with a paths filter on evals/**. This checklist is the wiring order.
AUDIENCE
Engineering teams adding evaluation gates to CI so prompt and model changes cannot silently regress an LLM feature - without turning every PR into a 20-minute, $10 lottery
OUTCOME
Eval gates that block real regressions, tolerate LLM nondeterminism without flaking, run in minutes on PRs, and cost pennies instead of burning the API budget on every push
SECTION 01
4 CHECKS
Gate Design and Triggers
Layer deterministic checks first; the LLM judge is the second tier, never the first
Schema validity, required fields, regex/exact-match assertions, refusal detection, and length bounds are free, instant, and zero-variance - Latitude's CI/CD guide explicitly orders pipelines deterministic-first, LLM-judge-second. Most shipped regressions (broken JSON, missing citation, wrong language) are catchable by code. Spending judge tokens to discover invalid JSON is the most common cost mistake in eval CI.
Trigger evals on path filters, not on every push
Run the eval workflow only when prompts, eval configs, or LLM-touching code change - Braintrust's eval-action README shows gating the workflow on paths: evals/**, and the promptfoo action is built around watched prompt files, running its comparison only on PRs that modify a prompt. An eval suite that runs on README typos trains the team to ignore it and burns budget for nothing.
Decide block vs warn per tier: hard-block only on fast, high-confidence checks
The practitioner consensus across the CI/CD eval guides: deterministic checks and the small high-signal smoke suite hard-block the merge; broad judge-scored suites report and warn, with humans reviewing regressions. Hard-blocking on a noisy judge score guarantees flaky merges, and the documented team response to flaky gates is bypassing them - which deletes the gate's value silently.
Provide an explicit, audited override path
There will be a legitimate reason to merge past a failing eval (the eval itself is wrong, an urgent fix regresses an edge case knowingly). Make the override a visible mechanism - a label plus a required second approval - rather than letting engineers discover force-merge. Unofficial bypasses become the default path within weeks; an audited override keeps the gate honest.
SECTION 02
4 CHECKS
Tiering and Runtime Budget
Keep the PR smoke suite curated and small - under 100 cases and under 10 minutes
Confident AI's DeepEval guidance recommends starting with 25-50 human-labeled test cases and not scaling beyond 100 until your evaluation criteria have stabilized - a suite that size finishes in minutes on a PR. Curate for coverage per token: your top intents, known past regressions, and the nastiest edge cases - not a random sample. Every case in the smoke tier should have blocked at least one hypothetical bad merge.
Run the full suite nightly or on merge to main, off the PR critical path
FutureAGI's 2026 CI workflow runs the full rubric sweep nightly against main on the versioned dataset - full datasets, multiple trials, slower judge rubrics - where a 45-minute runtime and a bigger token bill are acceptable. Nightly failures open an issue against the merge window instead of blocking a specific PR, which is the right granularity for regressions only visible at scale.
Cache eval results and parallelize trials
DeepEval's CI docs recommend caching so unchanged prompt-case pairs are not re-scored, plus parallel execution to keep wall-clock down; promptfoo similarly caches provider responses between runs. Without caching, every rebase re-buys the entire suite at API prices - with it, a typical PR re-scores only the cases its diff touches.
Track eval spend per run and alert on it
An eval suite is metered API usage that scales with PR volume, trials, and dataset growth - three numbers that all drift upward. Tag the eval API key separately from production and put a weekly spend alert on it: eval suites that quietly reach hundreds of dollars a month get deleted in the next cost review, taking the regression protection with them.
SECTION 03
4 CHECKS
Nondeterminism and Thresholds
Run 3-5 trials per case and gate on the aggregate, never a single run
Single-run pass/fail on a stochastic system is a coin flip wearing a checkmark. Run 3-5 trials per case, gate on the mean score or pass rate, and treat high variance across trials as its own reliability signal worth failing on. A case that passes 3 of 5 runs is telling you something a single green run would have hidden.
Gate against a baseline with a tolerance band, not an absolute threshold
Absolute thresholds ('score must exceed 0.8') break the day you change the rubric or the judge model. The guides recommend baseline-relative gating - compare the PR's scores against current main or a trailing baseline and fail on regression beyond a tolerance band; FutureAGI's 2026 guide goes further with a trailing 7-day baseline and significance tests (Welch's t-test for continuous rubrics, two-proportion z-test for binary). Relative gates survive rubric evolution; absolute gates rot.
Pin judge configuration: model version and temperature 0
If a judge model silently upgrades between the baseline run and the PR run, the delta measures the judge change, not your change. Pin the judge model version in config and score at temperature 0 for reproducibility (Latitude's guide makes the temperature-0 recommendation explicit). Judge upgrades are fine - as a deliberate PR that re-baselines everything at once.
Quarantine flaky cases instead of letting them poison the gate
The recommended flakiness protocol: re-run the failed subset once, and move cases that flip verdicts across reruns into a quarantine set excluded from blocking until fixed or relabeled. One ambiguous case that fails 20% of runs will block enough unrelated PRs to get the whole gate turned off - quarantine contains the damage to the case, not the pipeline.
SECTION 04
4 CHECKS
Tooling Wiring
Post eval results on the PR itself, not in a dashboard nobody opens
All three mainstream wirings do this out of the box: the promptfoo GitHub Action posts a before/after comparison comment with a link to the web viewer when prompt files change, and Braintrust's eval-action runs braintrust eval and comments the experiment summary with a link back to the platform. The PR comment is what reviewers actually see - an eval result that requires leaving the PR to find effectively does not exist in review.
If you live in pytest, wire thresholds as test assertions (DeepEval pattern)
DeepEval's CI model makes each metric threshold an ordinary pytest assertion, so the eval suite is just a test job - same runners, same reporting, same required-check plumbing, no bespoke gate infrastructure. The threshold lives in versioned test code, which means eval criteria changes go through code review like everything else.
Version eval datasets and prompts in the repo, and review dataset changes like code
The gate is only as trustworthy as its dataset, and the quiet way around a failing eval is editing the test case. Keep datasets and judge rubrics in-repo behind code review (with CODEOWNERS on the eval directory if stakes warrant), so weakening a gate is a visible diff with a reviewer - not a silent JSON edit in a web console.
Record eval config (model, judge, dataset version) in every result artifact
Score history is only interpretable if each run records what produced it. Emit the app model version, judge model version, dataset hash, and trial count alongside scores; when a trend line jumps, the first question is always 'what changed in the harness' and the artifact should answer it without archaeology.
Common mistakes
- 01Gating merges on a single stochastic run - the gate flakes, engineers stop trusting it, and it gets bypassed or deleted within weeks.
- 02Running the full eval suite on every push to every branch instead of path-filtering, then discovering the API bill in the monthly cost review.
- 03Using an LLM judge for properties a schema validator or regex checks for free, at zero variance.
- 04Absolute score thresholds that break silently when the rubric or judge model changes, instead of baseline-relative gates with tolerance bands.
- 05Letting the judge model float to 'latest', so score history measures provider upgrades instead of your changes.
- 06A smoke suite randomly sampled from production instead of curated for past regressions and edge cases - high token spend, low blocking power.
- 07Eval datasets editable outside code review, so the path of least resistance on a failing gate is weakening the test case.
- 08No override path, which guarantees the first urgent hotfix establishes force-merge as the unofficial process.