Agent observability production checklist
Datadog's 2026 State of AI Engineering report puts hard numbers on the problem: roughly 5% of AI model requests fail in production, and nearly 60% of those failures trace to capacity and rate limits - failures that happen mid-chain, invisible unless you instrument the individual model-call spans (Datadog report announcement, April 2026). And the error rate is only half the problem: a request that returns 200 with a wrong answer never shows up in it at all. Agent failures spread across seven layers (framework, model routing, memory, ingestion, tools, durable execution, observability - the taxonomy from our 7-failure-layers analysis), and this checklist walks the observability decisions layer by layer, including the honest buy-vs-build call: tracing infrastructure is a solved problem you should buy or adopt, while eval criteria and cost SLOs are your business logic and cannot be outsourced.
AUDIENCE
Engineering teams taking an agent from working demo to production service, and platform leads deciding which observability layers to buy versus build
OUTCOME
Every production agent run is traceable, evaluated, cost-attributed, and recoverable - before the first silent failure, not after
SECTION 01
4 CHECKS
Tracing coverage (buy this layer, don't build it)
Trace every agent step as a span - model calls, tool executions, retrievals, and subagent spawns - not just the top-level request
Silent failures live between spans: a tool call that returned plausible garbage looks fine at the request level. Datadog's 2026 data makes the case concretely - nearly 60% of production AI failures come from capacity and rate limits, and rate-limit errors alone hit ~8.4 million in a single month - errors that strike one call deep inside a chain and are invisible in top-level request metrics.
Adopt a hosted tracing platform before writing your own - know the real free-tier ceilings
This is the layer where buying wins outright - but read the billing units, not the headline numbers. LangSmith's free Developer plan includes 5,000 base traces/month, then $2.50 per 1,000 base traces (14-day retention) or $5.00 per 1,000 extended traces (400-day retention) - and one agent run with all its spans counts as one trace (langchain.com/pricing). Langfuse's free Hobby plan includes 50,000 units/month with 30-day data access, but a unit is an ingested event, so a 10-step agent run consumes ~10+ units; Core is $29/month for 100,000 units plus $8 per additional 100k (langfuse.com/pricing). For a 10-step agent, both free tiers land near ~5,000 runs/month despite the 10x difference in headline numbers - do this arithmetic against your own step count before choosing.
Emit OpenTelemetry GenAI semantic conventions (gen_ai.*) rather than a proprietary schema
The gen_ai.* conventions (gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens/output_tokens, agent and execute_tool spans) are still marked Development, not Stable, as of mid-2026 (opentelemetry.io semconv registry) - but Datadog, Arize/OpenInference, and the major LLM observability vendors already consume them. Emitting the standard keeps you portable across the exact vendor decision this checklist defers.
Capture prompt/completion content with explicit opt-in and redaction, not by default
Content capture is where tracing meets compliance: the OTel GenAI conventions treat prompt and completion recording as opt-in for a reason. Decide per-field redaction before launch - retrofitting redaction after PII has landed in 400-day-retention traces is an incident, not a ticket.
SECTION 02
3 CHECKS
Eval gates (build this layer - it is your product)
Gate every prompt, model, and tool-schema change behind an automated eval suite
The eval suite is the agent's regression test: prompts, model versions, and tool schemas change behavior exactly the way code changes do, but most teams still ship them without any equivalent of CI. A prompt change without evals is a deploy without tests - and 'we tweaked the prompt' is the classic unlogged root cause precisely because nothing gated it.
Build eval datasets from your own production traces, not synthetic examples
Traces you already collect are the only eval set that matches real input distribution. Feed failed and escalated runs back into the dataset weekly; vendors host the loop (LangSmith datasets, Langfuse scores) but cannot write your pass criteria - that judgment is the part you cannot buy.
Score outputs asynchronously in production, not only pre-deploy
Pre-deploy evals catch regressions you anticipated; online scoring (LLM-judge or heuristic checks on sampled production runs) catches drift you did not. The ~5% error rate Datadog measures only counts calls that actually raised errors - a confidently wrong answer returns 200, and only downstream grading ever sees it.
SECTION 03
3 CHECKS
Cost SLOs and alerting thresholds
Attribute token cost per span, per tenant, and per agent - not per invoice line
Span-level token and cost capture is the practice the 2026 production guidance converges on: without it, a cost spike cannot be traced to the tenant, loop, or model-routing change that caused it. Aggregate billing data arrives weeks after the anomaly.
Alert on spend-rate deviation from baseline, and set a hard error-rate threshold against your own history
There is no published industry-standard LLM error-rate cutoff - the defensible practice is baselining. With ~5% of AI model requests failing in production being typical (Datadog 2026), a fixed 1% alert pages constantly and a fixed 10% alert misses real fires; alert on deviation from your service's rolling baseline for both errors and cost.
Enforce budget ceilings at the gateway, with the observability stack as witness rather than enforcer
Dashboards observe; gateways enforce. Pair cost telemetry with actual budget enforcement (per-key/tenant caps that reject over-budget calls) - the full architecture is in our cost-guardrail pattern for agent platforms. An alert without an enforcement layer is a notification that money is already gone.
SECTION 04
3 CHECKS
Rollback paths and recovery
Version prompts, model choices, and tool schemas so any agent behavior change can be reverted in one step
An agent's effective behavior is code plus prompt plus model version plus tool schema. If any of those deploys without a version pin, 'roll back' has no meaning - you cannot revert to a state you never recorded.
Design compensation paths for multi-step tasks before launch, and surface them in the operator view
Multi-step agent tasks partially succeed: three side effects land, the fourth fails. Recovery requires checkpoints and compensation handlers registered per step (see our runtime-rollback pattern for agent tasks) plus an operator surface showing exactly which stage failed with what state - a raw log dump is not a recovery tool at 3 a.m.
Keep an audit log of runs, decisions, and interventions that outlives your tracing retention window
Tracing retention is short by default - 14 days on LangSmith base traces, 30 days of data access on Langfuse Hobby. Incident reconstruction, customer disputes, and compliance reviews routinely need runs older than that; export the durable subset (run metadata, decisions, outcomes) to storage you control.
Common mistakes
- 01Building a bespoke tracing pipeline first while shipping zero evals - inverting the buy-vs-build call on both layers at once.
- 02Choosing an observability vendor on headline free-tier numbers rather than billing units: LangSmith bills per trace (a whole agent run) while Langfuse bills per ingested event (every step counts), so 5k traces and 50k units converge to roughly the same included runs for a 10-step agent - the real divergence is in overage pricing ($2.50-5.00/1k traces vs $8/100k units) and retention.
- 03Treating 200-OK responses as success - with ~5% of production AI requests failing outright and correctness failures returning clean status codes, HTTP status is not an agent health metric; only output scoring is.
- 04Alerting on absolute thresholds copied from a blog post instead of the service's own baseline, producing either pager fatigue or missed incidents.
- 05Storing prompts and completions by default and discovering the compliance problem when the first data-deletion request arrives.
- 06Shipping prompt changes outside version control because 'it's just text' - making the most common cause of agent regressions the one change class with no rollback path.