Scope
Agent audit logging for compliance
Debugging traces and audit logs are not the same artifact, and the pricing sheets prove it. LangSmith's base plan retains traces for 14 days, with extended 400-day retention billed at $5.00 per 1,000 traces versus $2.50 for base traces (per docs.langchain.com/langsmith/usage-and-billing). Langfuse's free Hobby tier gives 30 days of data access, Core ($29/mo) gives 90 days, and only Pro ($199/mo) reaches 3 years (langfuse.com/pricing). Meanwhile SOC 2 Type II auditors sample your full observation window — practitioner guidance (Konfirmity's SOC 2 data-retention guide, auditkit.dev's audit-log requirements checklist) converges on at least 12 months of queryable logs, with published buffer recommendations running from 90 extra days to 15 months total. So the default retention on every popular observability tier is 4x to 26x too short for compliance. The verdict: treat your observability platform as a debugging tool with honest strengths — a managed vendor's 400-day extended tier beats most homegrown pipelines' durability — and treat the audit log as a separate, append-only record you design deliberately. This checklist covers what that record must contain, who should choose which retention strategy, and the redaction tradeoffs nobody prices in.
What every agent action record must capture
4 checksLog each tool call as its own entry: tool name, full input arguments, result or error, and UTC timestamp
SOC 2 audit-log guidance (auditkit.dev's requirements checklist, Rectify Cloud's 'what counts as evidence' guide) is explicit that aggregated summaries fail audits — auditors want individual, queryable events showing who triggered the run, which tool was called, the exact arguments, and the outcome. This is also the record you will reach for when an agent mutates the wrong customer's data.
Record every permission decision — allowed, denied, or escalated to a human — with the approver's identity and timestamp
An approval without an identity attached is not evidence. If your human-in-the-loop gate approves a refund, the log must show which human, when, and what exact payload they saw. Policy-engine denials matter as much as approvals: they demonstrate the control actually operates, which is what a Type II audit tests.
Capture model provenance on every run: provider, model name, exact version or snapshot ID, and system prompt version
Providers silently rotate model snapshots. When you investigate why an agent started approving bad refunds on March 3rd, 'claude-something' is useless — you need the version string and the prompt revision that were live at that timestamp, because those two fields are the only way to separate a model regression from a prompt regression.
Attach a stable correlation ID (session/run ID) linking user request, agent steps, tool calls, and side effects
Investigations reconstruct causal chains, not isolated events. Without a run ID that survives across services, proving 'this Stripe charge came from that user message' means grepping timestamps across systems — slow during a normal incident, fatal during a disclosure deadline.
Retention and storage: the build-vs-buy decision
4 checksMatch retention to your audit window: 12 months minimum for SOC 2 Type II, plus a 90-day buffer
Compliance practitioners (Konfirmity's SOC 2 data-retention guide, auditkit.dev) state auditors expect the full observation period — typically 12 months — to be reviewable, and published guidance recommends retaining a buffer beyond it: 90 extra days at minimum, 15 months total in some guides. LangSmith base retention is 14 days and Langfuse Hobby is 30 days of data access; shipping on defaults means your first Type II audit period is unrecoverable.
Price the extended-retention tiers before deciding to build your own pipeline
LangSmith extended retention costs $5.00 per 1,000 traces (vs $2.50 base); Langfuse Pro at $199/mo includes 3 years of data access. For a mid-volume product, paying the vendor is often genuinely cheaper than engineering an export pipeline plus a queryable store — an honest win for the managed platforms. Build your own only when data-residency or redaction requirements force it.
Export compliance-relevant events to append-only storage you control (object lock / WORM), independent of the observability vendor
Your trace platform's retention is governed by your subscription, not your audit obligations — downgrade the plan and the evidence evaporates. A nightly export of audit-relevant events to versioned, object-locked storage costs a few dollars a month and removes vendor lifecycle from your compliance story.
Make tampering evident: hash-chain entries or use storage-level immutability, and log access to the logs themselves
SOC 2 evidence guidance (Rectify Cloud's evidence-retention and audit-trail guides) emphasizes complete and immutable trails. A log an engineer can silently edit is testimony, not evidence. Hash chaining or bucket object-lock is cheap; retrofitting integrity onto a mutable Postgres table after an incident is not.
Redaction, privacy, and the retention conflict
3 checksDecide field-by-field what gets redacted before write, not at query time
Tool-call inputs routinely contain customer PII, tokens, and free-text prompts. Once written to a 13-month immutable store, you cannot un-write them — so redaction (masking card numbers, hashing user identifiers, stripping credentials) must happen in the logging path. Query-time redaction fails the moment someone exports raw data.
Reconcile GDPR deletion rights with immutable audit retention using pseudonymization
A deletion request against a hash-chained log is a real conflict. The standard resolution is to keep the audit record keyed to a pseudonymous ID and delete the mapping table entry — the trail's integrity survives while the personal link is severed. Decide this before your first deletion request, not during it.
Keep debug traces and audit events on separate retention clocks
Telemetry-retention guidance for SOC 2 (OneUptime's retention-policy writeup, auditkit.dev's retention-by-framework guide) splits this cleanly: 30-90 days for operational/debug logs, 12 months to 3 years for compliance evidence. Applying the compliance clock to full traces multiplies storage cost and PII exposure; applying the debug clock to audit events destroys evidence. Two streams, two policies.
Common Mistakes
- Assuming the observability platform is the audit log — LangSmith's 14-day base retention and Langfuse Hobby's 30-day data access both expire long before a 12-month SOC 2 observation window closes.
- Logging tool outputs but not permission decisions, leaving no evidence that the human-approval control ever operated — the exact thing a Type II auditor samples for.
- Writing raw prompts and tool arguments (with customer PII and credentials) into long-retention storage, then discovering redaction cannot be applied retroactively.
- Skipping model-version provenance, making it impossible to correlate a behavior regression with a provider's silent model snapshot rotation.
- Building a custom log pipeline for cost reasons without pricing the vendor tiers first — at moderate trace volume, $199/mo for Langfuse Pro's 3-year access is cheaper than the engineering time to replicate it.