Agent Native
Loading...

Architecture Pattern

Idempotent Retry Pattern for Agent Tool Calls (2026)

How to retry failed agent tool calls without double-charging, double-emailing, or double-ticketing. Covers idempotency keys, effect journals, at-most-once vs at-least-once per tool class, and how Inngest step memoization and Temporal's activity-retry model implement the pattern.

Pattern: Idempotent retry with effect journaling for agent tool calls•Updated 2026-07-25

Problem

No mainstream agent framework guarantees exactly-once tool execution. LangGraph's functional API docs state plainly that a task which started but did not finish may run again on resume, and Temporal's retry-policy documentation says activities execute at-least-once and must be written as idempotent. Meanwhile agent loops retry aggressively by default: Inngest gives every step 4 retries on top of the initial attempt (5 total executions possible), and Temporal's default RetryPolicy has unlimited maximum attempts with a 1-second initial interval and 2.0 backoff coefficient. When the tool being retried is a Stripe charge, an outbound email, or a support-ticket creation, each duplicate execution is a real-world incident: a double charge, a double email, a duplicate ticket. The failure window is narrow but unavoidable — the tool call succeeds against the external API, then the process crashes before the framework records the result, so the replay re-executes a side effect that already happened.

Use this pattern whenever an agent tool call mutates external state that a user can observe: payments, emails, tickets, CRM writes, calendar invites, webhooks. If you are already on a durable-execution runtime (Temporal, Inngest, LangGraph Platform with checkpointing), you still need it — those runtimes deduplicate step results in their own store, but they cannot deduplicate the external side effect that landed before the checkpoint was written. Choose per tool class: at-least-once with idempotency keys for tools where the provider supports dedupe (Stripe), effect-journal dedupe for tools where the provider does not (most email APIs), and at-most-once (no automatic retry, escalate to a human instead) for irreversible high-blast-radius tools where a duplicate is worse than a missed execution. Skip the pattern only for read-only tools — retrying a search or a database read duplicates nothing. This is the sibling of our runtime rollback pattern, not a replacement: rollback compensates for work that should not have happened; idempotency prevents work from happening twice. Production agents need both.

Components

  • Tool-class registry that tags every tool as read-only (retry freely), idempotent-by-provider (retry with a key), journal-deduped (retry with an effect journal check), or at-most-once (never auto-retry; escalate)
  • Idempotency-key generator that derives a stable key from durable workflow identity (run ID + step ID + attempt-independent operation hash) — never from timestamps, retry counters, or LLM output
  • Effect journal: a durable store recording INTENT (before the call) and COMMITTED (after the provider confirms), keyed by the idempotency key, for providers with no native dedupe
  • Retry classifier that separates transient failures (timeout, 429, 5xx — safe to retry) from semantic failures (validation error, declined payment — retrying repeats the same rejection)
  • Durable-execution substrate that memoizes completed steps so retries skip finished work (Inngest step.run/step.ai.infer checkpointing, Temporal Workflow History, or a LangGraph checkpointer)
  • Escalation surface for at-most-once tools and for journal entries stuck in INTENT state, where a human resolves 'did the side effect land or not' with provider-side evidence

Flow

  1. 1Classify each tool at registration time, not at incident time. A tool that sends email is journal-deduped; a tool that captures payment is idempotent-by-provider; a tool that wires funds or deletes production data is at-most-once.
  2. 2Generate the idempotency key before the first attempt and store it in durable workflow state, above the retry loop. The key must be identical across every retry of the same logical operation — Inngest achieves the same property internally by memoizing step results against the step ID, and the tool-call key should inherit that stability.
  3. 3For idempotent-by-provider tools, pass the key to the provider. Stripe stores idempotency keys for at least 24 hours in API v1 (30 days in API v2), accepts up to 255 characters, applies them to POST requests, replays the original status code and body on duplicate submission, and sets an Idempotent-Replayed: true header so you can detect the replay. Reusing a key with different parameters is rejected rather than silently merged (source: docs.stripe.com/api/idempotent_requests).
  4. 4For journal-deduped tools, write an INTENT record with the key before calling the provider. On retry, check the journal first: a COMMITTED record means return the recorded result without calling the provider; an INTENT record means the previous attempt died mid-call, so verify against the provider (search sent messages, look up the ticket by external reference) before re-executing.
  5. 5Execute the call. On success, mark the journal COMMITTED (or let the durable runtime checkpoint the step result) before doing anything else in the workflow.
  6. 6On failure, run the retry classifier. Transient failures retry with exponential backoff under the same key — Temporal's default is a 1s initial interval doubling per attempt, capped at 100x the initial interval (source: docs.temporal.io/encyclopedia/retry-policies). Semantic failures do not retry; they route to the agent's error-handling path or the rollback/compensation pattern.
  7. 7For at-most-once tools, a failure after the request was sent never auto-retries. The workflow parks, and the escalation surface shows the operator the INTENT record plus provider-side evidence so a human decides whether to re-issue.
  8. 8Emit the journal as an audit stream: every key, attempt count, dedupe hit, and replay detection becomes the dataset that tells you which tools actually get retried in production and how often dedupe saves you.

Tradeoffs

Buy the substrate vs build the journal

Temporal and Inngest already solve the inner half of this pattern better than a hand-rolled solution will: Inngest memoizes completed steps by step ID so a retried function skips finished steps entirely, and Temporal's Workflow History replays recorded activity results instead of re-executing them. If you are hand-rolling retry loops around agent tool calls today, adopting one of those runtimes is a bigger reliability win than building a bespoke effect journal. The journal remains necessary only for the gap neither closes: the crash window between the external side effect landing and the checkpoint being written.

At-least-once throughput vs at-most-once safety

At-least-once with dedupe keeps agent throughput high because retries are automatic, but it depends on the dedupe actually working — a buggy key derivation silently reintroduces duplicates. At-most-once eliminates duplicates by construction but converts every ambiguous failure into a human interruption. Assigning the wrong class is the expensive mistake in both directions: at-most-once on a chatty email tool drowns operators; at-least-once on a wire-transfer tool is how you send money twice.

Provider-side dedupe coverage is uneven

Stripe's idempotency support is a documented contract with defined retention and replay semantics. Most email and ticketing APIs offer nothing comparable, which is why the journal component exists at all. Budget for the asymmetry: the payment path of this pattern is a few lines (pass the key), while the email path is a full read-verify-write protocol against your own store.

Journal writes add latency and a new dependency

Two durable writes (INTENT, COMMITTED) bracket every side-effecting call, adding storage cost and a few milliseconds per tool call — and the journal store itself becomes a component that can fail. For high-frequency low-stakes tools this overhead is not worth it; that is what the read-only and idempotent-by-provider classes are for. Reserve journaling for tools where a duplicate has a customer-visible cost.

Key stability vs LLM nondeterminism

Agents re-plan. If the model retries a failed step by emitting a slightly different tool-call payload (new email wording, reordered ticket fields), a key derived from the raw arguments changes and dedupe silently misses. Derive keys from the logical operation (recipient + template + triggering entity ID), not from a hash of the full LLM-generated arguments — and expect to get this wrong once before the audit stream shows you why.

Failure Modes

  • Key generated inside the retry loop: each attempt derives a fresh key (timestamp or attempt counter leaks in), so the provider sees N distinct requests and dedupes nothing. The classic symptom is duplicates that only appear under load, when timeouts trigger retries.
  • Semantic failure retried as transient: the provider correctly rejected the call (card declined, validation error), but the classifier retries it under the same key. Stripe returns the cached rejection, which looks like the bug 'retry never works' — while a journal-deduped tool without parameter checking may re-execute a call that should have been abandoned.
  • Crash between provider success and journal COMMIT: the email sent, the process died, the journal still says INTENT. If the retry path re-executes on INTENT instead of verifying against the provider first, the pattern itself causes the double-send it was built to prevent.
  • Dedupe window shorter than the retry horizon: Stripe v1 guarantees key retention for only 24 hours, but Temporal's default unlimited-attempts policy can keep retrying a stuck activity for days. A retry landing after key expiry is processed as a brand-new request. Cap retry duration (Schedule-To-Close timeout in Temporal) below the provider's dedupe window.
  • Subagent side effects invisible to the parent: a spawned subagent executes tool calls under its own run identity, so its keys and journal entries are not in the parent's dedupe scope. When the parent retries the whole subtask, every side effect the subagent completed executes again. Journal entries must be keyed to the root workflow, not the spawning agent.
  • LLM-generated arguments drift between attempts, changing an argument-hash-derived key so the dedupe check misses and both variants of the email go out.

Implementation Notes

  • On Inngest, wrap every side-effecting tool call in its own step.run (or step.ai.infer for inference calls) — Inngest checkpoints each completed step's result against its step ID and replays it on retry instead of re-executing, with a default of 4 retries after the initial attempt, configurable per function (source: inngest.com/docs — 'Inngest steps' and 'error retries' pages). Keep exactly one side effect per step; two effects in one step means a crash between them re-runs the first.
  • On Temporal, one tool call per activity, and set Schedule-To-Close so total retry duration stays inside your providers' dedupe windows — with the default policy (1s initial interval, 2.0 backoff, unlimited attempts) an activity without that timeout can retry effectively forever (source: docs.temporal.io/encyclopedia/retry-policies). Derive idempotency keys from workflow ID + activity ID, which are stable across retries and available in the activity context.
  • In LangGraph, follow the functional API docs' own guidance: place every API call inside a task so it is checkpointed, and design task bodies to be idempotent because a task that started but did not finish may run again on resume (source: docs.langchain.com/oss/python/langgraph/functional-api). LangGraph's RetryPolicy adds backoff-with-jitter retries for transient errors, so unguarded side effects inside nodes will be re-executed.
  • Set non-retryable error types explicitly. Both Temporal (non_retryable_error_types in RetryPolicy) and Inngest (NonRetriableError) support marking semantic failures as terminal — use them so a declined payment or validation error stops the loop instead of burning attempts against a cached rejection.
  • For email specifically, do not trust the request path alone: before re-sending on an INTENT journal entry, query the provider's sent-message API filtered by your key (stored as a custom header or metadata field) to learn whether the first attempt landed. This read-before-retry check is what makes journal dedupe honest for providers without native idempotency.
  • Decide the boundary with the rollback pattern explicitly: idempotency governs the retry path of a single tool call; saga compensation governs unwinding completed steps when a later step fails semantically. If your incident review says 'we did the right thing twice', fix key derivation here. If it says 'we did the wrong thing once', you need the runtime rollback pattern for agent tasks instead.