Agent Native
Loading...
Back to Patterns

Updated 2026-07-26

Architecture Pattern

Checkpoint and Resume Pattern for Long-Running Agents (2026)

A production architecture pattern for agents that run hours or days: checkpoint granularity, sync vs async vs exit durability, idempotent resume after interrupts, and choosing Postgres vs Redis vs object storage as the checkpoint backend - before you commit to a framework checkpointer or a durable-execution engine.

Agent Native•Pattern: Checkpoint and resume for long-running agents•2026-07-26

Problem

An agent task that runs for hours - research loops, migration jobs, multi-step approvals - will be interrupted: process crashes, deploys, rate-limit stalls, and human-in-the-loop pauses that last days. Without durable checkpoints, every interruption restarts the run from zero, re-spends every token, and replays every side effect. With naive checkpoints, the failure mode inverts: code that ran before a pause re-executes on resume, so the agent sends the same email twice or double-charges a card. Checkpoint design - what to persist, when to persist it, and where - is the actual architecture decision, and it is orthogonal to which framework you use.

Use this pattern when a single agent run outlives a process lifetime: anything with human approval gates, tasks longer than a few minutes of wall-clock time, workflows where re-running completed steps costs real money (LLM tokens, paid API calls), or systems that must survive deploys mid-run. Skip it for short stateless request/response agents - a retry-from-scratch policy is simpler and usually cheaper below roughly a minute of work.

Components

  • Durable checkpoint store keyed by a stable run identifier (thread_id in LangGraph terms) - Postgres for durability-first deployments, Redis for latency-first with explicit persistence configuration
  • Checkpoint writer that snapshots agent state at every step boundary (LangGraph calls these supersteps), with a configurable durability mode: write-before-continue, write-in-parallel, or write-on-exit
  • Pending-writes ledger that records outputs of steps that succeeded within a partially failed parallel batch, so only the failed branch re-runs on resume
  • Side-effect wrapper (LangGraph's @task, or your own outbox/idempotency-key layer) that persists the result of non-deterministic or externally visible operations so they are returned from the checkpoint instead of re-executed on resume
  • Interrupt/resume gate for human-in-the-loop pauses: the run parks in the store indefinitely and re-enters when a resume command arrives with the operator's decision
  • Retention tier: recent checkpoints hot in the primary store, completed-run history pruned or archived to object storage - full-history checkpointing grows with every step

Flow

  1. 01Each run is created with a stable run identifier; every checkpoint read and write is keyed by it. In LangGraph, no thread_id means no persistence and no resume - this is the first thing to verify in code review (docs.langchain.com/oss/python/langgraph/checkpointers).
  2. 02At each step boundary the checkpoint writer snapshots the full channel state. The durability mode decides the guarantee: LangGraph documents three - 'sync' (checkpoint written before the next step starts, highest durability), 'async' (written while the next step executes, small crash window), and 'exit' (written only when the graph exits, no mid-run recovery at all).
  3. 03When parallel steps partially fail, successful branches' outputs are stored as pending writes, so resume re-executes only the failed branch - LangGraph's persistence docs describe exactly this fault-tolerance mechanism.
  4. 04Externally visible operations (send email, charge card, call a paid API) run inside the side-effect wrapper. The wrapper checks the store first: if a persisted result exists for this run and step, it returns the stored result instead of re-executing.
  5. 05When the agent needs human input, it raises an interrupt. The run's state is already durable, so the process can die, the pod can be rescheduled, and days can pass - the run is a row in the store, not a live process.
  6. 06Resume re-enters with the same run identifier and the operator's response. Critically, code in the interrupted node that ran before the interrupt call executes again on resume - LangGraph's own docs tell you to place side effects after the interrupt or make them idempotent.
  7. 07A retention job prunes checkpoint history for completed runs and archives anything needed for audit to object storage, keeping the hot store bounded.

Tradeoffs

Durability mode: sync vs async vs exit

LangGraph's checkpointer docs (docs.langchain.com/oss/python/langgraph/checkpointers) order the modes from least to most durable as exit, async, sync - exit gives best performance but no mid-run recovery, async overlaps the checkpoint write with the next step and can lose the in-flight checkpoint on a crash, sync pays a write on every step boundary for the strongest guarantee. The honest default for long-running work is async; reserve sync for runs where replaying even one step has an external cost (payments, notifications), and exit only for cheap, restartable runs.

Checkpoint granularity vs storage growth

Framework checkpointers snapshot full channel state at every superstep, and agent state usually contains an append-only message history - so checkpoint volume grows roughly quadratically with conversation length if you keep full history. The Redis checkpointer ships two savers for exactly this reason: RedisSaver keeps full checkpoint history (time travel, audit) while ShallowRedisSaver keeps only the latest checkpoint (redis-developer.github.io/langgraph-redis, MIGRATION_0.1.0 notes). Decide per run class: audit-relevant runs get full history plus archival; high-volume ephemeral runs get shallow checkpoints plus TTL.

Postgres vs Redis vs object storage as the backend

Postgres is the durability-first default - checkpoints survive restarts by construction and the history is queryable alongside your app data. Redis is faster per write and langgraph-checkpoint-redis adds native TTL management (configurable expiry, refresh-on-read, or ttl_minutes=-1 mapping to Redis PERSIST), but durability is an ops setting, not a library feature: a Redis running default in-memory settings loses every checkpoint on restart unless AOF/RDB persistence is enabled (redis.io persistence docs). Object storage is an archival tier, not a checkpoint target - per-step read/write latency disqualifies it from the hot path.

Framework checkpointer vs external durable-execution engine

A checkpointer inside your agent framework persists graph state; a durable-execution engine (Temporal-style) persists a workflow event history and replays code deterministically. The checkpointer route keeps one less system in your stack and covers interrupt/resume and crash recovery well; the engine route buys you cross-service workflows, versioned replay, and mature retry policies at the cost of a second runtime and its mental model. Exhaust the checkpointer route first - most single-agent workloads never need more - and treat the engine as the escalation when your 'agent' is really a multi-service saga.

Failure modes

  • 01Side effects placed before an interrupt call: on resume the node re-executes from its start, so the email sends twice or the payment double-fires. LangGraph's docs are explicit that code before interrupt() re-runs - the fix is moving side effects after the interrupt, wrapping them in a persisted task, or making them idempotent with a request key.
  • 02MemorySaver (or no checkpointer) in production: works in every demo, loses every in-flight run on the first deploy. Any run that must survive a restart needs a database-backed saver.
  • 03Redis checkpointer with default persistence settings: checkpoints written fast, then a node restart silently erases them because neither AOF nor RDB was enabled - the library cannot save you from the server's configuration.
  • 04Unbounded checkpoint history: full-state snapshots per superstep with message history in state fills the store within weeks at production volume; no TTL, no pruning job, and the first symptom is checkpoint write latency degrading every run.
  • 05Resuming across a code deploy that changed the state schema or graph shape: the stored checkpoint no longer matches the code, and resume fails or - worse - silently misroutes. Version your state schema and drain or migrate in-flight runs on breaking changes.
  • 06Missing or non-stable run identifier: two invocations get different thread_ids, resume creates a fresh run instead of continuing the old one, and the interrupted run leaks in the store forever.

Implementation notes

  • 01In LangGraph, checkpoints are snapshots of channel state saved at every superstep, keyed by thread_id; pending writes from successful nodes in a partially failed superstep are stored so those nodes are not re-run on resume (docs.langchain.com/oss/python/langgraph/checkpointers and the persistence concept doc in the langgraph repo). Pass thread_id in configurable on every invoke - without it there is no persistence.
  • 02Set durability explicitly per workload instead of accepting the default: graph.stream(..., durability="sync") for runs with expensive side effects, "async" for the general case. The reference type is documented at reference.langchain.com/python/langgraph/types/Durability.
  • 03Wrap non-deterministic and side-effecting calls in @task (LangGraph functional API, docs.langchain.com/oss/python/langgraph/functional-api): task results are persisted to the checkpoint and returned - not re-executed - on resume. This is the single highest-leverage line of defense against replayed side effects.
  • 04For human-in-the-loop, use interrupt() plus Command(resume=...) rather than polling loops: the run costs nothing while parked, survives deploys, and the approval can arrive days later. Treat everything in the interrupting node before the interrupt call as replayable code.
  • 05If you choose Redis, configure both layers deliberately: library-side TTL (the langgraph-checkpoint-redis TTL config supports default expiry and refresh-on-read) for retention, and server-side appendonly yes for durability. If you choose Postgres, call the saver's setup() once at deploy time and use a connection pool - a connection per superstep collapses under concurrency.
  • 06Size your recursion or step limit so a healthy run finishes well under it - checkpointing interacts with loop limits, because a resumed run continues the same step count. Our agent-loop-termination pattern covers sizing stop conditions; this pattern covers surviving the stops.