Memory That Guides the Next Action: What MemoryArena and AMA-Bench Break About RAG-Only Agents

In August we argued that agent memory is not a retrieval problem, because the failure that hurts is the one where the agent has the relevant memory and never thinks to look. That post was about the read path. This one is about how you would know whether any of it works.

Two benchmarks published this year answer that. MemoryArena, at ICML 2026, tests memory inside a loop of memory, agent and environment across sessions whose subtasks depend on each other. AMA-Bench, also at ICML 2026, tests long-horizon memory over agent trajectories across six domains and four capabilities. Both find the same thing. Agents that are near-saturated on existing long-context recall benchmarks perform poorly when the memory has to guide an action in a later session.

If your memory eval is a recall benchmark, you have been measuring the wrong thing.

What the benchmarks actually test

MemoryArena is built around interdependence. A task is split into subtasks across sessions, and later subtasks cannot be solved without what the agent learned, and remembered, from earlier ones. The domains are chosen so that different kinds of memory matter: bundled shopping and travel planning need preference constraints carried forward; progressive information search needs partial findings accumulated; sequential formal reasoning needs intermediate results kept exact. The framing is a loop: the agent acquires memory while acting, then relies on it to act again.

The paper's central result is a gap. Agents that score near the ceiling on existing long-context memory benchmarks like LoCoMo perform poorly in the agentic setting. The existing benchmarks assess memorisation and action in isolation; the agentic setting couples them.

AMA-Bench takes the trajectory view. Its episodes are long-horizon agent runs, characterised by machine-generated representations, causal dependencies and dense objective information, across web tasks, open-world tool question answering, text-to-SQL, software engineering, gaming and embodied tasks. The questions span four capabilities: recall, causal inference, state updating and state abstraction. The reference AMA-Agent gets to 57.22% average accuracy, 11.16 points above the strongest memory-system baseline, with two mechanisms: a causality graph that preserves objective information and causal dependencies, and tool-augmented retrieval that combines graph node search with keyword search.

Read the four capabilities again. Recall is one of four. The other three, why did this happen, what is the current state, what is the pattern, are exactly what a RAG-only memory does not represent, because a chunk of transcript has no notion of cause, current value or abstraction.

Why RAG-only memory scores well and fails anyway

A retrieval-only memory does one thing: given a query, return the most similar stored text. It scores well on recall benchmarks because recall is the thing it does. It fails in the agentic setting for three reasons the benchmarks make visible.

The query is not in the request. We covered this in the Hipocampus piece: the new request rarely contains the words that would retrieve the relevant memory. MemoryArena's interdependent subtasks are this failure at benchmark scale. Session two does not say "remember what you learned in session one".

State is not a chunk. If the agent learned in session one that the customer's plan changed to enterprise, and in session three that it changed back, a similarity search returns both. The agent needs the current state, which means something has to update, not append.

Causes are not text. "The deploy failed because the migration ran twice" is a causal edge between two events. Stored as prose, it is retrievable only if the query mentions deploys or migrations. Stored as an edge, it is reachable from either event, and from the question "what went wrong last time". That is the causality graph AMA-Agent uses, and it is why it wins on causal inference.

The production write path

The read side is well covered. The write side is where most systems are still a transcript dump. Here is the shape that passes the benchmarks' kind of test.

type MemoryEntry = {
id: string;
kind: "decision" | "state" | "cause" | "preference" | "fact";
scope: { tenant: string; project?: string; session?: string };
subject: string;                      // what this is about, for the index
content: string;                      // short, written for a future reader
supersedes?: string;                  // for state: the entry this replaces
causes?: string[];                    // for cause: entry ids that led here
provenance: { source: "user" | "tool" | "agent"; ref: string; at: string };
confidence: 1 | 2 | 3 | 4 | 5;
expiresAt?: string;                   // preferences and session facts expire; decisions may not
};

// A state change updates; it does not append a contradiction.
async function recordState(subject: string, content: string, prov: MemoryEntry["provenance"]) {
const current = await memory.currentState(subject);
return memory.write({
  kind: "state", subject, content, provenance: prov, confidence: 4,
  supersedes: current?.id,
  scope: currentScope(),
});
}

// A cause is an edge, reachable from both ends.
async function recordCause(effectId: string, causeIds: string[], why: string) {
return memory.write({
  kind: "cause", subject: effectId, content: why, causes: causeIds,
  provenance: { source: "agent", ref: currentRunId(), at: now() },
  confidence: 3, scope: currentScope(),
});
}

Four decisions in that shape do the work.

Entries are typed. A decision, a state, a cause and a preference behave differently on read and on expiry. A single "memory" type forces every read to be a similarity search.

State supersedes. The current value of a subject is one hop away, and the history is preserved for the causal questions.

Causes are edges. The graph is what makes "why" answerable. It also makes the index useful: a subject with many causes attached is one the agent should notice.

Provenance and scope are mandatory. Who said it, from where, for which tenant. This is the security boundary from the Hipocampus post, and it is what lets you delete on request without losing the rest.

The always-visible index from the read-path post sits on top: a small manifest of subjects and recent decisions the agent sees before it decides whether to search. That is the mechanism that fixes the "did not realise it should search" failure, and it is cheap because the index is tiny and the entries are short.

The eval checklist

Do not adopt MemoryArena or AMA-Bench wholesale. Adopt what they measure, on your tasks.

That last line is the honest one. AMA-Bench's authors report their reference agent at 57.22%, which is a long way from solved. A memory system is worth its complexity when it beats stuffing the context on your task distribution, at a cost you can defend. Sometimes it does not, and the benchmark is what tells you.

What the two benchmarks actually contain

It is worth being concrete about the tasks, because "multi-session" and "long-horizon" are doing a lot of work in the summaries.

AMA-Bench builds its episodes from agent trajectories, not conversations. The six domains are web task execution, open-world tool question answering, text-to-SQL, software engineering, gaming and embodied tasks. What these share is that the memory an agent would need is machine-generated, dense and objective: tool outputs, query results, state transitions, not a user reminiscing about their weekend. The questions then probe four capabilities. Recall: what happened. Causal inference: why it happened. State updating: what is true now, after several changes. State abstraction: what pattern the history shows. The reference AMA-Agent reaches 57.22 percent average accuracy across those, 11.16 points above the strongest memory-system baseline it was compared with, and the two mechanisms it credits are a causality graph that preserves objective information and causal dependencies, and tool-augmented retrieval that combines graph node search with keyword search. Note what is not in that list: a bigger embedding model.

MemoryArena is built the other way round, from tasks that are designed to be interdependent. Its four domains are web navigation, preference-constrained planning, progressive information search and sequential formal reasoning, with concrete tasks such as bundled shopping, progressive search, travel planning under stated preferences, and formal reasoning in mathematics and physics where an intermediate result must be carried exactly. A task is split into subtasks across sessions such that later ones cannot be solved without what the agent learned, and remembered, earlier. The dataset is public, each row an agentic task with its subtasks, answers and background context, and the paper's framing is the loop: memory, agent, environment, repeat.

The reason to know the domains is that they tell you which one is closest to your product. A support agent lives in AMA-Bench's tool question answering. A coding agent lives in its software engineering domain and in DreamBench-SWE, a separate benchmark this year specifically about memory hygiene for software agents across sessions. A shopping or planning assistant lives in MemoryArena. Build your own eval to resemble the closest one rather than adopting either wholesale.

A worked case: the plan that changed twice

Here is the state-updating failure in the concrete form we met it, with names changed.

Session one, in March: the customer's account manager tells the agent the customer is moving from the starter plan to enterprise at the end of the month. The agent writes it down. Session two, in April: billing confirms the move happened. Session three, in June: the customer downgrades back to starter after a reorganisation; the agent is told and writes it down. Session four, in July: a support request arrives asking why a feature is unavailable.

A transcript-and-chunks memory returns, for a query about the customer's plan, the two most similar chunks. Both mention enterprise, because "enterprise" is the distinctive word and it appears in the March and April sessions. The June downgrade is one line in a session about a reorganisation and does not rank. The agent answers as if the customer were on enterprise, and the feature that is unavailable is unavailable because they are not.

A typed memory holds one state entry for the subject "plan", superseded twice, with supersedes pointing at the previous version each time. The read is one hop: current state of subject "plan" is starter, since June, source is the account manager, confidence four. The history is still there for the causal question a week later, when someone asks why the feature was ever enabled.

Nothing about the second design is clever. It is the difference between storing what was said and storing what is true, and the benchmarks are the first instruments that score the difference.

The always-visible index, revisited

The Hipocampus post argued that the read-path fix for "did not think to search" is a small, always-visible index in front of the memory. The typed write path makes that index better, because the index can be generated from the entries instead of maintained by hand.

# memory index · tenant acme · generated 2026-09-27

## state (current values)
- plan: starter (since 2026-06-14, src: account manager)
- region: eu-west (since 2026-01-09, src: onboarding)

## decisions
- rate limiting: token bucket, 5/15min per IP (2026-09-04, src: ticket 4412)
- exports: stream above 10k rows (2026-09-23, src: ticket 4471)

## open causes
- 2026-09-20 deploy failed <- migration ran twice

## preferences (expiring)
- tabs over spaces (expires 2026-10-01)

A few hundred tokens, regenerated on every write, loaded at the start of every session. The agent does not have to guess whether a memory might matter; it can see that the customer is on starter before it reads the support request. When the request mentions a feature, the index has already made the connection the query never would have.

Governance: tenancy, deletion and shared memory

A memory that spans sessions also spans the moment someone asks for their data to be deleted, and the moment two principals share one memory. Two benchmarks this year are early markers for both.

GateMem benchmarks memory governance in multi-principal shared-memory agents: what happens when several users or agents read and write the same store, and who is allowed to see what. DreamBench-SWE, mentioned above, frames memory hygiene for software agents across sessions. Neither is mature enough to adopt as a target; both are a signal that the field has noticed memory is a data-governance surface.

The rules that make the typed write path governable are the same ones that made it useful. Provenance on every entry means deletion by source is a query: remove everything whose provenance is this ticket, this user, this session. Scope on every entry means tenancy is a filter on the read path rather than a hope. Expiry on every entry means the store shrinks on its own and a stale preference does not outlive its usefulness. A raw transcript has none of these properties, which is why "delete my data" against a chunk store is a project.

A four-week plan off transcript memory

For a team that has a working agent on chunk retrieval and wants to move without a rewrite.

Week one: add the typed write path beside the existing store. Every place the agent would have stored a transcript now also emits decision, state, cause and preference entries with provenance, scope and expiry. Nothing reads them yet.

Week two: generate the index from the typed entries and put it at the top of the context. Measure whether the agent volunteers memories it previously had to be asked for. This is the cheapest win and it usually shows immediately.

Week three: route the "what is true now" and "why" reads to the typed store, keep the chunk store for free-text recall. Build the eval from the checklist above, with at least one state-updating case and one causal case drawn from your own history.

Week four: run the eval against three configurations, chunks only, typed only, both, and against the plain long-context baseline with the transcript in the window. Keep whatever wins on cost and accuracy on your tasks. The honest outcome is sometimes that the long-context baseline wins for your window size and traffic, and the eval is what lets you say so with a number.

Where this goes next

The memory benchmarks did to memory what SWE-bench did to coding: they made the gap between a demo and a product measurable. The demo retrieves. The product notices, updates, explains, and forgets on schedule.

Build the write path first. Then the index. Then the eval that scores the action. Then, and only then, argue about vector databases.