Memory That Guides the Next Action: What MemoryArena and AMA-Bench Break About RAG-Only Agents

The paper's central result is a gap. Agents that score near the ceiling on existing long-context memory benchmarks like LoCoMo perform poorly in the agentic setting. The existing benchmarks assess memorisation and action in isolation; the agentic setting couples them.

AMA-Bench takes the trajectory view. Its episodes are long-horizon agent runs, characterised by machine-generated representations, causal dependencies and dense objective information, across web tasks, open-world tool question answering, text-to-SQL, software engineering, gaming and embodied tasks. The questions span four capabilities: recall, causal inference, state updating and state abstraction. The reference AMA-Agent gets to 57.22% average accuracy, 11.16 points above the strongest memory-system baseline, with two mechanisms: a causality graph that preserves objective information and causal dependencies, and tool-augmented retrieval that combines graph node search with keyword search.

Recall is one of the four. The other three, why did this happen, what is the current state, what is the pattern, are what a RAG-only memory does not represent, because a chunk of transcript has no notion of cause, current value or abstraction.

Why RAG-only memory scores well and fails anyway

A retrieval-only memory takes a query and returns the most similar stored text. It scores well on recall benchmarks because recall is the thing it does, and it fails in the agentic setting for three reasons the benchmarks make visible.

The query is not in the request. We covered this in the Hipocampus piece: the new request rarely contains the words that would retrieve the relevant memory. MemoryArena's interdependent subtasks are this failure at benchmark scale. Session two does not say "remember what you learned in session one".

State is not a chunk. If the agent learned in session one that the customer's plan changed to enterprise, and in session three that it changed back, a similarity search returns both. The agent needs the current state, which means something has to update, not append.

Causes are not text. "The deploy failed because the migration ran twice" is a causal edge between two events. Stored as prose, it is retrievable only if the query mentions deploys or migrations. Stored as an edge, it is reachable from either event, and from the question "what went wrong last time". That is the causality graph AMA-Agent uses, and it is why it wins on causal inference.

The production write path

The read side is well covered; the write side is where most systems are still a transcript dump. The shape below passes the kind of test the benchmarks run.

Four decisions in that shape do the work.

Entries are typed. A decision, a state, a cause and a preference behave differently on read and on expiry. A single "memory" type forces every read to be a similarity search.

State supersedes. The current value of a subject is one hop away, and the history is preserved for the causal questions.

Causes are edges. The graph is what makes "why" answerable. It also makes the index useful, because a subject with many causes attached is one the agent should notice.

Provenance and scope are mandatory. Who said it, from where, for which tenant. This is the security boundary from the Hipocampus post, and it is what lets you delete on request without losing the rest.

The always-visible index from the read-path post sits on top of these entries as a small manifest of subjects and recent decisions the agent sees before it decides whether to search. It fixes the "did not realise it should search" failure, and it is cheap because the index is tiny and the entries are short.

The eval checklist

Do not adopt MemoryArena or AMA-Bench wholesale. Adopt what they measure, on your tasks.

AMA-Bench's authors report their reference agent at 57.22%, which is a long way from solved. A memory system justifies its complexity when it beats stuffing the context on your task distribution, at a cost you can defend. Sometimes it does not, and the benchmark is what tells you.

What the two benchmarks contain

The summaries above lean on "multi-session" and "long-horizon", so the tasks deserve spelling out.

AMA-Bench builds its episodes from agent trajectories, not conversations. The six domains are web task execution, open-world tool question answering, text-to-SQL, software engineering, gaming and embodied tasks. What these share is that the memory an agent would need is machine-generated, dense and objective: tool outputs, query results, state transitions, not a user reminiscing about their weekend. The questions then probe four capabilities. Recall: what happened. Causal inference: why it happened. State updating: what is true now, after several changes. State abstraction: what pattern the history shows. The reference AMA-Agent reaches 57.22 percent average accuracy across those, 11.16 points above the strongest memory-system baseline it was compared with, and it credits the two mechanisms described above, the causality graph and the tool-augmented retrieval. A bigger embedding model is not on that list.

MemoryArena is built the other way round, from tasks that are designed to be interdependent. Its four domains are web navigation, preference-constrained planning, progressive information search and sequential formal reasoning, with concrete tasks such as bundled shopping, progressive search, travel planning under stated preferences, and formal reasoning in mathematics and physics where an intermediate result must be carried exactly. A task is split into subtasks across sessions such that later ones cannot be solved without what the agent learned, and remembered, earlier. The dataset is public, each row an agentic task with its subtasks, answers and background context, and the paper's framing is the loop: memory, agent, environment, repeat.

The domains tell you which benchmark is closest to your product. A support agent lives in AMA-Bench's tool question answering. A coding agent lives in its software engineering domain and in DreamBench-SWE, a separate benchmark this year specifically about memory hygiene for software agents across sessions. A shopping or planning assistant lives in MemoryArena. Build your own eval to resemble the closest one rather than adopting either wholesale.

A worked case: the plan that changed twice

We met the state-updating failure in this form, with names changed.

Session one, in March: the customer's account manager tells the agent the customer is moving from the starter plan to enterprise at the end of the month. The agent writes it down. Session two, in April: billing confirms the move happened. Session three, in June: the customer downgrades back to starter after a reorganisation; the agent is told and writes it down. Session four, in July: a support request arrives asking why a feature is unavailable.

A transcript-and-chunks memory returns, for a query about the customer's plan, the two most similar chunks. Both mention enterprise, because "enterprise" is the distinctive word and it appears in the March and April sessions. The June downgrade is one line in a session about a reorganisation and does not rank. The agent answers as if the customer were on enterprise, and the feature that is unavailable is unavailable because they are not.

A typed memory holds one state entry for the subject "plan", superseded twice, with supersedes pointing at the previous version each time. The read is one hop: current state of subject "plan" is starter, since June, source is the account manager, confidence four. The history is still there for the causal question a week later, when someone asks why the feature was ever enabled.

Nothing about the second design is clever. It is the difference between storing what was said and storing what is true, and the benchmarks are the first instruments that score the difference.

The always-visible index, revisited

The Hipocampus post argued that the read-path fix for "did not think to search" is a small, always-visible index in front of the memory. The typed write path makes that index better, because the index can be generated from the entries instead of maintained by hand.

A few hundred tokens, regenerated on every write, loaded at the start of every session. The agent does not have to guess whether a memory might matter; it can see that the customer is on starter before it reads the support request. When the request mentions a feature, the index has already made the connection the query never would have.

Governance: tenancy, deletion and shared memory

A memory that spans sessions also spans the moment someone asks for their data to be deleted, and the moment two principals share one memory. Two benchmarks this year are early markers for both.

GateMem benchmarks memory governance in multi-principal shared-memory agents: what happens when several users or agents read and write the same store, and who is allowed to see what. DreamBench-SWE, mentioned above, frames memory hygiene for software agents across sessions. Neither is mature enough to adopt as a target; both are a signal that the field has noticed memory is a data-governance surface.

The rules that make the typed write path governable are the same ones that made it useful. Provenance on every entry means deletion by source is a query: remove everything whose provenance is this ticket, this user, this session. Scope on every entry means tenancy is a filter on the read path rather than a hope. Expiry on every entry means the store shrinks on its own and a stale preference does not outlive its usefulness. A raw transcript has none of these properties, which is why "delete my data" against a chunk store is a project.

A four-week plan off transcript memory

For a team that has a working agent on chunk retrieval and wants to move without a rewrite.

Week one: add the typed write path beside the existing store. Every place the agent would have stored a transcript now also emits decision, state, cause and preference entries with provenance, scope and expiry. Nothing reads them yet.

Week two: generate the index from the typed entries and put it at the top of the context. Measure whether the agent volunteers memories it previously had to be asked for. This is the cheapest win and it usually shows immediately.

Week three: route the "what is true now" and "why" reads to the typed store, keep the chunk store for free-text recall. Build the eval from the checklist above, with at least one state-updating case and one causal case drawn from your own history.

Week four: run the eval against three configurations, chunks only, typed only, both, and against the plain long-context baseline with the transcript in the window. Keep whatever wins on cost and accuracy on your tasks. Sometimes the long-context baseline wins for your window size and traffic, and the eval is what lets you say so with a number.

Where to start

Start with week one of the plan above. Add the typed write path beside the store you already have, with provenance, scope and expiry on every entry, and let nothing read it yet. The index is generated from those entries, the state and causal reads route to them, and the eval scores the actions they guide. Whether that beats the plain long-context baseline on your tasks is the question week four answers, with a number.