Better Memory Architecture for Long-Running Agents. It’s NOT RAG.

Agent memory systems start with the same assumption:

If the information is stored, retrieval will solve the rest.

So we add vector embeddings, BM25, rerankers, larger context windows. maybe a memory database.

Then we tell the agent to search its history whenever a new request arrives.

That works well when the new request contains enough clues to form the right query but it fails in a more dangerous case: the agent has relevant context, but does not realize that it should search for it.

Imagine this:

Three weeks ago, your team discussed API rate limiting with an agent and decided on a token-bucket strategy.

Today you ask:

Refactor this endpoint for the new payment flow.

The words rate limit, token bucket or throttling never appear in the new request.

A normal memory tool searches for payment flow, and it may retrieve payment code, Stripe notes, or old endpoint work but the rate-limiting decision stays buried.

The agent failed to notice that a relevant memory existed.

That distinction is the interesting idea behind Hipocampus, an open-source memory harness for coding and agentic workflows.

Hipocampus adds a small, always-visible memory index in front of search so the agent can answer a different question first:

Do I already know something that might matter here?

That is a very important systems primitive.

Agent Memory Needs Awareness.

A typical RAG pipeline look roughly like this:

user request ↓ generate search query ↓ retrieve top-k chunks ↓ stuff chunks into context ↓ answer

This architecture assumes the request itself contains enough information to produce a useful search query.

For explicit questions, that is reasonable:

"What did we decide about Redis eviction?"

Search for Redis eviction and you probably get the right discussion.

But many real agent tasks are implicit:

"Can you clean up the caching layer before release?"

Maybe the important historical context is that two months ago the team rejected Redis entirely because of a production incident.

Maybe the decision lives in a session about deployment reliability, not caching.

Vector search can bridge synonyms but it cannot reliably bridge arbitrary cross-domain relevance.

Reactive retrieval and proactive memory are different capabilities.

MemAware benchmark tests 900 questions over three months of conversation history.

The user does not explicitly ask the agent to remember anything. but the agent has to surface relevant context on its own.

The baseline results:

The relative difference is meaningful.

Editor’s note: If you want to dive deep into Local LLMs and Agentic Stack, you can join our Agent Foundry program for hands-on, in-depth trainings.

Hipocampus + vector at 21.6x the no-memory baseline and 5.1x search alone overall.

More importantly, the architecture changes where the decision to retrieve happens, as search becomes the second step, not the first.

This also lines up with broader long-context research.

Lost in the Middle previously showed that simply placing more information inside a long context does not guarantee that a model will use it robustly.

MemGPT explored OS-inspired hierarchical memory, and LongMemEval formalized long-term memory as an indexing, retrieval, and reading problem.

Hipocampus makes a pragmatic addition to that stack:

Give the model a compact map of its history before asking it to retrieve details.

The Core Design: A 3-Tier Memory Hierarchy

Hipocampus organizes files like a CPU cache hierarchy.

The important part is that each tier has a different latency, token cost, and information density.

The project layout looks like this:

project/ ├── SCRATCHPAD.md ├── WORKING.md ├── TASK-QUEUE.md ├── memory/ │ ├── ROOT.md │ ├── 2026-08-18.md │ ├── daily/ │ ├── weekly/ │ └── monthly/ ├── knowledge/ ├── plans/ └── hipocampus.config.json

Layer 1: Hot memory

The hot tier contains operational state plus memory/ROOT.md.

ROOT.md is the interesting file.

It is intentionally small, around 3K tokens by default, and is designed to represent the entire history as a topic map rather than as a detailed summary.

A typical shape is:

Active Context

payments: endpoint migration, rollout checklist

Recent Patterns

API changes often affect rate limiting and observability

Historical Summary

2026-06: reliability work, queue migration

Topics Index

rate-limiting [project, 21d]: token bucket, burst limits → knowledge/rate-limits.md
payments [project, 2d]: new flow, retries → plans/payment-migration.md

The agent does not need the full rate-limiting conversation in every prompt.

It only needs enough information to notice, and that is the awareness layer.

Layer 2: Warm memory

Detailed information stays in normal files:

memory/YYYY-MM-DD.md knowledge/.md plans/.md

Daily logs are append-only source records.
Knowledge files are curated documents.
Plans hold task-specific state.

The agent reads them only when the hot index points toward them or when the task explicitly requires them.

Layer 3: Cold memory

Older history is represented as a compaction tree:

Raw ↓ Daily ↓ Weekly ↓ Monthly ↓ ROOT

Retrieval goes in the opposite direction:

That gives the agent two complementary tools:

tree traversal for browsing and discovery
qmd search for targeted retrieval

Hipocampus uses qmd for local search. qmd combines BM25, vector search, query expansion, and reranking.

The architecture is therefore is:

small global index + hierarchical summaries + RAG when needed.

Quick Start: Add Hipocampus to an Existing Agent Project

Hipocampus is distributed as an npm CLI.

One compatibility note before installing: the Hipocampus package currently declares Node >=18, while current qmd documentation requires Node >=22. Because the default setup installs qmd, Node 22+ is the safer baseline today.

From the root of your project:

Code

bash

npx hipocampus init

If you use the Claude Code plugin flow:

/plugin marketplace add kevin-hs-sohn/hipocampus /plugin install hipocampus@kevin-hs-sohn/hipocampus

Then initialize the project:

Code

bash

npx hipocampus init

You can also force a platform:

Code

bash

npx hipocampus init --platform claude-code
npx hipocampus init --platform opencode
npx hipocampus init --platform openclaw
npx hipocampus init --platform codex

If you do not want vector models:

Code

bash

npx hipocampus init --no-vector

If you want only the compaction tree and no qmd search:

Code

bash

npx hipocampus init --no-search

There is also tenant-oriented setup:

Code

bash

npx hipocampus init --tenant acme

That is useful if your agent host maintains isolated memory trees per customer, workspace, or agent identity.

What `init` Actually Does

The setup command is more than a template copier.

Reading through [cli/init.mjs](https://github.com/kevin-hs-sohn/hipocampus/blob/main/cli/init.mjs), the initialization flow is roughly:

Code

text

detectPlatform()
ensureMemoryDirectories()
installAgentSkills()
writeConfig()
registerQmdCollections()
buildSearchIndex()
injectPlatformProtocol()

That is adapted pseudocode, but it captures the sequence.

The CLI creates these memory directories:

memory/ memory/daily/ memory/weekly/ memory/monthly/ memory/agents/compaction/ memory/agents/recall/ knowledge/ plans/

It then creates a default config similar to:

Code

json

{
"platform": "claude-code",
"search": {
  "vector": true,
  "embedModel": "auto"
},
"compaction": {
  "rootMaxTokens": 3000,
  "cooldownHours": 3
}
}

When qmd is enabled, Hipocampus registers both memory and knowledge as searchable collections, updates the index, and optionally generates vector embeddings.

In practical terms, your project gets a file-backed memory subsystem without adding Postgres, Redis, Pinecone, a memory service, or another HTTP dependency.

The First Workflow to Understand: Recall

Suppose your agent receives:

Refactor the payment endpoint before we enable the new checkout flow.

The [hipocampus-recall](https://github.com/kevin-hs-sohn/hipocampus/blob/main/skills/recall/SKILL.md) skill defines a three-step retrieval fallback.

Step 1: Check `ROOT.md`

The agent first looks at the Topics Index already in context.

If it sees:

rate-limiting [project, 21d]: token bucket, burst limits payments [project, 2d]: checkout migration, retries

it has enough evidence to say: there may be a connection.

That check is effectively constant-time from the agent’s perspective because the index is already loaded.

Step 2: Build a tiny manifest

If ROOT.md does not contain an obvious match but the request feels connected to previous work, Hipocampus instructs the agent to inspect metadata rather than loading full files.

The manifest may look conceptually like:

monthly/2026-06.md: reliability, queues, deployment weekly/2026-W27.md: API limits, retry behavior knowledge/payments.md: checkout, idempotency knowledge/observability.md: tracing, alerts

The model selects up to five candidates, then loads only those and this is a clever middle ground.

A pure vector system asks an embedding model to decide relevance from lexical/semantic proximity.

Manifest selection asks the LLM itself:

Given a map of what exists, which files might matter to this request?

That makes cross-domain connections possible without dumping the archive into context.

Step 3: Use qmd

Only then does targeted search become the fallback:

Code

bash

qmd query "payment rate limiting"
qmd search "token bucket endpoint"
qmd vsearch "API overload protection"

The three modes map to hybrid, BM25-only, and vector-only retrieval.

This sequence is the part I would steal even if I never installed Hipocampus:

awareness ↓ candidate selection ↓ targeted retrieval

Most agent memory systems jump directly to the last box.

Memory Has a Write Path Too

A memory system is not useful if ingestion depends on a human remembering to update it.

Hipocampus handles writes through checkpoints.

The Claude Code core skill says that after a task completes, the agent should append a structured entry to the current daily log through a subagent.

Conceptually:

Payment endpoint refactor [project]

request: refactor endpoint for new checkout flow
analysis: checked retries, idempotency, rate limits
decisions: preserve token-bucket middleware
outcome: endpoint updated and tests passed
references: knowledge/rate-limits.md

The type is important.

Hipocampus supports four memory classes:

This is a stronger model than treating every chunk equally.

A user correction like:

"Never modify generated migration files manually."

should not disappear because an old month needs compaction.

Memory retention becomes policy-driven instead of purely similarity-driven.

Compaction Is Where the System Becomes Operationally Interesting

Raw logs grow forever and Hipocampus runs a mechanical and LLM-assisted compaction pipeline.

The implementation in [cli/compact.mjs](https://github.com/kevin-hs-sohn/hipocampus/blob/main/cli/compact.mjs) uses three line-count thresholds:

raw → daily: ~200 lines daily → weekly: ~300 lines weekly → monthly: ~500 lines

Below the threshold, content is copied or concatenated.
Above it, the node is marked for LLM summarization.

This is a good engineering decision because summarization is lossy.

If a daily log is only 80 lines, compressing it buys little and risks deleting useful details.

Copying verbatim preserves information and avoids an LLM call.

The compaction tree also distinguishes tentative and fixed nodes.

type: weekly status: tentative period: 2026-W33

An active week is regenerated as new information arrives.

After the period is safely in the past, the node becomes fixed and stops changing.

That reduces repeated work and makes the historical tree increasingly immutable.

The root is different: it never becomes fixed and it is continuously re-compacted to stay under the configured token budget.

When ROOT.md grows too large, the rules prioritize the parts most valuable for future awareness:

preserve active context
preserve the topics index
compress older historical summaries first

The Raw Log Is the Source of Truth

One design choice I strongly agree with: compaction nodes are indexes, not replacements.

Raw files are permanent.

memory/2026-08-18.md memory/2026-08-19.md ...

The tree can be regenerated.
Summaries can be improved.
Embedding models can be replaced.
Retrieval logic can change.

But the original session record still exists.

This is important because every summarization system eventually makes a bad compression decision.

If the summary becomes canonical, the error is permanent.
If the summary is only an index, you can drill back down to raw memory.

For production agent systems, I would keep this principle even if the storage backend changes:

Never make your lossy memory representation the only copy of history.

There Is Also a Security Boundary

Persistent memory quietly turns transient prompt content into stored data.

That changes the risk profile of an agent.

Hipocampus includes secret scanning during mechanical compaction.

The current implementation checks for patterns resembling API keys, passwords, bearer tokens, GitHub tokens, and private keys, replacing suspicious lines before writing compaction nodes.

That is useful, but developers should treat it as a guardrail, not a complete DLP system.

Regex-based secret detection will always have false negatives.

If you deploy this pattern in an agentic SaaS product, I would add stronger boundaries around:

tenant isolation
encryption at rest
retention windows
user deletion
PII classification
access-control-aware retrieval
audit logging
secrets that do not match known token formats

The file-first design is simple, but persisted memory is still persisted customer data.

Why Not Just Use a 1M-Token Context Window?

Because context capacity and context quality are not the same thing.

If your agent has 400K tokens of history, you can theoretically place all of it in a giant prompt.

You then pay for three things:

larger input processing
more irrelevant evidence competing for attention
repeated transport of mostly unchanged history

And you still do not get a guarantee that the model will use the right middle section.

Hipocampus instead tries to keep the always-loaded representation intentionally small.

The system prompt gets the map and the detail stays outside.

This is closer to how good software systems handle expensive storage generally:

small index large backing store selective reads

Why Not Just Search Every Time?

Because search has both a recall problem and a pollution problem.

If you always retrieve five chunks for every request, irrelevant memory becomes part of the model’s evidence.

That has a cost even when search latency is negligible.

The model now has to decide whether five historical snippets matter, poor retrieval does not simply fail harmlessly, it can change the answer.

MemAware measures the token side of this failure.

Its BM25 baseline consumes several thousand tokens per question while producing only a small proactive-recall gain over no memory.

The better architecture is conditional retrieval as it turns memory from mandatory context stuffing into a decision.

That should improve not only cost, but also answer cleanliness.

The Most Important Configuration Knob Is Probably `rootMaxTokens`

The default config uses:

Code

json

{
"compaction": {
  "rootMaxTokens": 3000
}
}

MemAware includes an experiment that increases the root budget to 10K tokens.

Overall proactive accuracy rises from 17.3% to 21.0%.
Easy questions improve from 26% to 34%.
Hard questions remain at 8%.

That result is interesting for two reasons.

First, coverage matters since alarger topic map exposes more possible connections.
Second, root size is not the only bottleneck, because at some point the agent still needs to reason across unrelated domains correctly.

So I would treat rootMaxTokens as a product knob rather than a constant.

For a small coding project, 3K may be enough and for a long-lived personal agent with hundreds of projects, preferences, and reference topics, 3K may under-index the history.

The right value depends on:

history diversity × topic count × model quality × per-request token budget

A production memory system should measure this rather than guess.

The Pattern I Would Steal for Any Agentic Product

You do not need to adopt the repository exactly to use the core idea.

Here’s the reusable pattern:

Append immutable raw interaction records.2. Compact them hierarchically.3. Maintain a tiny always-visible topic index.4. Use that index to decide whether memory is relevant.5. Retrieve details only after relevance is suspected.6. Preserve important memory classes differently.7. Keep summaries disposable and raw history recoverable.

That is a much more complete memory architecture.

Hipocampus combines those missing pieces in a form that developers can inspect with normal tools:

Code

bash

cat memory/ROOT.md
ls memory/weekly
git diff
qmd query "..."

This is the difference between an agent that merely has a database and an agent that actually has continuity.

Agent Memory Needs Awareness.

The Core Design: A 3-Tier Memory Hierarchy

Layer 1: Hot memory

Active Context

Recent Patterns

Historical Summary

Topics Index

Layer 2: Warm memory

Layer 3: Cold memory

Quick Start: Add Hipocampus to an Existing Agent Project

What init Actually Does

The First Workflow to Understand: Recall

Step 1: Check ROOT.md

Step 2: Build a tiny manifest

Step 3: Use qmd

Memory Has a Write Path Too

Payment endpoint refactor [project]

Compaction Is Where the System Becomes Operationally Interesting

type: weekly status: tentative period: 2026-W33

The Raw Log Is the Source of Truth

There Is Also a Security Boundary

Why Not Just Use a 1M-Token Context Window?

Why Not Just Search Every Time?

The Most Important Configuration Knob Is Probably rootMaxTokens

The Pattern I Would Steal for Any Agentic Product

What `init` Actually Does

Step 1: Check `ROOT.md`

The Most Important Configuration Knob Is Probably `rootMaxTokens`