What Is AI Agent Memory? Why Context Needs to Survive the Session

AI agent memory is a store that lives outside the model, which an agent writes to during a session and recalls from in later ones. It holds what the agent decided, tried, and learned, not just what was said. Context has to survive the session because the session is where the expensive work happens, and a context window throws all of it away on close.

This blog post covers what persistence costs, what it breaks, and what a memory layer needs before you put a second agent on it.

What is AI agent memory, exactly?

An agent memory layer is a persistent store with a write path, a read path, and a lifecycle. The agent writes a finding, the store enriches and indexes it, and any authorized agent can recall it later by meaning. Three things get called memory, and only one of them has all three properties.

The context window is a working set, not memory

Everything an agent worked out this session evaporates when the session closes. Tomorrow it re-derives the same conclusions, and you pay for the derivation again. Richmond Alake, who works on agent memory at MongoDB, put it plainly in a conference talk: “Yes, we have, like, large context window, but that’s not for you to stuff all your data in. That’s for you to pull in the relevant memory and structure it in a way that is effective.”

RAG is a read path, not memory

Retrieval-augmented generation answers “what do the documents say?” It has no write path, so it cannot answer “what did we decide, what did we try, and how did it turn out?” That knowledge exists in no document. It gets produced by agents at runtime, and it is exactly what disappears at session close.

Memory has a lifecycle

A memory gets written, classified, recalled, corrected when the world changes, and retired when it is no longer true. Alake lists the operations as generation, storage, retrieval, integration, updating, and deletion, then corrects himself on the last one: you implement forgetting; you do not delete. That correction matters more than it sounds, and section five explains why.

What kinds of memory does an agent actually need?

The four-type split you will see everywhere (semantic, episodic, procedural, working) comes from cognitive architecture research, and it is a reasonable starting point. The current academic picture is broader. Memory in the Age of AI Agents, a December 2025 survey with 47 authors that has been cited more than 250 times as of September 2026, argues the long-term versus short-term split has stopped being useful and reads memory through three lenses instead.

Forms covers how memory is physically realized: token-level, parametric, or latent.

Functions covers what it is for: factual, experiential, or working.

Dynamics covers how memory is formed, evolves, and gets retrieved over time. The familiar four types all sit inside Functions.

That framing is useful for a specific reason. It shows you that the type taxonomy is a description of purpose, and purpose is the easy part. Classifying a memory as episodic tells you nothing about who may read it or what to do when it contradicts the memory written last Tuesday.

Why does context need to survive the session?

Two reasons, and the second is the one that decides whether a multi-agent deployment is affordable.

Because re-sending the transcript scales linearly and recall does not

In a naive loop, each agent carries its own growing transcript and pays for the early turns again on every later one. Input tokens scale with agents, multiplied by history length and redundancy, and none of those three multipliers is capability. A bigger context window raises the ceiling on a bill that did not need to exist, and attention cost is roughly quadratic, so latency climbs with it.

Retrieval flattens the curve. Instead of re-sending the whole transcript every turn, an agent recalls only the memories the current task needs. Caura’s own benchmark results put the reduction at 96.6% on LoCoMo and 98.2% on LongMemEval, with accuracy holding at 77.6% and 72.5% respectively, inside the same narrow band the field clusters in. Search latency measures 23ms p50 warm.

Because in a fleet, one agent’s discovery is every agent’s discovery

This is the multiplier unique to multi-agent systems. Without a shared layer, agent seven solves a problem on Tuesday and agent twelve hits the same wall on Thursday and pays full price to solve it again. Shared memory collapses that redundant work to a single write. At eToro, roughly 291 agent identifiers write to one governed store that holds 26,500+ memories and 1,372 shared skills, and a workflow written once by one agent propagates across 300+ agents overnight.

What breaks once memory outlives the session?

Persistence solves the amnesia problem and creates four new ones. Caura’s paper, Governed Shared Memory for Multi-Agent LLM Systems, formalizes them, and they are worth naming precisely because semantic similarity cannot fix any of them.

Scope failure: An agent retrieves a memory outside its authorized boundary. A support agent pulls billing notes meant only for finance.

Time failure: A fact changes, and the old version stays retrievable at equal weight. The caura-long-run-fleet demo reproduces this on purpose: eight days of writes say a competitor charges $299, then on day nine the price becomes $349. A naive store now returns nine memories with no way to rank the correction above its originals.

Resolution failure: Conflicting facts coexist, and both stay live. Append-only storage has no principled way to choose.

Provenance failure: A retrieved fact cannot be traced to its writer, source, or time. Debugging turns into guesswork, and audits become unverifiable.

The caura-cross-fleet-gov demo shows scope and resolution failing together in a shape any company with a legal team will recognize. A sales agent holds an active $420k renewal for HealthSystem Inc. A legal agent holds an active GDPR hold on the same company. Neither can see the other. Only an admin agent with cross-fleet read recalls both, labels each by source fleet, and flags the conflict for a human.

What does a memory layer need, then?

Enough to answer four questions on every read: is this caller permitted to see this row, is this row still current, where did it come from, and how relevant is it. Relevance comes last, which is the whole design argument.

Governed recall is a pipeline, not a lookup

A vector database returns the top k rows by similarity, and every caller gets the same rows. Governed recall runs semantic candidate generation, then policy filtering on scope and trust, then temporal resolution, then provenance enrichment, and only then ranks. The same query returns different rows to different callers by design.

The distinction that matters in practice is where the boundary is enforced. In Caura, a fleet boundary becomes a WHERE fleet_id IN (...) predicate that executes before the search runs, so out-of-scope rows are never loaded, never scored, never ranked. That is different from a prompt instruction telling an agent not to look. Caura’s own docs are honest about the limit: in the open-source deploy, the predicate filters to whatever fleet_ids the agent declares and does not validate that declaration against the agent’s identity. Separate tenants give you the hard boundary.

Writes need enrichment and contradiction detection

On write, Caura’s pipeline classifies the memory type, extracts entities and relations into a knowledge graph, scores importance, flags PII, generates embeddings, and compares RDF triples plus LLM semantic analysis against existing rows to detect contradictions. Conflicting older entries get marked outdated automatically. On the day-nine price change above, recall then returns one result, $349, and reports how many entries were suppressed. The suppression comes from storage status, not from a prompt asking the model to ignore old data.

Policy needs to be mandatory, not discretionary

Recall is a library card. An agent might not surface the right memory at the right moment, and “should have recalled the rule” is a poor way to enforce EU data residency. Keystones are rules the platform serves to every agent at session start, ungated by trust level so a brand-new agent gets its full rulebook on turn one. Caura’s write-up on the cold-start problem covers how the two layers divide: ingestion answers what the org knows; keystones answer how the org requires the agent to behave.

If your agents are already contradicting each other across a pipeline, Caura is Apache 2.0 and self-hosts on Docker Compose in about five minutes, or runs managed with a free tier of 10K memories and unlimited agents.

What actually breaks when you measure it?

This is the section almost nobody writes, so it is the one worth reading. Caura built an open test rig called ArgusFleet and ran it against its own live production service, one experiment per governance dimension, then published the failures alongside the clean results.

The clean numbers: all 50 depth-four provenance chains reconstructed with the correct writer identity at every hop, completeness and accuracy both 1.000. Cross-fleet leak rate of 0.000 across 80 probes. A freshly written fact became visible to authorized readers in 0.83s p50.

The two failures are more instructive.

A dedup optimization starved a correctness mechanism. Contradiction resolution scored 1.000 when both conflicting writes were admitted, across 90 runs. Across all 200 fact-runs, it scored 0.490. The detector was fine. A synchronous near-duplicate gate was rejecting the second, contradictory write before the asynchronous contradiction detector ever saw it. An optimization built to suppress noise was suppressing the signal.

A read path resolved identity and then ignored it. Tenant isolation held everywhere, but the GET-by-id path evaluated only the tenant projection of a row’s scope and skipped the sub-tenant check. A low-trust agent could fetch a cross-fleet row by identifier that the trust ladder should have denied. The search path enforced the full predicate; direct fetch did not. It is the textbook confused-deputy pattern. It was remediated mid-study and a re-probe found zero leaks across 36 attempts.

Neither failure is visible in a design document, and neither is measurable on a single-agent benchmark. Take one thing from this section: when you evaluate a memory layer, ask whether the vendor has measured governance against a running service, and ask what broke. A vendor with no negative results has either not looked or not told you.

Which repo should you read first?

Caura ships three demo fleets, each isolating one failure mode. They run locally, including fully offline against Ollama.

Start with caura-build-fleet if your problem is agents making contradictory choices. A performance agent writes the constraint “no external JavaScript”. Two steps later an SEO agent recalls that constraint and picks inline JSON-LD instead of a CDN library. The two never communicate directly. A code review agent then cites both memory IDs in its verdict. The manager agent in that pipeline never receives the write tool definition at all, which makes read-only isolation structural rather than instructed.

The engine itself lives at caura-ai/caura, Apache 2.0, with an MCP server that drops into Claude Code, Cursor, or Windsurf with one config block.

When do you not need this?

If you are running one agent for one user, you do not need fleet governance, and you should not pay the complexity for it. Single-agent memory tools handle that case well, and the accuracy numbers across the field sit close together. The decision flips on three signals: more than one agent writing to the same store, more than one team or trust level reading from it, or a compliance requirement that someone can reconstruct who knew what and when.

Below that line, scoping and audit are overhead. Above it, they are the difference between a memory layer that runs in a demo and one that runs in production.

Conclusion

You now have the two questions that separate a memory demo from a memory deployment: who is allowed to read this row, and is this row still true? Ask them about whatever you are evaluating, including your own build.

The smallest useful first step takes an afternoon. Clone caura-build-fleet, run the five-agent pipeline once, and watch a constraint written by one agent change the decision of an agent two steps downstream that it never spoke to. If it changes how you think about your own pipeline, Caura is free to start on, 10K memories and unlimited agents and fleets, no credit card.

Frequently asked questions

Is AI agent memory the same as RAG?

No. RAG retrieves passages from documents you already have and has no write path. Agent memory captures what happens while agents work: decisions, outcomes, failed approaches, tribal knowledge that exists in no document. Memory also has a lifecycle, so entries can be superseded when facts change, which document retrieval cannot do.

Can’t I just use a bigger context window?

A bigger window raises the token ceiling without removing the redundancy. You still re-send the transcript every turn, still duplicate the same facts across every agent, and still re-derive what a sibling agent already learned. Models also lose the thread in the middle of very long contexts, so you pay more for noisier answers.

Is a vector database enough to build agent memory?

A vector database is a building block. On top of it you still need enrichment, memory typing, deduplication, visibility scopes, contradiction detection, provenance, and an agent-facing API. That is most of the work, and it is the part that decides whether the system survives a second agent.

How do multiple agents share one memory without leaking data?

Every memory carries a visibility scope at write time (agent-private, team-shared, or org-wide), and recall applies that scope as a query predicate before the search executes. Per-agent trust tiers then control who can read across team boundaries. For hard isolation that cannot be bypassed at the prompt layer, separate tenants are the stronger pattern.

What happens when a stored fact becomes wrong?

In a well-designed layer, the new write triggers contradiction detection, and conflicting older entries transition to an outdated status rather than being deleted. Recall filters them out by status, and the correction is always ranked above the row it replaced. Deleting instead of superseding destroys the audit trail, which is why forgetting and deletion are different operations.

How long does it take to add memory to an existing agent?

If your agent runs in an MCP client, it is one config block, and the memory tools appear. Self-hosting the open-source engine on Docker Compose takes about five minutes to first write. Example wiring for CrewAI, LangGraph/LangChain, and AutoGen ships in the caura-build-fleet README, and the REST API works with anything.

Related reading: What Is Agent Fleet Memory? · Persistent Memory for AI Agents · Solving the Agent Cold-Start Problem