Time to Stop Asking Your Agents to Decide What to Remember

Memory as a model behavior is a reliability bug

Every agent framework now ships with memory. Almost none of them agree on who is responsible for it. There are two answers: either the agent manages its own — the model decides, mid-conversation, to call a memory tool. Or the harness manages it — the orchestration layer reads before every call, injects context, extracts after every call, and writes through policy. The agent never sees a memory tool. It doesn’t know memory exists.

The first is agentic memory. The second is deterministic memory. When memory is a tool, remembering is a decision the model makes. Which means remembering is probabilistic. The failure modes are familiar to anyone who has run tool-based memory in production:

• The model forgets to write. The user states a critical fact; the agent answers well and never calls save. The fact evaporates. Nothing errors. You find out three sessions later.
• The model writes twice, or writes a paraphrase of an existing fact, and your store fills with near-duplicates that degrade every future retrieval.
• The model writes what the input told it to write. This is documented, not hypothetical: a poisoned document planted persistent false memories in Gemini by tricking the model into invoking its own memory-write tool (Rehberger, 2025); the same class of attack turned ChatGPT’s memory into a persistent exfiltration channel (“SpAIware,” 2024); and the academic version, MINJA (NeurIPS 2025), poisons an agent’s memory bank through queries alone with a 98% injection success rate.
• The model drifts on schema. Free-form writes decay into inconsistent formats that no downstream consumer can rely on.

None of these failures throw. All of them are silent. And they compound: at 99% per-step reliability, a ten-step workflow lands near 90% end-to-end. (That arithmetic assumes independent failures — treat it as an illustration, not a measurement. It illustrates the right thing: every memory decision you delegate to the model adds a step to the product.)

Deterministic memory removes the model’s vote. The write step happens every turn, unconditionally, because the harness executes it as code. What that buys — and what it doesn’t — is the next section, because this is where the argument is usually oversold.

What determinism buys — and what it doesn’t

Look at the loop again: recall → inject → run → extract → write. That extract step is an LLM call. “What did the user say that will still matter in a month?” is a judgment, and no amount of harness engineering makes judgment deterministic. The stochastic component doesn’t leave the system — it moves from the agent to the extractor.

So split memory failure into its two parts, because determinism fixes exactly one of them. Whether the memory step runs — genuinely fixed. The agent can no longer forget to save, skip the write under load, or be talked out of writing. The loop runs every turn, as code; the silent-failure class from §01 is closed, completely. Whether the step produces the right content — not fixed, relocated. Run the four failure modes through the harness loop and score them honestly:

The precise claim, then, is narrower than “failures become deterministic” — and stronger for being true: deterministic memory guarantees invocation and governance, not correctness. It doesn’t remove the model’s judgment; it removes the model’s vote on whether judgment runs at all.

And that relocation is worth a great deal. The irreducible stochastic judgment now lives in a component that always runs, uses one pinned model, sits behind policy checks — and, the real prize, can be evaluated in isolation in CI, against a fixed test set, with a regression bar. You can never do that to a judgment buried inside an agent’s tool-calling whims. Contained stochasticity, not removed stochasticity. That’s the trade, and it’s a good one.

Governance only holds at the harness

For anyone deploying more than one agent, this is the decisive argument — and it needs no hedge.

An agent that decides its own writes cannot be governed — only advised. You can tell it in the prompt not to store credentials, not to write across tenant boundaries, not to record health data. It will comply most of the time. “Most of the time” is not a compliance posture.

When the harness owns every read and write, policy is enforced structurally. Scope checks, PII rules, retention, audit logs — these run as code on every operation, and the agent has no path around them because the agent has no memory path at all. The same holds for reads: the harness decides which scope gets injected, so an agent physically cannot recall another tenant’s facts, no matter what the prompt injection asks for. Note how this composes with §02: the extractor can still be fooled into proposing a poisoned fact — but the proposal lands in a policy gate with provenance and an audit trail, not directly in the store.

This is the difference between a policy and a permission system. Prompts are policies. Harnesses are permission systems.

Where the field actually is

The convergence is real but uneven, and it deserves to be reported precisely — including the one counterexample. Each claim below was checked against the vendor’s current documentation:

• OpenAI Agents SDK — confirmed. Sessions are a first-class runtime concern: pass a session object into Runner.run and the SDK automatically prepends stored history before each run and persists new items after — no model decision anywhere — backed by SQLite, Redis, SQLAlchemy, and encrypted variants. The April 2026 update formalized the split further: in OpenAI’s own words, a “trusted application runtime” owns the loop, tools, approvals, and secrets, separate from sandboxed execution. (Scope note: sessions are conversation-history persistence, not long-term semantic memory.)
• LangGraph — plumbing confirmed. Checkpointers snapshot thread-scoped state at every super-step boundary, by the graph, with zero model involvement, and the long-term store is namespace-organized infrastructure. Honesty requires the other half: LangChain also actively promotes agentic writes to that store — “hot path” memory where the agent decides what to remember. The persistence plumbing is deterministic; what enters it is often still a model decision in LangChain’s own recommended patterns.
• Google ADK — partial. Long-term memory does live in a MemoryService wired into the Runner, with a framework-enforced split from session state — the storage architecture is exactly the pattern. But usage is opt-in, and ADK offers both dialects: preload_memory auto-injects at the start of every turn (deterministic recall), while load_memory is a tool the model decides to call (agentic recall), and writes require an explicit add_session_to_memory call or a developer-wired callback. ADK gives you deterministic memory as an option, not as a default.
• Claude Code — confirmed, for reads. CLAUDE.md project files and memory indexes are injected into context unconditionally at session start; the model never decides to read them. The write side is the mirror image: its auto-memory writes are model-initiated — the agent decides what’s worth saving. A read-deterministic, write-agentic hybrid — notable because it is the exact inverse of the split we argue for in §07.
• Zep & Mem0 — confirmed, primary API. Zep’s canonical loop is pure application-layer: add messages, retrieve a context block before the model call — the harness is the client. Mem0’s primary API is the same shape (add/search wrapped around chat calls), though Mem0 also first-party-ships OpenMemory MCP, a model-called path — so “the harness is the client” describes their primary architecture, not their only one.
• Letta (MemGPT) — the counterexample. Letta is the flagship of the opposite pattern, and pretending otherwise would be spin: its agents edit their own memory via built-in tools (memory_insert/memory_replace, the MemGPT lineage), and its background consolidation is performed by model-driven subagents, not harness code. What makes the exception interesting: Letta’s own CTO argued in 2026 that “memory is the harness, not a plugin” — the camps are converging on harness ownership of the loop, and disagreeing about how much editing discretion the model keeps inside it.

The honest reading: the components that need to be reliable have broadly been pulled out of the model and into the loop — unanimously so for conversation persistence and governance surfaces — while the field still splits on how much write discretion the model keeps.

Two stateless agents, two vendors, one governed store

Talk is cheap. Here is the entire pattern in about 130 lines of Python. Two agents — support running on Gemini, billing running on Anthropic — with zero tools and zero memory instructions. A Caura store running locally in Docker. A harness that does everything: recall, inject, run, extract, write. And zero SDKs: the harness speaks raw REST to all three services — two model APIs and the store — because that is the demo’s quiet second lesson: a harness is deterministic code written against wire contracts, and vendor is just a parameter.

First, the store — four commands, no signup, no API key:

git clone https://github.com/caura-ai/caura.git
cd caura
cp .env.example .env && echo "IS_STANDALONE=true" >> .env
docker compose up -d        # Postgres + pgvector + Redis + API (~30s)

Set an embedding provider key in .env (OPENAI_API_KEY=sk-...) so search is semantic rather than keyword-only — or run the fully local embedder profile if you want zero external calls.

# -- Caura over REST: ranked recall in, governed writes out ------------------
def recall(query: str) -> list[str]:
    r = requests.post(f"{CAURA}/api/v1/search", headers=HEADERS, timeout=15,
        json={"tenant_id": TENANT, "query": f"{SCOPE} {query}",
              "top_k": 8, "min_similarity": 0.0})
    r.raise_for_status()
    return [m.get("content") or m.get("title", "") for m in r.json().get("items", [])]


def remember(fact: str, agent_id: str) -> None:
    r = requests.post(f"{CAURA}/api/v1/memories", headers=HEADERS, timeout=30,
        json={"tenant_id": TENANT, "agent_id": agent_id,
              "content": f"[{SCOPE}] {fact}",
              "write_mode": "strong"})
    if r.status_code != 409:  # 409 = exact duplicate already in the store
        r.raise_for_status()


# -- Post-turn extraction: the stochastic step, pinned and scheduled ----------
EXTRACT = """Extract durable facts from this turn: things the user stated that
would still matter in a month. No inferences, no transient state.
Return ONLY a JSON array of short strings; [] if nothing qualifies.

"""


def extract_facts(vendor: str, user_msg: str, reply: str) -> list[str]:
    raw = complete(vendor, "You return only JSON.",
                   EXTRACT + json.dumps({"user": user_msg, "assistant": reply}), 500)
    raw = raw.strip().removeprefix("```json").removesuffix("```").strip()
    try:
        return [f for f in json.loads(raw) if isinstance(f, str)]
    except (json.JSONDecodeError, TypeError):
        return []


# -- The harness turn: recall -> inject -> run -> extract -> write -----------
def turn(agent_id: str, vendor: str, role: str, user_msg: str, memory_on: bool = True) -> str:
    facts = recall(user_msg) if memory_on else []
    reply = run_agent(vendor, role, user_msg, facts)
    if memory_on:
        for fact in extract_facts(vendor, user_msg, reply):
            remember(fact, agent_id)
    return reply

Run it. Session 1: the Gemini support agent hears “we deploy in eu-west-1, we’re on the Growth plan, and our contract renews in March 2027”; after the reply, the harness distills the facts and posts each one to /api/v1/memories, where the server enriches it with a type, title, tags, and importance weight, flags PII, stamps a visibility scope, and logs the write against support-agent. Session 2: the Anthropic billing agent — a different agent, from a different vendor, in production a different process on a different machine — asks the store what it knows, gets ranked results back, and answers correctly from facts a Gemini agent wrote. Session 3 is the kill shot: same store, same question, harness memory switched off — and the agent, correctly, says it doesn’t know.

Two runs, one diff. Neither agent changed. Only the harness did. Memory is a property of the system — not the model, and not the vendor.

And the cross-vendor case falls out for free. In the agentic model, cross-vendor sharing means teaching two different tool-calling dialects a compatible memory protocol and hoping they stay compatible. In the deterministic model, the models never touch the protocol at all — the harness is the store’s only client — so sharing is a config change: point two harness loops at the same store and scope. Fleet memory, including cross-vendor fleet memory, falls out of the architecture instead of being engineered on top of it.

Four honest footnotes on the demo

1. Scoping by content prefix is demo-grade. In production, account:acme maps onto Caura’s real isolation primitives — tenants, fleets, and per-memory visibility scopes (scope_agent, scope_team, scope_org) — not a string in the content field.
2. Pin one extraction model in production. The demo lets each agent’s own vendor run the extraction pass, to prove any of them can. In production, pin extraction to a single small, cheap model regardless of which vendor the agent runs on — it’s the highest-volume LLM call in the loop, and two extractors writing two phrasing styles into one store is self-inflicted schema drift (see §02 — this is the mitigation, not the determinism). Treating 409 as success on the write catches exact echoes; Caura’s crystallization is the backstop for paraphrases.
3. The store can be down when the agent isn’t. Production harnesses wrap recall so a store outage degrades to an empty context instead of a failed turn, and queue missed writes. raise_for_status() is demo-grade.
4. The retrieval knobs are tuned for legibility, not production. min_similarity: 0.0 with top_k: 8 returns the eight nearest memories however weak the match — right on a near-empty demo store, a noise source in a populated one; set a floor in production.

What a governed store buys at the harness level — and what it costs

You could run this loop against a hand-rolled local store — a table of facts and a SELECT. Here is precisely what the governed store changes, stated as trade-offs rather than marketing:

The honest summary: a governed store moves the hard parts of deterministic memory — retrieval quality, policy enforcement, dedup, audit — out of your harness and into infrastructure built for them, at the price of running that infrastructure. Below a handful of agents, the price may not be worth it. At fleet scale, the DIY alternative is a distributed-policy problem you don’t want.

What deterministic memory gives up — and the honest answer

One thing, and it’s real: agent-initiated recall. A pure harness-only design retrieves what its heuristic predicts is relevant before the agent runs. If the agent realizes mid-task that it needs something the heuristic didn’t fetch, it has no way to ask. The smarter the model, the more this constraint costs — and the field is visibly drifting toward more capable models doing their own retrieval.

So the production answer is not harness-only. It’s harness-first:

1. Writes are always deterministic. No exceptions. This is where silent failure and governance risk live, and the model gets no vote.
2. Baseline injection is always deterministic. Every call starts with the facts the scope guarantees.
3. Reads may optionally be agentic — a recall tool the agent can call when the injected context isn’t enough. With Caura this is the MCP surface on the same server: same scopes, same audit, one extra config block — because a failed read is recoverable (the agent asks again, or asks the user) while a failed write is silent data loss.

The asymmetry is the design principle: read failures degrade a single answer; write failures corrupt the future. Put determinism where the corruption is.

So — is it the future?

For writes and governance, it is largely the present: OpenAI’s runtime persists sessions automatically, LangGraph checkpoints as graph infrastructure, Zep and Mem0’s primary APIs assume the application is the client — and enterprise deployments will keep demanding it, because “the model usually remembers to follow policy” does not survive a security review. The convergence is real, if not unanimous: ADK still offers both dialects, and Letta bets on model-managed editing inside a harness-owned loop. For reads: expect hybrids, with the deterministic layer as the floor and agentic search as an optional ceiling.

The deeper shift is conceptual. We spent two years trying to make models responsible for their own memory, the way we’d ask a human employee to keep their own notes. That was the wrong analogy. The agent is stateless labor; the harness is the employer of record. Payroll doesn’t ask the worker to file their own records — and no one calls that a limitation. They call it a system.

Stop asking the model to remember. Make the system remember — and be precise about the promise: the system guarantees that remembering happens, under policy, every turn. Making what’s remembered good is still an engineering job. Now it’s one you can actually do.

FAQ

What is deterministic memory for AI agents?

Memory that the orchestration layer owns rather than the model. The harness recalls before every model call, injects the result into the prompt, extracts durable facts after the reply, and writes them through policy — unconditionally, as code. The agent has no memory tool and no memory decision to make. The opposite pattern, agentic memory, exposes memory as a tool the model may or may not choose to call, which makes remembering probabilistic and its failures silent.

What does deterministic memory actually guarantee?

Invocation and governance, not correctness. The extraction step is still an LLM call — a judgment — so determinism can’t make the content of memory perfect. What it guarantees is that the memory step runs every turn, through policy, with provenance and an audit trail — and that the one stochastic component left is pinned, isolated, and testable in CI. Contained stochasticity, not removed stochasticity.

Should the agent or the harness own memory?

Split it by direction. Writes and baseline injection belong to the harness, always — that is where silent data loss and governance risk live, and a missed write is invisible until three sessions later. Additional reads can be agentic: give the model an optional recall tool for the case where the injected context isn’t enough. A failed read degrades one answer and the agent can ask again; a failed write corrupts everything downstream of it. Put the determinism where the corruption is.

If the harness speaks REST, do I still need MCP?

Only for the discretionary reads — and note that protocol doesn’t determine ownership: a harness can call MCP deterministically, and a model can be handed REST-derived tools. In practice, deterministic harness code wants a versioned /api/v1 contract with an OpenAPI schema, while MCP’s runtime discovery exists for the case where the model is deciding. Caura serves both from the same server under one set of scopes, trust tiers, and audit rules, so adding the agentic read tool later doesn’t add a second security model.

Getting started

The harness loop above is the whole pattern — one loop, two agents, two vendors, ~130 lines in the full script, no framework and no SDKs required. Caura is the governed memory platform for agent fleets: Apache 2.0, self-hosted or managed, MCP-native, and the engine is on GitHub.

Sources

• attackRehberger — Hacking Gemini’s memory with prompt injection (Feb 2025)
• attackRehberger — SpAIware: persistent exfiltration via ChatGPT memory (2024)
• attackMINJA: memory injection attacks on LLM agents (NeurIPS 2025)
• surveyA survey on the security of long-term memory in LLM agents (2026)
• docsOpenAI Agents SDK — Sessions · The next evolution of the Agents SDK (Apr 2026)
• docsLangGraph — Persistence · Memory
• docsGoogle ADK — Memory
• docsZep — Quickstart · Mem0 — Memory operations · Letta — Memory blocks
• debateWooders — Memory is the harness, not a plugin · Chase — Your harness, your memory (Apr 2026)
• conceptAnthropic — Effective context engineering for AI agents (2025)
• concept12-Factor Agents — Own your context window (2025)
• termDMF: a deterministic memory framework (2026) — prior, different use of the phrase

Related reading: Beyond System Prompts: How Keystones Make AI Agents Obey Policy · What Is Agent Fleet Memory? · Shared Governed Memory