458 / 500 under GPT-4o, 449 / 500 under Gemini 3.5 Flash-Lite. One configuration, no dataset labels, Gemini 3.8 Flash answering, 48.6k-token median context.
Everything behind the number is public: the harness, the saved reader context for every question, every judge's per-question verdicts, and the scripts that produce the tables below are in caura-ai/caura-longmemeval (see REPRODUCE.md for the exact commands and server version).
This post explains what the benchmark measures, how the run is set up, what the answering model was given, what we deliberately did not do, and where the remaining errors are.
What LongMemEval measures
LongMemEval (Wu et al., ICLR 2025) tests whether an assistant can remember things a user told it across a long history of chat sessions. Each of the 500 questions comes with a haystack of roughly 50 sessions, about 115,000 tokens of conversation as the paper counts them (nearer 150,000 under the Gemini tokenizer this run uses), and a question asked at a later date. The answer is somewhere in the haystack, often in one sentence, sometimes spread across several sessions months apart.
The six question types are the ones that matter for a real assistant:
- single-session user: a fact the user stated once ("What breed is my dog?");
- single-session assistant: something the assistant said earlier and the user now refers back to;
- multi-session: information that has to be assembled from several conversations ("How many charity events have I done this year?");
- temporal reasoning: questions whose answer depends on when things happened ("How long after I moved did I start the new job?");
- knowledge update: a fact that changed, where the current value is the right answer and the old one is a trap;
- preference: recommendations that should reflect what the user has told you about themselves.
Thirty of the 500 questions have no answer in the history. The correct response to those is to say so.
A memory system's job here is to store the history and, at question time, hand the answering model the right evidence. The score is a joint property of the memory system and the answering model, which is why the choice of model and judge has to be stated with every number, and why we report the token budget the answering model was given next to the accuracy.
The result
| correct | accuracy | |
|---|---|---|
| Overall, GPT-4o judge | 458 / 500 | 91.6% |
| Overall, Gemini 3.5 Flash-Lite judge | 449 / 500 | 89.8% |
| single-session user | 66 / 70 | 94.3% |
| single-session assistant | 56 / 56 | 100.0% |
| multi-session | 114 / 133 | 85.7% |
| temporal reasoning | 120 / 133 | 90.2% |
| knowledge update | 76 / 78 | 97.4% |
| preference | 26 / 30 | 86.7% |
Per-category numbers are GPT-4o verdicts. Answering model: Gemini 3.8 Flash. Judge prompts: the official LongMemEval task-specific prompts, byte-identical to evaluate_qa.py. Abstention questions judged with the official abstention prompt; 27 of 30 are correct.
Two more runs frame the number. A second full pass over the same store with the same configuration (fresh retrieval, fresh generation) scores 92.0% under GPT-4o and 89.2% under Flash-Lite. Retrieval is deterministic (both passes put the same 863 of 886 gold turns in context), so the half-point spread is all answering model; the first pass’s number is the headline because we fixed it before the second pass’s result was known. An oracle run, in which the same answering pipeline is handed exactly the sessions that contain the answer and nothing else, scores 94.6% under GPT-4o (92.8% Flash-Lite); that is the reader's ceiling with perfect retrieval, and the three-point gap to it is what retrieval costs.
How the run is set up: store the turn, read the session
Chat sessions are the wrong unit to retrieve. A session-sized chunk about smoothie recipes, a charity walk and a new apartment embeds as a blend of all three, and the sentence "I just did the Walk for Hunger" ranks ninetieth while the smoothie part ranks fifth. Measured on this dataset, storing sessions in 4,000-character parts puts the answer-carrying turn in the reader's context for 91.7% of gold turns; the right session is usually retrieved, the wrong part of it is. So the store and the read use different units:
- Ingest at turn granularity. Each user turn plus the assistant reply that follows it is its own memory (about 800 characters median, 1,200 max), with the session date and an opaque session label in a header. The source text is stored as written; the one edit is that assistant replies longer than the limit are split, and each piece is prefixed with the user turn it answers so the embedding stays centred on what the user said. About three quarters of the stored chunks carry that (in reply to) prefix. This is 5.6 times more memories per haystack than session parts, all written through Caura's standard bulk endpoint. The header label is a 12-hex hash with the same shape for every session; neither the dataset's session id (which names every gold session answer_<hash>) nor the question id reaches stored text, and a test enforces it.
- Read at session granularity. At question time we run one search with the raw question against the server's default hybrid (dense + full-text) profile, take the top hits, and pull the rest of each hit's session from the store before handing the context to the answering model, in chronological order, under a 150,000-character budget on the stored text itself (session headers are added on top, so the assembled context runs a little longer). The turn-level vector picks the right sessions; the whole session supplies the number or date the question needs. Caura also derives its own memories (facts, preferences, tasks) from what it stores, asynchronously; /search returns them alongside the stored turns, and the harness drops them before expansion, so the context is stored conversation turns only.
Gold-turn coverage (the fraction of answer-carrying turns present in the reader's context, measured by scripts/coverage.py) is 0.974 — 863 of the 886 answer-carrying turns. Counted per question rather than per turn, 450 of the 470 answerable questions have every gold turn in context, which is 0.957. Knowledge-update questions, where the current value tends to sit in one short turn near the end of a session, come out best (0.993).
The small unit makes per-category retrieval profiles unnecessary. A single flat configuration matches or beats tuned per-type profiles on every category, so turn mode disables them. The number above uses no information about the question type, which is the honest setting for a system that has to answer whatever the user asks.
The answering side is a four-stage pipeline on Gemini 3.8 Flash: extract evidence from the full context, draft an answer, infer or fall back to a direct answer when the draft is unsupported, verify against the evidence. Its prompts were iterated against failures on this benchmark; the illustrations in them are synthetic (terms that occur in no question, answer or gold turn, checked by a test), no questions were held out, and readers should treat the reader prompt as tuned on this set.
What the answering model was given
Accuracy without the token budget is half a number. Mem0 publishes a mean of 6,787 retrieved tokens per query; Mastra reports an average context of about 30k tokens; Honcho reports a median of 5% of the haystack. Zep is the sharpest of these: its own published figure is 90.2%, a little over a point below ours, on roughly a tenth of the tokens. We are not putting these systems in a league table, because the answering model differs across every published figure — ours is a Flash-tier model where most of the others use a frontier one — and a ranking column would imply a comparability that does not exist. A run of this same store and pipeline with GPT-4o answering is the follow-up that makes the comparison honest. Ours, measured with the reader's own tokenizer on the saved contexts, and with every reader call counted from the provider's usage reports:
| per question | median | p95 |
|---|---|---|
| retrieved context handed to the reader | 48,568 tokens (157,448 characters) | 50,099 tokens |
| share of the haystack (median haystack: 490k characters, ~150k Gemini tokens) | 32% | |
| reader calls (extract, answer, infer or fallback, verify, retries) | 4 | 5 |
| total reader input tokens, all calls | 51,110 | 101,530 |
| total reader output tokens, all calls, thinking included | 1,528 | 4,999 |
| total reader tokens, all calls | 52,692 | 105,446 |
| retrieval latency | 7.6 s | 15.0 s |
| generation latency (all reader calls) | 20.9 s | 40.1 s |
Token counts from the Gemini tokenizer on the saved contexts and from provider usage reports for every call.
The budget caps the stored text at 150,000 characters; with the per-session headers added, the assembled median context is 157,448 characters, which the Gemini tokenizer counts as 48,568 tokens (0.31 tokens per character on chat text; a chars/4 estimate would be a fifth low). Only the extraction stage reads the full context; the later stages read the extracted evidence, so the sum over all calls is close to one context. The p95 input doubles because extraction is retried on about one question in ten and re-reads the whole context. Generation latency depends on API load: the same pipeline has run at 8 s and at 21 s median on different days, so read it as a range, not a property of the system.
This is a large budget by the standards of the systems above: we pull whole sessions rather than extracted facts, and Mem0's single-call answerer reads roughly a seventh of what our four-stage reader does. We think the trade is worth making explicit rather than hiding, and it is the obvious place to work next.
What we did not do
Memory benchmarks are easy to overfit, and some of the tricks are hard to spot from the outside. The things below were available to us and are not in this number.
- No question-type labels. The dataset tags each question with its category. The run never reads it.
- No dataset labels in the store. Session ids and question ids, which encode which sessions are gold and which questions are unanswerable, are replaced by opaque hashes at ingest.
- No judge shopping. GPT-4o is the benchmark's reference judge and the one most published numbers use, so it is the headline. The work ran from 10 to 14 September over ten full-500 evaluations; Flash-Lite was the only judge until 13 September, and GPT-4o became the headline after the four-judge comparison. Gemini 3.5 Flash-Lite is our development judge and the strictest of the four we have compared (on one identical set of answers — the previous full run, not this one — four judges spread from 89.0% to 92.2%); it is reported beside GPT-4o on every run. The highest-scoring judge in that comparison (GPT-5.6 Luna) is also the cheapest and made two clear errors in our favour; that combination is exactly what selecting a judge after seeing the score looks like, so it is not the headline. The protocol is fixed in the harness, and the judge snapshot is recorded in every verdict file from the 14 September runs onward — the 13 September four-judge files predate that field, and we have not backfilled it.
- No fact extraction in the harness, and no server-derived facts in the context. Several systems rewrite conversations into atomic facts with an LLM at ingest. The number above is retrieved from stored conversation turns only.
- No prompt tuned to the judge. The answering pipeline was tuned against this benchmark's failures (see above) but never against a judge's verdicts.
- No cherry-picked sample. All 500 questions, all six types, abstention items included. Balanced subsamples project two to three points differently from the full set in both directions; we do not quote them.
- No valid_at. Caura can anchor retrieval at the question date. It scored lower in our A/B, so it is off here; the temporal-reasoning number is the reader doing the arithmetic from session dates.
Where the remaining 42 errors are
Once the evidence is in front of the model, the failures change character. Of the 39 non-abstention errors under GPT-4o, 17 still have a gold turn missing from context (10 partially covered, mostly multi-session questions where one of two or three gold sessions did not make it in; 7 with no gold turn at all, four of them single-session user questions whose one gold session was neither a search hit nor pulled in by expansion), and 22 have every gold turn present and the model still answers wrong. The second group is dominated by three patterns: counting questions where the model hedges between two numbers, abstaining when the fact is there, and multi-session arithmetic. Those are answering-model problems, and they can be worked on against the saved contexts without touching the memory store; the oracle result above says how much is left there.
The other 3 errors are abstention items: questions with no answer in the history where the model answered anyway.
The preference category is where judge disagreement is largest: the same 30 answers score 86.7% under GPT-4o and 93.3% under Flash-Lite, and 76.7% under GPT-4o on the second pass. Rubric-graded questions are the least stable part of this benchmark and we would not draw conclusions from a two-answer difference there.
* The four-judge comparison was run on the previous full pass, whose answers all four judges scored. This run’s answers were scored by the two protocol judges only.
Related reading: Caura benchmarks · What is AI agent memory? · Persistent memory for AI agents