LoCoMo and LongMemEval: What Memory Benchmarks Measure

Memory vendors publish LoCoMo and LongMemEval scores, and two numbers on the same benchmark often cannot be compared. A score depends on choices made outside the memory system: which questions were counted, which model wrote the answers and which model graded them.

This post explains what each benchmark contains, how a score is produced, and what a report has to disclose before its number means anything.

What LoCoMo tests

LoCoMo comes from "Evaluating Very Long-Term Conversational Memory of LLM Agents" (Maharana et al., ACL 2024). It released 10 long conversations, each spread over multiple sessions, annotated with questions and reference answers. The questions fall into five categories:

• Single-hop: the answer sits in one place in the conversation.
• Multi-hop: the answer combines information from more than one session.
• Temporal: the answer depends on when something happened, or in what order.
• Open-domain: the answer combines what a speaker said with general knowledge.
• Adversarial: the question is built to mislead, and the right response is that the conversation does not say.

Each category exercises a different part of a memory system. Single-hop is mostly retrieval. Multi-hop needs several memories in the same result. Temporal questions need dates, which only works if the run passes each session's date to the memory system and the system keeps it. Adversarial questions reward a system, and a reader model, that decline to answer.

What LongMemEval tests

LongMemEval comes from "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory" (Wu et al., ICLR 2025). It has 500 questions that test five abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention.

Its question types are single-session-user, single-session-assistant, single-session-preference, knowledge-update, temporal-reasoning and multi-session. Two types and one ability deserve a close look before you trust a score:

• knowledge-update asks about a fact the user later changed, so the right answer is the latest value. A system that keeps every version and ranks an old one first gets these wrong.
• single-session-assistant asks about something the assistant said. A memory design that keeps only facts about the user has nothing to retrieve for these questions.
• Abstention covers questions about something that was never said, where the right response is to decline.

The mix of question types favors some designs over others. A score on the full set and a score that leaves out a type measure different things, even when both carry the benchmark's name.

How a benchmark score is produced

1. Ingest. Each conversation is written into the memory system: turn by turn, session by session, or as facts the system extracts.
2. Retrieve. For each question, the system is queried and returns some context, bounded by a number of results (k) or a token budget.
3. Answer. A reader model receives the question and the retrieved context and writes an answer.
4. Grade. A judge compares the answer with the reference answer and marks it. The score is the share of counted questions marked correct.

Only the first two steps exercise the memory system. The reader and the judge are language models chosen by whoever runs the benchmark, and both move the score.

We run the public LoCoMo and LongMemEval benchmarks against the same public API an integration uses, so ingestion and retrieval go through the routes your application calls rather than an internal shortcut.

Why vendor scores do not compare

Two scores on the same benchmark are comparable only when these choices match:

• The judge. A strict judge accepts only answers that match the reference in substance. A lenient judge also accepts answers that are close or partly right. A binary judge gives no partial credit, and a graded one does.
• The question subset. Leaving out a category changes what is measured and shrinks the denominator. Excluding adversarial questions removes the ones that reward declining to answer.
• The reader model. A stronger reader can answer from weaker context, so a higher score can come from the reader rather than from the memory.
• k and the token budget. More retrieved context makes it more likely that the answer is somewhere in the prompt, at the cost of more tokens per question. A score reported without a token count hides that trade.

In our runs, the same system's LoCoMo score moved by about 15 points between a strict judge and the binary judge used by the protocol Mem0 publishes, which also excludes adversarial questions. The memory system was the same in both runs. The grading and the question set changed.

That is why this post gives no absolute scores. Without the protocol next to it, a number invites comparison with figures produced under different rules.

What a credible report includes

• Subset: which categories or question types were run, and which were left out and why.
• Denominator: how many questions were counted, and whether questions that errored or timed out count as wrong or were dropped.
• Judge: the model, its version and its prompt, and whether grading is binary or graded.
• Reader: the model, its version and the prompt that turns retrieved context into an answer.
• k and tokens: how many results were retrieved per question, and the mean number of context tokens sent to the reader.
• Ingestion: what was written (raw turns, whole sessions or extracted facts) and whether session dates were included.
• Reproducibility: code and configuration that someone outside the vendor can run against the same API.

Our own policy follows the same list: benchmark results are published only with their evaluated subset, denominator, method, configuration, limitations and reproducibility status.

A score with items missing may still be accurate, but you cannot place it next to another score.

Using benchmarks in a decision

Use a benchmark as a filter. A system that does poorly under a disclosed protocol deserves a closer look before you build on it, and a score published without its protocol tells you little either way.

Read the categories, not only the overall number. If your users change facts often, knowledge-update and temporal results matter more to you than open-domain ones. If your assistant needs to recall its own earlier answers, look at single-session-assistant.

When a report leaves out the judge or the denominator, ask the vendor for them before you compare it with anything.

Then test on your own data, because no public benchmark contains your users' facts or questions. How to Evaluate Memory Retrieval on Your Own Data shows how to build an evaluation set from real requests and score it with a short Python script.