Benchmarks for agent memory
A memory layer earns its place when the history no longer fits in a context window. Public benchmarks at large history sizes are the one way to compare that across products. Here is what BEAM measures, what has been published, and how to read it without being sold to.
What BEAM is
BEAM comes from the paper Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs (Tavakoli et al., arXiv:2510.27246, first posted October 2025, revised February 2026). Its authors generate long, coherent conversations between a user and an assistant, then ask probing questions about them. The benchmark has 100 conversations and 2,000 human-validated questions, at four history sizes: 128K, 500K, 1M and 10M tokens. For the 10M tier, the authors chain ten interlocking conversation plans into one longer narrative.
The paper’s headline finding is the reason the benchmark matters for this guide: even models with 1M-token context windows, with and without retrieval added, struggle as the dialogues get longer. A bigger window is not a memory.
What it tests
Questions target ten memory abilities. Three are new with BEAM (instruction following, event ordering and contradiction resolution):
- Information extraction: recalling facts and details from far back
- Multi-hop reasoning: combining evidence from separate parts of the history
- Information update: using the newest version of a fact that changed
- Temporal reasoning: durations, dates and timelines
- Abstention: declining to answer when the history doesn't say
- Contradiction resolution: noticing and reconciling statements that conflict
- Event ordering: putting things in the order they happened
- Instruction following: keeping to constraints given long before
- Preference following: adapting to preferences that change
- Summarization: condensing information across sessions
For coding agents, several of these map straight onto the failure modes on the compaction page: instruction following is the constraint stated once, information update is the decision that changed, contradiction resolution is two sessions that disagree.
Published results
Every figure below is copied from the source in its row, as that source published it. They are separately published runs, so the models and judges differ; read the next section before comparing rows.
| System | 100K | 500K | 1M | 10M | Run by | Source |
|---|---|---|---|---|---|---|
| OpenCrawl | 90.27% | 87.76% | 89.57% | 83.69% | OpenCrawl (average of three full runs over BEAM's 2,000 questions) | getopencrawl.com |
| Hindsight | 86.2% | 80.1% | 79.1% | 64.1% | As published at the source | agentmemorybenchmark.ai |
| Exabase M-1 | 76.9% | not published | 75% | 68% | Exabase | exabase.io |
| Honcho | 63% | 64.9% | 63.1% | 40.6% | Plastic Labs | plasticlabs.ai |
| mem0 | not published | not published | 64.1% | 48.6% | Mem0 | mem0.ai |
Results pages label the smallest tier “100K”; the paper describes it as 128K tokens.
On these published figures, OpenCrawl is ahead at every size, and its lead grows with the history: 4.1 points over the next result at 100K, 15.7 at 10M. Its figures are the average of three full runs over BEAM’s 2,000 questions using the same public answer prompts as other top published scores, as its benchmarks section states. The pattern worth noticing across every system is the drop at 10M tokens: that tier is where approaches separate.
How to read a result
- Who ran it? A vendor’s own run is evidence, not proof. Look for the method: prompts, model, judge, number of runs. A result that publishes those is checkable; one that doesn’t isn’t.
- Same run or separate runs? A table that gathers each provider’s own published number compares runs with different models and judges. It shows the field, not a controlled experiment. Say so when you quote it.
- Which sizes? A result at 100K is about a history that fits in today’s larger windows. The 1M and 10M tiers are the ones that test memory. Be wary of a product that only publishes its best tier.
- One run or several? LLM-judged scores move between runs. An average of several full runs is worth more than a single run, and the spread matters if two systems are within a few points.
- Which abilities? An overall score can hide a weak ability. If your agents live on constraints and changing decisions, look for instruction following, information update and contradiction resolution in the per-ability breakdown when the source gives one.
- Does it look like your work? BEAM’s histories are user-assistant conversations. A coding agent’s history is mostly tool calls, diffs and logs. Treat a benchmark as evidence about retrieval over long histories, then test on your own sessions.
Other benchmarks
- LongMemEval (arXiv:2410.10813): Tests chat assistants' long-term memory over sustained multi-session conversations. Its histories are far shorter than BEAM's largest tier, so it says less about 10M-token runs.
- LoCoMo (arXiv:2402.17753): Very long-term conversational memory across many sessions between two speakers. Widely reported by memory vendors; also far smaller than BEAM's top tiers.
What benchmarks can’t tell you
BEAM measures one of the six criteria well: scale. It says nothing about whether a memory reaches your other agents, whether agents see each other’s work while it happens, or whether you can run it yourself. That is why the ranking scores those separately, and why a benchmark result alone never decides a rank.