Benchmarks / MemoryAgentBench
measuredMemoryAgentBench
FactConsolidation: haystacks of 6k to 262k tokens where a later line silently overwrites an earlier one. A typed search over parsed facts resolves the question with no language model reading anything.
- dataset
- MemoryAgentBench
- licence
- MIT
- introduced by
- Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions — Hu, Wang, McAuley, UC San Diego, 2025
- harness
- hanzoai/benchmarks · brain
- our paper
- Linkage, Not Similarity
node results.mjsregenerates every table on this page from the raw runs
What is measured
FactConsolidation. A haystack is a numbered list of templated facts in which a later line silently replaces an earlier one about the same subject and relation, and nothing marks the update. Questions ask for the current value, single-hop and multi-hop, over haystacks of 6k to 262k tokens. One hundred questions per haystack. The metric is substring exact match, the benchmark’s own; an unanswered question scores zero. Dev is the 6k haystacks, test is 32k, 64k and 262k.
No language model reads the memory or writes the answer in any row here. The question is compiled into a plan sketch and a deterministic typed search over the parsed fact store resolves it — 45 templates, no line unparsed at any size, the supersession chain keyed by subject and relation. That is why these rows are not comparable to a pipeline that reads the haystack with a frontier model, and it is also the point: the retrieval is doing the work.
Results
substring exact match · 6k from dev, 32k / 64k / 262k from test · no reader · pooled = the mean of the four haystacks
single-hop
| configuration | 6k | 32k | 64k | 262k | pooled |
|---|---|---|---|---|---|
| plain resolver | 96.0 | 97.0 | 95.0 | 94.0 | 95.5 |
| typed search | 100.0 | 98.0 | 98.0 | 97.0 | 98.3 |
multi-hop
| configuration | 6k | 32k | 64k | 262k | pooled |
|---|---|---|---|---|---|
| plain resolver | 76.0 | 59.0 | 59.0 | 33.0 | 56.8 |
| typed search | 93.0 | 89.0 | 89.0 | 67.0 | 84.5 |
Typed search pools to 98.3 single-hop and 84.5 multi-hop against the plain resolver’s 95.5 and 56.8. The largest single move is 262k multi-hop, from 33.0 to 67.0.
Numbers other systems publish
Quoted with their protocols, because the protocols differ, and none of them re-run here. CAR (arXiv 2606.01435) reports 78.0 single-hop / 30.2 multi-hop pooled over 6k–262k with gpt-4o-mini and 94.8 / 51.5 with gpt-4o, and 82 / 27 and 93 / 41 respectively at 262k. Its own baselines at 262k multi-hop: HippoRAG-v2 5, gpt-4o whole-context 5, BM25 3, Cognee and MemGPT 3, Mem0 2, Zep 3. That is the nearest thing to a like-for-like comparison anywhere on this site — same data, same split, same substring-EM metric — and it is still not one, because CAR reads the haystack with a frontier model and the rows above use no model at all.
Pith publishes per-lane multi-hop figures on a vendor page: 83 / 84 at 32k and 64k, where typed search scores 89.0 / 89.0.
raw runs, plans, traces and scoring: hanzoai/benchmarks · brain · every table above regenerated by node results.mjs