# MemoryAgentBench benchmark — Hanzo AI

> MemoryAgentBench benchmark: Hanzo Context doubles multi-hop substring EM from 33 to 67 on 262k haystacks without LLM memory reads.

[Benchmarks](https://hanzo.ai/benchmarks) / MemoryAgentBench

measured

FactConsolidation: haystacks of 6k to 262k tokens where a later line silently overwrites an earlier one. A typed search over parsed facts resolves the question with no language model reading anything.

[Start on Hanzo Cloud](https://platform.hanzo.ai/)

single-hop

configuration6k

- plain resolver**96.0**
- typed search**100.0**

```
$ node results.mjs
```

regenerates every table on this page from the raw runs

dataset

[MemoryAgentBench](https://huggingface.co/datasets/ai-hyz/MemoryAgentBench)

licence

MIT

introduced by

[Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions](https://arxiv.org/abs/2507.05257) — Hu, Wang, McAuley, UC San Diego, 2025

harness

hanzoai/benchmarks · brain

our paper

[Linkage, Not Similarity](https://github.com/hanzoai/papers/blob/main/site/public/pdfs/hanzo-context-memory.pdf)

## What is measured

FactConsolidation. A haystack is a numbered list of templated facts in which a later line silently replaces an earlier one about the same subject and relation, and nothing marks the update. Questions ask for the current value, single-hop and multi-hop, over haystacks of 6k to 262k tokens. One hundred questions per haystack. The metric is substring exact match, the benchmark’s own; an unanswered question scores zero. Dev is the 6k haystacks, test is 32k, 64k and 262k.

## Results

substring exact match · 6k from dev, 32k / 64k / 262k from test · no reader · pooled = the mean of the four haystacks

Results
| configuration | 6k | 32k | 64k | 262k | pooled |
| --- | --- | --- | --- | --- | --- |
| plain resolver | 96.0 | 97.0 | 95.0 | 94.0 | 95.5 |
| typed search | 100.0 | 98.0 | 98.3 | 97.0 | 98.3 |

## Numbers other systems publish

Quoted with their protocols, because the protocols differ, and none of them re-run here. CAR (arXiv 2606.01435) reports 78.0 single-hop / 30.2 multi-hop pooled over 6k–262k with gpt-4o-mini and 94.8 / 51.5 with gpt-4o, and 82 / 27 and 93 / 41 respectively at 262k. Its own baselines at 262k multi-hop: HippoRAG-v2 5, gpt-4o whole-context 5, BM25 3, Cognee and MemGPT 3, Mem0 2, Zep 3. That is the nearest thing to a like-for-like comparison anywhere on this site — same data, same split, same substring-EM metric — and it is still not one, because CAR reads the haystack with a frontier model and the rows above use no model at all.

```
$ node results.mjs
```

PreviousLoCoMo · All Benchmarks Board · NextLongMemEval

### Related in memory

[LoCoMo](https://hanzo.ai/benchmarks/locomo)[LongMemEval](https://hanzo.ai/benchmarks/longmemeval)[LoCoMo · subject scope](https://hanzo.ai/benchmarks/locomo-subject-scope)[LoCoMo-Conv](https://hanzo.ai/benchmarks/locomo-conv)

- Other benchmarks

[CronQuestions](https://hanzo.ai/benchmarks/cronquestions) · [RepoBench-R](https://hanzo.ai/benchmarks/repobench-r) · [Fleet residency](https://hanzo.ai/benchmarks/fleet) · [Live agent footprint](https://hanzo.ai/benchmarks/goroutine) · [Sandbox cold start](https://hanzo.ai/benchmarks/sandbox) · [Engine prefill on CUDA vs llama.cpp](https://hanzo.ai/benchmarks/inference) · [GPQA-Diamond](https://hanzo.ai/benchmarks/gpqa)

raw runs, plans, traces and scoring: hanzoai/benchmarks · brain · every table above regenerated by `node results.mjs`

## Build on Hanzo Cloud

[Start on Hanzo Cloud](https://platform.hanzo.ai/)[View Models & Pricing](https://hanzo.ai/pricing)

Hanzo AI Cloud
