# MemoryAgentBench benchmark — Hanzo AI > Hanzo Context on MemoryAgentBench FactConsolidation: multi-hop substring EM from 33 to 67 on the 262k haystack with no language model reading the memory, and the analysis of why 262k stops exactly there. [Benchmarks](https://hanzo.ai/benchmarks) / MemoryAgentBench measured # MemoryAgentBench FactConsolidation: haystacks of 6k to 262k tokens where a later line silently overwrites an earlier one. A typed search over parsed facts resolves the question with no language model reading anything. dataset [MemoryAgentBench](https://huggingface.co/datasets/ai-hyz/MemoryAgentBench) licence MIT introduced by [Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions](https://arxiv.org/abs/2507.05257) — Hu, Wang, McAuley, UC San Diego, 2025 harness [hanzoai/cloud · bench/brain](https://github.com/hanzoai/cloud/tree/main/bench/brain/mab) `node results.mjs`regenerates every table on this page from the raw runs ## What is measured FactConsolidation. A haystack is a numbered list of templated facts in which a later line silently replaces an earlier one about the same subject and relation, and nothing marks the update. Questions ask for the current value, single-hop and multi-hop, over haystacks of 6k to 262k tokens. One hundred questions per haystack. The metric is substring exact match, the benchmark’s own; an unanswered question scores zero. Dev is the 6k haystacks, test is 32k, 64k and 262k. No language model reads the memory or writes the answer in any row here. The question is compiled into a plan sketch and a deterministic typed search over the parsed fact store resolves it — 45 templates, no line unparsed at any size, the supersession chain keyed by subject and relation. That is why these rows are not comparable to a pipeline that reads the haystack with a frontier model, and it is also the point: the retrieval is doing the work. ## Results substring exact match · 6k from dev, 32k / 64k / 262k from test · no reader · pooled = the mean of the four haystacks ### single-hop configuration 6k 32k 64k 262k pooled plain resolver 96.0 97.0 95.0 94.0 95.5 typed search 100.0 98.0 98.0 97.0 98.3 ### multi-hop configuration 6k 32k 64k 262k pooled plain resolver 76.0 59.0 59.0 33.0 56.8 typed search 93.0 89.0 89.0 67.0 84.5 Typed search pools to 98.3 single-hop and 84.5 multi-hop against the plain resolver’s 95.5 and 56.8. The largest single move is 262k multi-hop, from 33.0 to 67.0. ## Where it stops The lane that does not move is 262k multi-hop, at 67.0, and the harness says why rather than leaving it as a number. A gold path exists in the parsed store for 98 of 100 questions there, so this is not a parsing failure. But 29 of the 35 misses want an earlier version of a key that a later line overwrites, and the harness’s own ceiling table puts a latest-version rule at exactly 67 on that haystack — which is where the search sits. Six signals a store could read to tell those apart — object degree, whether the object has facts of its own, corroboration back to the subject, contested keys, serial distance, list position — were each tested against those hops and none separates the edit the gold follows from the edit it ignores. Under this protocol the remaining error at 262k is not a retrieval failure. It is unidentifiable from the haystack. ## Numbers other systems publish Quoted with their protocols, because the protocols differ, and none of them re-run here. CAR (arXiv 2606.01435) reports 78.0 single-hop / 30.2 multi-hop pooled over 6k–262k with gpt-4o-mini and 94.8 / 51.5 with gpt-4o, and 82 / 27 and 93 / 41 respectively at 262k. Its own baselines at 262k multi-hop: HippoRAG-v2 5, gpt-4o whole-context 5, BM25 3, Cognee and MemGPT 3, Mem0 2, Zep 3. That is the closest thing to a like-for-like comparison anywhere on this site — same data, same split, same substring-EM metric — and it is still not one, because CAR reads the haystack with a frontier model and the rows above use no model at all. Pith reports 68.0 substring EM on multi-hop at 262k on a vendor page that is not peer reviewed. Its per-lane figures are 95 / 83 / 84 / 68 at 6k / 32k / 64k / 262k, against typed search’s 93.0 / 89.0 / 89.0 / 67.0 — above at 32k and 64k, below at 6k and 262k. Neither the protocol nor the runs behind those figures are published, so a row-by-row reading is the most that can be said of them. ## The other benchmarks [LoCoMo](https://hanzo.ai/benchmarks/locomo) · [LongMemEval](https://hanzo.ai/benchmarks/longmemeval) · [RepoBench-R](https://hanzo.ai/benchmarks/repobench-r) · [LoCoMo · subject scope](https://hanzo.ai/benchmarks/locomo-subject-scope) · [LoCoMo-Conv](https://hanzo.ai/benchmarks/locomo-conv) · [Fleet residency](https://hanzo.ai/benchmarks/fleet) · [Live agent footprint](https://hanzo.ai/benchmarks/goroutine) · [Sandbox cold start](https://hanzo.ai/benchmarks/sandbox) · [Inference vs llama.cpp](https://hanzo.ai/benchmarks/inference) · [GPQA-Diamond](https://hanzo.ai/benchmarks/gpqa) · [all of them, and the head-to-head](https://hanzo.ai/benchmarks) raw runs, plans, traces and scoring: [hanzoai/cloud · bench/brain](https://github.com/hanzoai/cloud/tree/main/bench/brain/mab) · every table above regenerated by `node results.mjs`