Benchmarks / LoCoMo

LoCoMo

Ten long conversations, 1,540 questions with annotated evidence turns. Hanzo Context measured against cosine over turns and against the annotated gold turns, on the same split, embedder, reader and k.

dataset

LoCoMo

licence

CC BY-NC 4.0

introduced by

Evaluating Very Long-Term Conversational Memory of LLM Agents — Maharana, Lee, Tulyakov, Bansal, Barbieri, Fang, ACL 2024

harness

hanzoai/cloud · bench/brain

our paper

Hops Do Not Fix Multi-Hop Retrieval

regenerates every table on this page from the raw runs

What is measured

Ten conversations, 5,882 turns, 1,540 questions, each annotated with the turns that answer it. Retrieval is scored against those turns; answering is scored by token F1 and exact match under LoCoMo’s own normalisation. Configurations are chosen on conversations 0–2 and frozen at a commit before conversations 3–9 run once. Adversarial questions are excluded, as Mem0 and Zep exclude them, and the reason is arithmetic rather than convenience: a system that replies “No information available” to every question scores 0.228 on the full set, above every arm that has to find its own evidence, because the metric awards a point for declining a trap. Any aggregate over the whole set rewards silence, so there is no aggregate over the whole set here.

Retrieval

Each row adds one candidate generator — BM25, the atomic fact index, canonical entities, the timeline, adjacency, one hop of typed expansion, then a second retrieval hop asked with what the first resolved. On the held-out split multi-hop ALL@20 goes from 18.8 to 37.5 and R@20 from 70.9 to 80.7. It costs candidates and time: 77 candidates examined against 40, and 1.67 ms against 0.55 ms — three times the retrieval latency, on a budget three orders of magnitude below the reader that consumes the result. The greyed rows are the shapes that lost, kept because a design is only informative next to what it was chosen over.

Answers

The same questions, answered by a reader from the retrieved context. Within a reader the rows differ only in what was retrieved: cosine over turns, the engine, or the annotated gold turns. That last row is the ceiling and it is the column to read first, because it separates retrieval’s share of the error from the reader’s.

The comparison below is gemma4:31b on the 208 multi-hop questions of the test split, because it is the one that finished: the same reader answered every one of them under all three policies. Cosine reaches 36.3 token F1, the engine 39.0, and the annotated gold turns 50.8. So retrieval is worth 2.7 points here and perfect retrieval would be worth about 11.8 more — the larger part of the remaining multi-hop error is the reader, not what was retrieved.

3 readers finished the same questions from the same context, which puts a number on that share: gemma4:31b 39.0, enso-flash 43.1, gpt-oss-120b 43.1 — a spread of 4.1 points from the choice of reader alone, against the 2.7 points retrieval contributes. That is why every table here names its reader, and why a score quoted without one is not a score.

Where it loses

The multi-hop answer gain does not clear its interval. Retrieval’s multi-hop ALL@20 roughly doubles — 18.8 to 37.5 — and the answer gain that follows it, 36.3 to 39.0, sits inside the bootstrap interval of the baseline. Two marginal intervals that overlap do not establish that there is no difference, and they do not establish that there is one. Finding the right turns is not the same problem as reasoning over them, and this benchmark says so.

Open-domain answers are below the baseline. On the 92 open-domain questions the earlier four-term re-rank scores 14.9 token F1 against cosine’s 17.7, on intervals wide enough that the ordering is all the table establishes. It is the one row where a shipped configuration is worse than the thing it replaced, and it is in the table above rather than in a footnote.

The full-question-set runs did not finish. The qwen3.6:35b-a3b runs are the only ones that answer all four question categories rather than multi-hop alone, and each stopped 11 questions short of its declared set. Their tables are above, dimmed, with both counts on them. They are not in the comparison, they carry the only open-domain figures on this page — which is why that loss is quoted from them and labelled — and the honest reading is that the whole-set answer picture on LoCoMo is not settled here yet.

The other LoCoMo harness

hanzoai/semantic · bench/locomo measures the same dataset and its numbers do not belong beside these. It uses lexical retrieval with no embedding model, a fixed extractive reader with no language model anywhere, and all 1,986 questions including the adversarial quarter, reported apart. Its absolute F1 is far below anything on this page for that reason, and comparing the two would be comparing readers, not memories. It has its own page and its own paper.

The other benchmarks

MemoryAgentBench · LongMemEval · RepoBench-R · LoCoMo · subject scope · LoCoMo-Conv · Fleet residency · Live agent footprint · Sandbox cold start · Inference vs llama.cpp · GPQA-Diamond · all of them, and the head-to-head

raw runs, traces, prompts and scoring: hanzoai/cloud · bench/brain · protocol in METHOD.md · every table above regenerated by node results.mjs