Retrieval scoped by subject
A second LoCoMo harness, in Go, with no language model anywhere in it: lexical retrieval and a fixed extractive reader, with retrieval scoped to the person a question names, against a flat index. Its numbers are on their own scale, not the engine’s.
go run ./bench/locomo
regenerates every table on this page from the raw runs
- dataset
- LoCoMo
- licence
- CC BY-NC 4.0
- introduced by
- Evaluating Very Long-Term Conversational Memory of LLM Agents — Maharana, Lee, Tulyakov, Bansal, Barbieri, Fang, ACL 2024
Why this is a separate page
This measures the same dataset as LoCoMo and its numbers are not on the same scale, so they are not in the same table. A benchmark page is one protocol, not one dataset, and three things differ here. Retrieval is lexical — IDF-weighted stemmed terms, cosine, no embedding model anywhere. The reader is a fixed extractive one: it picks the sentence covering most of the question’s terms and returns a span, and no language model is called in any arm. And all 1986 questions are used, including the adversarial quarter, reported apart rather than excluded.
What is measured
Ten conversations, 272 sessions, 5,882 turns, 1986 questions, scored with LoCoMo’s own metric — ported to Go and verified against the original Python on all 1986 questions and every arm. The question is whether restricting retrieval to the person a question names changes what is found. The flat index embeds whole turns; the scoped one indexes what one person said in one turn and filters by the subject the question names. Both are cosine over the same terms.
Results
token F1 · k=5 · abstention floor 0.30 · interval over conversations, subject scope minus the flat index · go run ./bench/locomo at v0.3.0
Results
| questions | n | subject-scoped F1 | flat index F1 | paired 95% interval |
|---|---|---|---|---|
| adversarial | 446 | 0.502 | 0.200 | [+0.247, +0.347] |
A quarter of LoCoMo’s questions are adversarial: they ask about something the conversation never said, and the right answer is to decline. Scoping retrieval to the person a question names scores 0.502 on the 446 of them against the flat index’s 0.200, because an assertion about the wrong person is not retrieved for the right one. The mechanism is visible in retrieval alone: the scoped memory’s retrieval confidence separates answerable from adversarial questions at AUC 0.698, against the flat index’s 0.514.
Reproducing it
Go 1.25, no third-party dependencies, and the dataset is fetched on first run. From a fresh clone at v0.3.0: go run ./bench/locomo prints the table and go run ./bench/locomo -stats out/answers.json the intervals. make check rescores the same predictions with the benchmark authors’ own Python and matches to three decimals across all 1986 questions and every arm.
Reproducing it
On this side, node scripts/locomo.mjs runs all of it against a checkout and writes what it prints into lib/data/benchmarks-semantic.json, which is what this page renders. Nothing above is transcribed, and the tag, commit b34efcf and dataset digest that produced it travel in the same file.
The paper
Retrieval Scoped by Subject reports the subject-scope result on this page, from the same harness.
go run ./bench/locomo
PreviousRepoBench-R · All Benchmarks Board · NextLoCoMo-Conv
Related in memory
LoCoMoMemoryAgentBenchLongMemEvalLoCoMo-Conv- Other benchmarks
CronQuestions · RepoBench-R · Fleet residency · Live agent footprint · Sandbox cold start · Engine prefill on CUDA vs llama.cpp · GPQA-Diamond
harness: hanzoai/semantic · bench/locomo at v0.3.0, MIT · dataset snap-research/locomo sha256 79fa87e90f04… · captured 2026-09-11
Build on Hanzo Cloud
Hanzo AI Cloud