Retrieval scoped by subject

A second LoCoMo harness, in Go, with no language model anywhere in it: lexical retrieval and a fixed extractive reader, with retrieval scoped to the person a question names, against a flat index. Its numbers are on their own scale, not the engine’s.

dataset: LoCoMo
licence: CC BY-NC 4.0
introduced by: Evaluating Very Long-Term Conversational Memory of LLM Agents — Maharana, Lee, Tulyakov, Bansal, Barbieri, Fang, ACL 2024
harness: hanzoai/semantic · bench/locomo
our papers: Retrieval Scoped by Subject, Linkage, Not Similarity

go run ./bench/locomo

regenerates every table on this page from the raw runs

Why this is a separate page

This measures the same dataset as LoCoMo and its numbers are not on the same scale, so they are not in the same table. A benchmark page is one protocol, not one dataset, and three things differ here. Retrieval is lexical — IDF-weighted stemmed terms, cosine, no embedding model anywhere. The reader is a fixed extractive one: it picks the sentence covering most of the question’s terms and returns a span, and no language model is called in any arm. And all 1986 questions are used, including the adversarial quarter, reported apart rather than excluded.

What is measured

Ten conversations, 272 sessions, 5,882 turns, 1986 questions, scored with LoCoMo’s own metric — ported to Go and verified against the original Python on all 1986 questions and every arm. The question is whether restricting retrieval to the person a question names changes what is found. The flat index embeds whole turns; the scoped one indexes what one person said in one turn and filters by the subject the question names. Both are cosine over the same terms.

Results

token F1 · k=5 · abstention floor 0.30 · interval over conversations, subject scope minus the flat index · go run ./bench/locomo at v0.3.0

questions	n	subject-scoped F1	flat index F1	paired 95% interval
adversarial	446	0.502	0.200	[+0.247, +0.347]

A quarter of LoCoMo’s questions are adversarial: they ask about something the conversation never said, and the right answer is to decline. Scoping retrieval to the person a question names scores 0.502 on the 446 of them against the flat index’s 0.200, because an assertion about the wrong person is not retrieved for the right one. The mechanism is visible in retrieval alone: the scoped memory’s retrieval confidence separates answerable from adversarial questions at AUC 0.698, against the flat index’s 0.514.

The paper

Retrieval Scoped by Subject reports the subject-scope result on this page, from the same harness.

Reproducing it

Go 1.25, no third-party dependencies, and the dataset is fetched on first run. From a fresh clone at v0.3.0: go run ./bench/locomo prints the table and go run ./bench/locomo -stats out/answers.json the intervals. make check rescores the same predictions with the benchmark authors’ own Python and matches to three decimals across all 1986 questions and every arm.

On this side, node scripts/locomo.mjs runs all of it against a checkout and writes what it prints into lib/data/benchmarks-semantic.json, which is what this page renders. Nothing above is transcribed, and the tag, commit b34efcf and dataset digest that produced it travel in the same file.

Hanzo AI Cloud