Benchmarks / LoCoMo · subject scope

Retrieval scoped by subject

A second LoCoMo harness, in Go, with no language model anywhere in it: lexical retrieval, a fixed extractive reader, an arm that reads the graph rather than ranking text, and the control that says which arms were reading it at all. Its numbers do not compare with the engine’s.

dataset

LoCoMo

licence

CC BY-NC 4.0

introduced by

Evaluating Very Long-Term Conversational Memory of LLM Agents — Maharana, Lee, Tulyakov, Bansal, Barbieri, Fang, ACL 2024

harness

hanzoai/semantic · bench/locomo

our paper

Retrieval Scoped by Subject

regenerates every table on this page from the raw runs

Why this is a separate page

This measures the same dataset as LoCoMo and its numbers are not on the same scale, so they are not in the same table. A benchmark page is one protocol, not one dataset, and three things differ here. Retrieval is lexical — IDF-weighted stemmed terms, cosine, no embedding model anywhere. The reader is a fixed extractive one: it picks the sentence covering most of the question’s terms and returns a span, and no language model is called in any arm. And all 1986 questions are used, including the adversarial quarter, reported apart rather than excluded.

The consequence is that absolute F1 here is far below any published model number on LoCoMo, and the harness says so. Put the two tables under one heading and a reader compares them, which would be comparing readers rather than memories. Two pages, each saying what the other is, makes the wrong reading harder than one page with a footnote would.

What is measured

Ten conversations, 272 sessions, 5,882 turns, 1986 questions, scored with LoCoMo’s own metric — ported to Go and verified against the original Python on all 1986 questions and every arm. The question is whether restricting retrieval to the person a question names changes what is found. The flat index embeds whole turns; the scoped one indexes what one person said in one turn and filters by the subject the question names. Both are cosine over the same terms, so the index unit and the filter are the only variables, and a 2×2 separates them.

Results

token F1 · k=5 · abstention floor 0.30 · go run ./bench/locomo at v0.3.0

The effect is on the adversarial half: 0.502 against 0.200 on 446 questions, a 95% interval over conversations of [+0.247, +0.347], at no measurable cost on the 1540 answerable ones (0.142 against 0.144, interval [-0.010, +0.006]). The mechanism is reader-independent and visible in retrieval alone. With the result limit lifted past the size of the conversation, the scoped memory reaches 76% of the annotated evidence for answerable questions and 27% for adversarial ones, where the flat index reaches 99% and 96%. It is selectively blind, and the blindness falls where the trap turns are. The flat index’s retrieval confidence separates answerable from adversarial questions at AUC 0.514, which is chance; the scoped memory’s separates them at 0.698.

The aggregate is degenerate, and that is the result

A system that replies “No information available” to all 1986 questions scores 0.228 on this metric. That is above every arm here that has to find its own evidence — the best of those is 0.223 — and within 0.024 of an oracle handed the dataset’s own annotated evidence, at 0.252. A quarter of LoCoMo’s questions are adversarial and the official metric awards a point for declining them, so any mean over the full set measures the abstention rate before it measures the memory.

So the two halves are reported apart and the aggregate is printed to be discounted. Our own harness printed that row for weeks; this analysis is what removed it. It is also why the engine’s LoCoMo page excludes adversarial questions outright — two different treatments of the same defect, and neither of them an average over the whole set.

Reading the graph instead of ranking text

Everything above ranks text. A third memory does not: path stands on the nodes a question names and follows the relations out of them, so its evidence is chosen by what the graph connects rather than by what the words resemble. On the 282 multi-hop questions it scores 0.032 against 0.020 for subject scope and 0.023 for the flat index — with near-identical evidence recall, 0.222 against 0.228. It is not finding more of the evidence. It is citing a different five turns out of the same reach, and the difference is that the five best turns by wording are five sayings of one thing while the five best relations are five things. It is also better at silence, 0.576 against 0.502, because an assertion about the wrong person is not reached from the right person’s node at all.

the same 1986 questions, same scorer, same settings · go run ./bench/locomo -arms walk at v0.3.0

Four memories, one reader. The reader here is the same extractive one used above: it answers from the best piece or two of what it was handed.

the same four runs, read by stating every piece of evidence returned

One memory, two readers — and the reader is worth more than the memory, in exactly one category. Multi-hop goes 0.023 to 0.066 on the flat index and 0.032 to 0.071 on the walk. The figure that says what kind of bound this is, though, is the oracle: handed the annotated evidence, it goes 0.068 to 0.119. Perfect retrieval read the old way scored 0.068; perfect retrieval read this way scores 0.119. The ceiling moved, so a retrieval result can now show up underneath it — and it is a trade rather than a gain, because single hop falls 0.111 to 0.049 and adversarial falls 0.200 to 0.128. Listing everything you have is free where the answer is a list and expensive where it is not.

The control

A memory that mentions a graph and a memory that reads one produce the same kind of table, and the table cannot tell them apart. So the harness carries a flag that settles it: -blind empties every graph after building it and changes nothing else — same claims, same pieces, same vectors, same subject resolution. An arm that was never reading the graph cannot notice.

go run ./bench/locomo -arms walk -blind at v0.3.0

5 of the 7 arms come back byte-identical — graph, oracle, oracle+set, vector, vector+set — which is the measured form of a claim this page used to make in prose: those memories never read the graph, and the effect they show is the subject filter, not structure and not inference. The two that do read it, path and path+set, go to 100% quiet and no evidence at all, scoring 0.004 on the answerable questions and 0.228 overall — which is what constant abstention scores, and the reason that row is dim above. The comparison is made cell by cell when the capture is taken rather than asserted here, so the sentence cannot outlive the result.

How far to walk

Two and three hops were built and measured, guided and unguided. The second hop moves multi-hop by 0.000 guided and 0.003 unguided and costs 0.029 and 0.258 on the adversarial questions; the third moves nothing at three decimals either way. Reach was never the constraint — one hop from the node a question names already reaches 90.1% of the multi-hop evidence, against 34.0% of the adversarial evidence, which is exactly the trap those questions are built around. So the arm walks one hop and there is no hop setting to turn. Those figures are printed by the same command and are not in the capture, because the harness prints that comparison as a separate table; reach_test.go asserts the shape on the first two conversations, where it runs in six seconds.

Where it loses

Multi-hop barely exists, for any of them. The best multi-hop score on this page is 0.071, from walking the graph and stating everything found, against a ceiling of 0.119 from the annotated evidence read the same way. When the ceiling is 0.119 the reader is still most of the constraint, and a gain from 0.020 to 0.032 should be read as what it is: a small effect with a mechanism, not a solution.

Walking costs single-hop. 0.111 against 0.121 for subject scope and 0.118 for the flat index, on 841 questions — five sixths of the answerable set. Relations are the wrong unit when the answer sits in one turn and the words of the question are in it.

The subject-scope effect is not a graph effect. The 2×2 puts numbers on it: scope alone is worth most of the adversarial gain, the claim-span index unit alone is worth slightly less than nothing, and the largest single part is in their product. That figure is on the index. Presenting the adversarial result as “graph retrieval beats vector retrieval” would be the easiest available misreading, and the control above is what rules it out.

Abstention is a swept parameter, not a property. The floor is 0.30 by choice, and the whole sweep is published because a single threshold is arguable. At matched silence the ordering holds, but a single row quoted without the sweep would misrepresent it.

The paper, and what it predates

Retrieval Scoped by Subject reports the first two tables on this page — the subject-scope result, the degenerate aggregate, the 2×2 and the reach analysis — and its statement that emptying the graph changes nothing is true of every arm it measures. It predates the walk arm. The multi-hop figures here, the Gather reader and the control that separates the arms are newer than the paper and are not in it; cite the paper for the subject-scope result and this page, or the harness, for the rest.

Reproducing it

Go 1.25, no third-party dependencies, and the dataset is fetched on first run. From a fresh clone at v0.3.0: go run ./bench/locomo prints the first table, go run ./bench/locomo -arms walk the walk, go run ./bench/locomo -arms walk -blind the control, go run ./bench/locomo -arms factor the 2×2, go run ./bench/locomo -sweep floor the abstention sweep, and go run ./bench/locomo -stats out/answers.json the constant-abstention row and the intervals. make check rescores the same predictions with the benchmark authors’ own Python and matches to three decimals across all 1986 questions and every arm.

On this side, node scripts/locomo.mjs runs all of it against a checkout and writes what it prints into lib/data/benchmarks-semantic.json, which is what this page renders. Nothing above is transcribed, and the tag, commit b34efcf and dataset digest that produced it travel in the same file.

The other benchmarks

LoCoMo · MemoryAgentBench · LongMemEval · RepoBench-R · LoCoMo-Conv · Fleet residency · Live agent footprint · Sandbox cold start · Inference vs llama.cpp · GPQA-Diamond · all of them, and the head-to-head

harness: hanzoai/semantic · bench/locomo at v0.3.0, MIT · dataset snap-research/locomo sha256 79fa87e90f04… · captured 2026-09-11