# LoCoMo, retrieval scoped by subject — Hanzo AI > A Go LoCoMo harness with no language model in it: scoping retrieval to the person a question names moves adversarial F1 from 0.200 to 0.502, walking the graph moves multi-hop from 0.020 to 0.032, and emptying the graph proves which arms were reading it. [Benchmarks](https://hanzo.ai/benchmarks) / LoCoMo · subject scope measured # Retrieval scoped by subject A second LoCoMo harness, in Go, with no language model anywhere in it: lexical retrieval, a fixed extractive reader, an arm that reads the graph rather than ranking text, and the control that says which arms were reading it at all. Its numbers do not compare with the engine’s. dataset [LoCoMo](https://github.com/snap-research/locomo) licence CC BY-NC 4.0 introduced by [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://arxiv.org/abs/2402.17753) — Maharana, Lee, Tulyakov, Bansal, Barbieri, Fang, ACL 2024 harness [hanzoai/semantic · bench/locomo](https://github.com/hanzoai/semantic/tree/v0.3.0/bench/locomo) our paper [Retrieval Scoped by Subject](https://github.com/hanzoai/papers/tree/main/locomo-subject-scope) `go run ./bench/locomo`regenerates every table on this page from the raw runs ## Why this is a separate page This measures the same dataset as [LoCoMo](https://hanzo.ai/benchmarks/locomo) and its numbers are not on the same scale, so they are not in the same table. A benchmark page is one protocol, not one dataset, and three things differ here. Retrieval is lexical — IDF-weighted stemmed terms, cosine, no embedding model anywhere. The reader is a fixed extractive one: it picks the sentence covering most of the question’s terms and returns a span, and no language model is called in any arm. And all 1986 questions are used, including the adversarial quarter, reported apart rather than excluded. The consequence is that absolute F1 here is far below any published model number on LoCoMo, and the harness says so. Put the two tables under one heading and a reader compares them, which would be comparing readers rather than memories. Two pages, each saying what the other is, makes the wrong reading harder than one page with a footnote would. ## What is measured Ten conversations, 272 sessions, 5,882 turns, 1986 questions, scored with LoCoMo’s own metric — ported to Go and verified against the original Python on all 1986 questions and every arm. The question is whether restricting retrieval to the person a question names changes what is found. The flat index embeds whole turns; the scoped one indexes what one person said in one turn and filters by the subject the question names. Both are cosine over the same terms, so the index unit and the filter are the only variables, and a 2×2 separates them. ## Results token F1 · k=5 · abstention floor 0.30 · `go run ./bench/locomo` at v0.3.0 questions n graph F1 recall quiet graph+edge F1 recall quiet oracle F1 recall quiet vector F1 recall quiet single hop 841 0.121 0.507 24% 0.056 0.507 21% 0.199 1.000 42% 0.118 0.546 20% multi hop 282 0.020 0.228 18% 0.025 0.228 17% 0.068 0.973 44% 0.023 0.194 17% temporal 321 0.338 0.532 23% 0.336 0.532 22% 0.379 0.997 33% 0.357 0.581 16% open domain 96 0.018 0.238 56% 0.018 0.238 55% 0.032 0.951 79% 0.016 0.258 45% adversarial 446 0.502 0.135 50% 0.484 0.135 48% 0.426 1.000 43% 0.200 0.511 20% answerable 1540 0.142 0.444 25% 0.107 0.444 23% 0.202 0.991 43% 0.144 0.471 20% every question — discount this row 1986 0.223 0.375 31% 0.191 0.375 29% 0.252 0.993 43% 0.156 0.480 20% The effect is on the adversarial half: 0.502 against 0.200 on 446 questions, a 95% interval over conversations of [+0.247, +0.347], at no measurable cost on the 1540 answerable ones (0.142 against 0.144, interval [-0.010, +0.006]). The mechanism is reader-independent and visible in retrieval alone. With the result limit lifted past the size of the conversation, the scoped memory reaches 76% of the annotated evidence for answerable questions and 27% for adversarial ones, where the flat index reaches 99% and 96%. It is selectively blind, and the blindness falls where the trap turns are. The flat index’s retrieval confidence separates answerable from adversarial questions at AUC 0.514, which is chance; the scoped memory’s separates them at 0.698. ## The aggregate is degenerate, and that is the result A system that replies “No information available” to all 1986 questions scores 0.228 on this metric. That is above every arm here that has to find its own evidence — the best of those is 0.223 — and within 0.024 of an oracle handed the dataset’s own annotated evidence, at 0.252. A quarter of LoCoMo’s questions are adversarial and the official metric awards a point for declining them, so any mean over the full set measures the abstention rate before it measures the memory. So the two halves are reported apart and the aggregate is printed to be discounted. Our own harness printed that row for weeks; this analysis is what removed it. It is also why [the engine’s LoCoMo page](https://hanzo.ai/benchmarks/locomo) excludes adversarial questions outright — two different treatments of the same defect, and neither of them an average over the whole set. ## Reading the graph instead of ranking text Everything above ranks text. A third memory does not: `path` stands on the nodes a question names and follows the relations out of them, so its evidence is chosen by what the graph connects rather than by what the words resemble. On the 282 multi-hop questions it scores 0.032 against 0.020 for subject scope and 0.023 for the flat index — with near-identical evidence recall, 0.222 against 0.228. It is not finding more of the evidence. It is citing a different five turns out of the same reach, and the difference is that the five best turns by wording are five sayings of one thing while the five best relations are five things. It is also better at silence, 0.576 against 0.502, because an assertion about the wrong person is not reached from the right person’s node at all. the same 1986 questions, same scorer, same settings · `go run ./bench/locomo -arms walk` at v0.3.0 questions n subject-scoped F1 recall quiet walking relations F1 recall quiet flat index F1 recall quiet oracle F1 recall quiet single hop 841 0.121 0.507 24% 0.111 0.467 27% 0.118 0.546 20% 0.199 1.000 42% multi hop 282 0.020 0.228 18% 0.032 0.222 26% 0.023 0.194 17% 0.068 0.973 44% temporal 321 0.338 0.532 23% 0.353 0.512 21% 0.357 0.581 16% 0.379 0.997 33% open domain 96 0.018 0.238 56% 0.016 0.213 61% 0.016 0.258 45% 0.032 0.951 79% adversarial 446 0.502 0.135 50% 0.576 0.130 58% 0.200 0.511 20% 0.426 1.000 43% answerable 1540 0.142 0.444 25% 0.141 0.415 28% 0.144 0.471 20% 0.202 0.991 43% every question — discount this row 1986 0.223 0.375 31% 0.239 0.351 34% 0.156 0.480 20% 0.252 0.993 43% Four memories, one reader. The reader here is the same extractive one used above: it answers from the best piece or two of what it was handed. the same four runs, read by stating every piece of evidence returned questions n flat index F1 recall quiet walking relations F1 recall quiet oracle F1 recall quiet single hop 841 0.089 0.546 12% 0.049 0.467 24% 0.203 1.000 42% multi hop 282 0.066 0.194 13% 0.071 0.222 25% 0.119 0.973 44% temporal 321 0.356 0.581 15% 0.354 0.512 21% 0.379 0.997 33% open domain 96 0.017 0.258 36% 0.024 0.213 60% 0.036 0.951 78% adversarial 446 0.128 0.511 13% 0.563 0.130 56% 0.426 1.000 43% answerable 1540 0.136 0.471 15% 0.115 0.415 26% 0.214 0.991 43% every question — discount this row 1986 0.134 0.480 14% 0.216 0.351 33% 0.262 0.993 43% One memory, two readers — and the reader is worth more than the memory, in exactly one category. Multi-hop goes 0.023 to 0.066 on the flat index and 0.032 to 0.071 on the walk. The figure that says what kind of bound this is, though, is the oracle: handed the annotated evidence, it goes 0.068 to 0.119. Perfect retrieval read the old way scored 0.068; perfect retrieval read this way scores 0.119. The ceiling moved, so a retrieval result can now show up underneath it — and it is a trade rather than a gain, because single hop falls 0.111 to 0.049 and adversarial falls 0.200 to 0.128. Listing everything you have is free where the answer is a list and expensive where it is not. ## The control A memory that mentions a graph and a memory that reads one produce the same kind of table, and the table cannot tell them apart. So the harness carries a flag that settles it: `-blind` empties every graph after building it and changes nothing else — same claims, same pieces, same vectors, same subject resolution. An arm that was never reading the graph cannot notice. `go run ./bench/locomo -arms walk -blind` at v0.3.0 arm under an emptied graph multi-hop F1 evidence recall quiet graph every figure identical 0.020 → 0.020 0.228 → 0.228 18% → 18% oracle every figure identical 0.068 → 0.068 0.973 → 0.973 44% → 44% oracle+set every figure identical 0.119 → 0.119 0.973 → 0.973 44% → 44% path collapses 0.032 → 0.007 0.222 → 0.000 26% → 100% path+set collapses 0.071 → 0.007 0.222 → 0.000 25% → 100% vector every figure identical 0.023 → 0.023 0.194 → 0.194 17% → 17% vector+set every figure identical 0.066 → 0.066 0.194 → 0.194 13% → 13% 5 of the 7 arms come back byte-identical — graph, oracle, oracle+set, vector, vector+set — which is the measured form of a claim this page used to make in prose: those memories never read the graph, and the effect they show is the subject filter, not structure and not inference. The two that do read it, path and path+set, go to 100% quiet and no evidence at all, scoring 0.004 on the answerable questions and 0.228 overall — which is what constant abstention scores, and the reason that row is dim above. The comparison is made cell by cell when the capture is taken rather than asserted here, so the sentence cannot outlive the result. ## How far to walk Two and three hops were built and measured, guided and unguided. The second hop moves multi-hop by 0.000 guided and 0.003 unguided and costs 0.029 and 0.258 on the adversarial questions; the third moves nothing at three decimals either way. Reach was never the constraint — one hop from the node a question names already reaches 90.1% of the multi-hop evidence, against 34.0% of the adversarial evidence, which is exactly the trap those questions are built around. So the arm walks one hop and there is no hop setting to turn. Those figures are printed by the same command and are not in the capture, because the harness prints that comparison as a separate table; `reach_test.go` asserts the shape on the first two conversations, where it runs in six seconds. ## Where it loses Multi-hop barely exists, for any of them. The best multi-hop score on this page is 0.071, from walking the graph and stating everything found, against a ceiling of 0.119 from the annotated evidence read the same way. When the ceiling is 0.119 the reader is still most of the constraint, and a gain from 0.020 to 0.032 should be read as what it is: a small effect with a mechanism, not a solution. Walking costs single-hop. 0.111 against 0.121 for subject scope and 0.118 for the flat index, on 841 questions — five sixths of the answerable set. Relations are the wrong unit when the answer sits in one turn and the words of the question are in it. The subject-scope effect is not a graph effect. The 2×2 puts numbers on it: scope alone is worth most of the adversarial gain, the claim-span index unit alone is worth slightly less than nothing, and the largest single part is in their product. That figure is on [the index](https://hanzo.ai/benchmarks). Presenting the adversarial result as “graph retrieval beats vector retrieval” would be the easiest available misreading, and the control above is what rules it out. Abstention is a swept parameter, not a property. The floor is 0.30 by choice, and the whole sweep is published because a single threshold is arguable. At matched silence the ordering holds, but a single row quoted without the sweep would misrepresent it. ## The paper, and what it predates [Retrieval Scoped by Subject](https://github.com/hanzoai/papers/tree/main/locomo-subject-scope) reports the first two tables on this page — the subject-scope result, the degenerate aggregate, the 2×2 and the reach analysis — and its statement that emptying the graph changes nothing is true of every arm it measures. It predates the walk arm. The multi-hop figures here, the Gather reader and the control that separates the arms are newer than the paper and are not in it; cite the paper for the subject-scope result and this page, or the harness, for the rest. ## Reproducing it Go 1.25, no third-party dependencies, and the dataset is fetched on first run. From a fresh clone at v0.3.0: `go run ./bench/locomo` prints the first table, `go run ./bench/locomo -arms walk` the walk, `go run ./bench/locomo -arms walk -blind` the control, `go run ./bench/locomo -arms factor` the 2×2, `go run ./bench/locomo -sweep floor` the abstention sweep, and `go run ./bench/locomo -stats out/answers.json` the constant-abstention row and the intervals. `make check` rescores the same predictions with the benchmark authors’ own Python and matches to three decimals across all 1986 questions and every arm. On this side, `node scripts/locomo.mjs` runs all of it against a checkout and writes what it prints into `lib/data/benchmarks-semantic.json`, which is what this page renders. Nothing above is transcribed, and the tag, commit b34efcf and dataset digest that produced it travel in the same file. ## The other benchmarks [LoCoMo](https://hanzo.ai/benchmarks/locomo) · [MemoryAgentBench](https://hanzo.ai/benchmarks/memoryagentbench) · [LongMemEval](https://hanzo.ai/benchmarks/longmemeval) · [RepoBench-R](https://hanzo.ai/benchmarks/repobench-r) · [LoCoMo-Conv](https://hanzo.ai/benchmarks/locomo-conv) · [Fleet residency](https://hanzo.ai/benchmarks/fleet) · [Live agent footprint](https://hanzo.ai/benchmarks/goroutine) · [Sandbox cold start](https://hanzo.ai/benchmarks/sandbox) · [Inference vs llama.cpp](https://hanzo.ai/benchmarks/inference) · [GPQA-Diamond](https://hanzo.ai/benchmarks/gpqa) · [all of them, and the head-to-head](https://hanzo.ai/benchmarks) harness: [hanzoai/semantic · bench/locomo](https://github.com/hanzoai/semantic/tree/v0.3.0/bench/locomo) at v0.3.0, MIT · dataset snap-research/locomo sha256 79fa87e90f04… · captured 2026-09-11