# LoCoMo benchmark — Hanzo AI > Hanzo Context on LoCoMo: multi-hop retrieval ALL@20 from 18.8 to 37.5 on the held-out split, answers from four readers against the annotated gold-turn ceiling, and the rows that lost. Every number regenerated by one public command. [Benchmarks](https://hanzo.ai/benchmarks) / LoCoMo measured # LoCoMo Ten long conversations, 1,540 questions with annotated evidence turns. Hanzo Context measured against cosine over turns and against the annotated gold turns, on the same split, embedder, reader and k. dataset [LoCoMo](https://github.com/snap-research/locomo) licence CC BY-NC 4.0 introduced by [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://arxiv.org/abs/2402.17753) — Maharana, Lee, Tulyakov, Bansal, Barbieri, Fang, ACL 2024 harness [hanzoai/cloud · bench/brain](https://github.com/hanzoai/cloud/tree/main/bench/brain) our paper [Hops Do Not Fix Multi-Hop Retrieval](https://github.com/hanzoai/papers/tree/main/hanzo-multi-hop-retrieval) `node results.mjs`regenerates every table on this page from the raw runs ## What is measured Ten conversations, 5,882 turns, 1,540 questions, each annotated with the turns that answer it. Retrieval is scored against those turns; answering is scored by token F1 and exact match under LoCoMo’s own normalisation. Configurations are chosen on conversations 0–2 and frozen at a commit before conversations 3–9 run once. Adversarial questions are excluded, as Mem0 and Zep exclude them, and the reason is arithmetic rather than convenience: a system that replies “No information available” to every question scores 0.228 on the full set, above every arm that has to find its own evidence, because the metric awards a point for declining a trap. Any aggregate over the whole set rewards silence, so there is no aggregate over the whole set here. ## Retrieval Each row adds one candidate generator — BM25, the atomic fact index, canonical entities, the timeline, adjacency, one hop of typed expansion, then a second retrieval hop asked with what the first resolved. On the held-out split multi-hop ALL@20 goes from 18.8 to 37.5 and R@20 from 70.9 to 80.7. It costs candidates and time: 77 candidates examined against 40, and 1.67 ms against 0.55 ms — three times the retrieval latency, on a budget three orders of magnitude below the reader that consumes the result. The greyed rows are the shapes that lost, kept because a design is only informative next to what it was chosen over. ### all ten conversations · zen-embedding-0.6b · k=20 facts LoCoMo’s own observation annotations · weights frozen on dev at 33d0f8749bd1 · ALL = every annotated turn inside the top 20; ANY = at least one; R@20 = |retrieved ∩ gold| / |gold| configuration multi-hop ALL multi-hop ANY single-hop ALL temporal ALL open-domain ALL R@k nDCG@k supported candidates tokens/q p50 ms semantic only 22.7 79.8 78.2 77.3 33.7 71.9 49.6 81.0 40 660 0.55 +lexical 22.7 78.0 82.5 78.8 37.0 74.6 53.2 83.3 52 665 0.61 +facts 36.9 88.3 82.3 82.6 42.4 78.2 59.6 86.4 71 787 0.83 +adjacency 39.0 88.7 84.2 83.5 44.6 79.7 61.4 87.8 74 782 0.83 +iterative hops 39.0 87.9 86.8 84.4 45.7 80.9 62.6 88.0 76 788 1.65 failed: global RRF 33.7 89.0 86.8 83.8 40.2 80.5 56.2 89.0 137 762 1.73 ### all ten conversations · all-MiniLM-L6-v2 · k=10 facts LoCoMo’s own observation annotations · weights frozen on dev at 33d0f8749bd1 · ALL = every annotated turn inside the top 10; ANY = at least one; R@10 = |retrieved ∩ gold| / |gold| configuration multi-hop ALL multi-hop ANY single-hop ALL temporal ALL open-domain ALL R@k nDCG@k supported candidates tokens/q p50 ms semantic only 9.6 49.6 51.4 46.1 16.3 45.3 31.1 53.1 40 283 0.25 +lexical 13.8 58.9 67.2 62.0 23.9 59.2 43.7 67.6 54 301 0.31 +facts 20.9 73.4 73.1 72.0 25.0 66.7 52.0 75.6 73 364 0.42 +adjacency 21.6 74.5 74.6 72.6 27.2 67.7 53.3 76.3 77 361 0.42 +iterative hops 24.1 78.0 77.5 73.5 27.2 70.1 55.8 78.5 78 377 0.76 failed: global RRF 20.2 74.8 73.4 74.1 28.3 67.2 48.6 76.1 143 382 0.43 ### test split, conversations 3–9 · zen-embedding-0.6b · k=20 facts LoCoMo’s own observation annotations · weights frozen on dev at 33d0f8749bd1 · ALL = every annotated turn inside the top 20; ANY = at least one; R@20 = |retrieved ∩ gold| / |gold| configuration multi-hop ALL multi-hop ANY single-hop ALL temporal ALL open-domain ALL R@k nDCG@k supported candidates tokens/q p50 ms semantic only 18.8 80.3 78.3 72.3 35.6 70.9 48.8 80.5 40 651 0.55 +lexical 21.2 78.4 82.8 73.6 38.4 73.9 52.9 83.2 53 656 0.62 +facts 35.1 88.5 83.0 78.8 43.8 77.9 58.7 86.4 71 771 1.15 +entities 35.1 88.5 83.0 78.8 43.8 77.9 58.7 86.4 71 771 1.10 +timeline 34.6 88.5 83.3 79.7 45.2 78.2 59.9 86.8 71 769 1.04 +adjacency 37.5 88.5 84.6 80.1 45.2 79.1 60.5 87.5 75 767 1.03 +typed graph 37.5 88.5 84.6 80.1 45.2 79.1 60.5 87.5 75 767 0.98 +iterative hops 37.5 88.0 87.7 81.4 45.2 80.7 61.6 87.8 77 771 1.67 failed: PRF 19.7 75.5 72.4 69.3 35.6 66.4 46.3 75.8 634 665 1.10 failed: chain search 37.5 88.0 83.8 78.8 45.2 78.5 56.7 86.3 79 764 1.64 failed: surface entities 35.6 88.0 83.0 78.8 43.8 77.9 58.2 86.2 82 773 0.98 failed: global RRF 32.2 88.5 86.7 80.1 38.4 79.6 55.5 88.4 139 751 1.20 ## Answers The same questions, answered by a reader from the retrieved context. Within a reader the rows differ only in what was retrieved: cosine over turns, the engine, or the annotated gold turns. That last row is the ceiling and it is the column to read first, because it separates retrieval’s share of the error from the reader’s. The comparison below is gemma4:31b on the 208 multi-hop questions of the test split, because it is the one that finished: the same reader answered every one of them under all three policies. Cosine reaches 36.3 token F1, the engine 39.0, and the annotated gold turns 50.8. So retrieval is worth 2.7 points here and perfect retrieval would be worth about 11.8 more — the larger part of the remaining multi-hop error is the reader, not what was retrieved. 3 readers finished the same questions from the same context, which puts a number on that share: gemma4:31b 39.0, enso-flash 43.1, gpt-oss-120b 43.1 — a spread of 4.1 points from the choice of reader alone, against the 2.7 points retrieval contributes. That is why every table here names its reader, and why a score quoted without one is not a score. ### locomo-all-single-k20-gemma4-31b k=20 · reader gemma4:31b · 282 of 282 answered · commit ddc076f5dd0506ce50428416c336e45c4a7ada73 · run locomo-all-single-k20-gemma4-31b questions n in set F1 EM tokens/q all 282 282 37.6[33.6, 41.5] 9.9[6.7, 13.5] 835 all:multi-hop 282 282 37.6[33.6, 41.5] 9.9[6.7, 13.5] 835 dev 74 74 41.1[33.2, 49.0] 6.8[1.4, 12.2] 831 dev:multi-hop 74 74 41.1[33.2, 49.0] 6.8[1.4, 12.2] 831 test 208 208 36.3[31.5, 40.9] 11.1[7.2, 15.4] 836 test:multi-hop 208 208 36.3[31.5, 40.9] 11.1[7.2, 15.4] 836 ### locomo-all-context-k20-gemma4-31b k=20 · reader gemma4:31b · 282 of 282 answered · commit ddc076f5dd0506ce50428416c336e45c4a7ada73 · run locomo-all-context-k20-gemma4-31b questions n in set F1 EM tokens/q all 282 282 40.5[36.4, 44.3] 9.9[6.7, 13.5] 971 all:multi-hop 282 282 40.5[36.4, 44.3] 9.9[6.7, 13.5] 971 dev 74 74 44.7[36.8, 53.2] 9.5[4.1, 17.6] 984 dev:multi-hop 74 74 44.7[36.8, 53.2] 9.5[4.1, 17.6] 984 test 208 208 39.0[34.7, 43.6] 10.1[6.3, 14.4] 967 test:multi-hop 208 208 39.0[34.7, 43.6] 10.1[6.3, 14.4] 967 ### locomo-all-oracle-k20-gemma4-31b k=20 · reader gemma4:31b · 282 of 282 answered · commit ddc076f5dd0506ce50428416c336e45c4a7ada73 · run locomo-all-oracle-k20-gemma4-31b questions n in set F1 EM tokens/q all 282 282 53.0[49.0, 57.0] 16.0[12.1, 20.2] 169 all:multi-hop 282 282 53.0[49.0, 57.0] 16.0[12.1, 20.2] 169 dev 74 74 59.2[51.9, 67.0] 13.5[6.8, 21.6] 135 dev:multi-hop 74 74 59.2[51.9, 67.0] 13.5[6.8, 21.6] 135 test 208 208 50.8[46.0, 55.7] 16.8[12.0, 21.6] 180 test:multi-hop 208 208 50.8[46.0, 55.7] 16.8[12.0, 21.6] 180 ### locomo-all-context-k20-enso-flash k=20 · reader enso-flash · 282 of 282 answered · commit 9eab30f73493a28244d1cd00b3fa53b83fdfba3c · run locomo-all-context-k20-enso-flash questions n in set supported F1 EM tokens/q all 282 1536 13.5 45.1[41.2, 49.1] 13.8[9.9, 18.1] 971 dev 74 383 13.5 50.9[42.8, 58.6] 14.9[6.8, 23.0] 984 test 208 1153 13.5 43.1[37.9, 47.5] 13.5[9.1, 18.3] 967 ### locomo-all-context-k20-gpt-oss-120b k=20 · reader gpt-oss-120b · 282 of 282 answered · commit 9eab30f73493a28244d1cd00b3fa53b83fdfba3c · run locomo-all-context-k20-gpt-oss-120b questions n in set supported F1 EM tokens/q all 282 1536 13.5 45.3[41.6, 49.5] 13.5[9.9, 17.4] 971 dev 74 383 13.5 51.5[42.9, 59.4] 14.9[6.8, 23.0] 984 test 208 1153 13.5 43.1[38.4, 47.8] 13.0[8.7, 17.3] 967 ### locomo-all-cer-k20-gemma4-31b k=20 · reader gemma4:31b · 282 of 282 answered · commit ddc076f5dd0506ce50428416c336e45c4a7ada73 · run locomo-all-cer-k20-gemma4-31b questions n in set F1 EM tokens/q all 282 282 40.7[36.7, 44.9] 11.0[7.4, 14.5] 968 all:multi-hop 282 282 40.7[36.7, 44.9] 11.0[7.4, 14.5] 968 dev 74 74 44.5[36.6, 53.1] 10.8[4.1, 17.6] 964 dev:multi-hop 74 74 44.5[36.6, 53.1] 10.8[4.1, 17.6] 964 test 208 208 39.3[34.6, 44.2] 11.1[7.2, 15.4] 970 test:multi-hop 208 208 39.3[34.6, 44.2] 11.1[7.2, 15.4] 970 ### locomo-all-cer-k20-enso-flash k=20 · reader enso-flash · 282 of 282 answered · commit 885a21a63c3254415e40555f7905c8c04aa0be65 · run locomo-all-cer-k20-enso-flash questions n in set supported F1 EM tokens/q all 282 1536 11.0 42.5[38.1, 46.5] 11.7[8.2, 15.2] 968 all:multi-hop 282 282 11.0 42.5[38.1, 46.5] 11.7[8.2, 15.2] 968 dev 74 383 10.8 49.6[41.3, 57.7] 14.9[6.8, 24.3] 964 dev:multi-hop 74 74 10.8 49.6[41.3, 57.7] 14.9[6.8, 24.3] 964 test 208 1153 11.1 39.9[35.5, 45.0] 10.6[6.7, 14.9] 970 test:multi-hop 208 208 11.1 39.9[35.5, 45.0] 10.6[6.7, 14.9] 970 ### locomo-all-cer-k20-gpt-oss-120b k=20 · reader gpt-oss-120b · 282 of 282 answered · commit 885a21a63c3254415e40555f7905c8c04aa0be65 · run locomo-all-cer-k20-gpt-oss-120b questions n in set supported F1 EM tokens/q all 282 1536 11.0 42.6[38.8, 46.9] 11.3[7.8, 15.2] 968 all:multi-hop 282 282 11.0 42.6[38.8, 46.9] 11.3[7.8, 15.2] 968 dev 74 383 10.8 48.9[40.5, 57.0] 13.5[6.8, 21.6] 964 dev:multi-hop 74 74 10.8 48.9[40.5, 57.0] 13.5[6.8, 21.6] 964 test 208 1153 11.1 40.4[35.9, 45.2] 10.6[6.7, 14.9] 970 test:multi-hop 208 208 11.1 40.4[35.9, 45.2] 10.6[6.7, 14.9] 970 ### locomo-all-single-k20-qwen3.6-35b-a3b — partial k=20 · reader qwen3.6:35b-a3b · 1525 of 1536 answered · commit bea5d0ad184462b76db19ad2d601f39c329b7c09 · run locomo-all-single-k20-qwen3.6-35b-a3b questions n in set F1 EM tokens/q all 1525 1525 48.8[46.6, 51.0] 26.2[23.9, 28.5] 841 all:multi-hop 282 282 36.7[32.9, 41.0] 9.2[6.0, 12.8] 835 all:temporal 321 321 47.5[43.3, 51.7] 17.4[13.7, 21.8] 856 all:open-domain 92 92 17.7[11.5, 25.4] 12.0[5.4, 19.6] 781 all:single-hop 830 830 56.9[54.1, 59.5] 37.0[33.9, 40.1] 844 dev 383 383 48.8[44.9, 52.7] 24.8[20.6, 28.7] 864 dev:multi-hop 74 74 40.9[33.4, 49.6] 8.1[2.7, 14.9] 831 dev:temporal 90 90 52.7[45.2, 60.3] 21.1[13.3, 30.0] 893 dev:open-domain 19 19 4.5[0.0, 12.0] 0.0[0.0, 0.0] 782 dev:single-hop 200 200 54.2[48.2, 59.7] 35.0[28.5, 41.5] 872 test 1142 1142 48.8[46.5, 51.4] 26.7[24.2, 29.5] 833 test:multi-hop 208 208 35.2[31.0, 40.3] 9.6[5.8, 13.5] 836 test:temporal 231 231 45.4[40.5, 50.5] 16.0[11.7, 20.8] 842 test:open-domain 73 73 21.2[13.4, 29.6] 15.1[6.8, 23.3] 781 test:single-hop 630 630 57.7[54.5, 61.2] 37.6[33.8, 41.4] 835 ### locomo-all-context-k20-qwen3.6-35b-a3b — partial k=20 · reader qwen3.6:35b-a3b · 1525 of 1536 answered · commit 12468828cfba9497ebbb927af840534bb7e3fce2 · run locomo-all-context-k20-qwen3.6-35b-a3b questions n in set F1 EM tokens/q all 1525 1525 53.3[51.2, 55.4] 29.0[26.9, 31.3] 969 all:multi-hop 282 282 43.0[38.7, 47.1] 13.1[9.6, 17.4] 971 all:temporal 321 321 49.5[45.1, 53.6] 18.1[14.0, 22.7] 994 all:open-domain 92 92 19.8[13.0, 27.4] 13.0[6.5, 20.7] 917 all:single-hop 830 830 61.9[59.1, 64.4] 40.4[37.1, 43.5] 964 dev 383 383 51.9[48.0, 55.8] 26.4[22.2, 30.8] 1015 dev:multi-hop 74 74 45.1[36.6, 53.6] 13.5[6.8, 21.6] 984 dev:temporal 90 90 52.0[44.2, 59.0] 18.9[11.1, 26.7] 1058 dev:open-domain 19 19 7.4[1.3, 16.0] 0.0[0.0, 0.0] 987 dev:single-hop 200 200 58.6[52.9, 64.0] 37.0[30.5, 44.0] 1010 test 1142 1142 53.7[51.5, 56.1] 29.9[27.3, 32.6] 954 test:multi-hop 208 208 42.3[37.9, 47.0] 13.0[8.7, 17.8] 967 test:temporal 231 231 48.5[43.3, 53.3] 17.7[13.0, 22.9] 969 test:open-domain 73 73 23.0[15.1, 31.7] 16.4[8.2, 26.0] 899 test:single-hop 630 630 63.0[59.9, 66.1] 41.4[37.6, 45.2] 950 ### locomo-all-oracle-k20-qwen3.6-35b-a3b — partial k=20 · reader qwen3.6:35b-a3b · 1525 of 1536 answered · commit 12468828cfba9497ebbb927af840534bb7e3fce2 · run locomo-all-oracle-k20-qwen3.6-35b-a3b questions n in set F1 EM tokens/q all 1525 1525 61.8[59.7, 63.8] 34.6[32.5, 36.9] 82 all:multi-hop 282 282 55.6[51.2, 59.9] 20.2[15.6, 24.8] 169 all:temporal 321 321 54.4[50.3, 58.7] 22.7[18.4, 27.7] 64 all:open-domain 92 92 26.4[18.7, 34.9] 19.6[12.0, 28.3] 117 all:single-hop 830 830 70.8[68.2, 73.1] 45.8[42.2, 49.0] 56 dev 383 383 61.2[57.4, 65.1] 32.6[27.9, 37.3] 76 dev:multi-hop 74 74 60.6[53.0, 69.3] 18.9[10.8, 28.4] 135 dev:temporal 90 90 57.0[49.1, 64.5] 28.9[20.0, 37.8] 63 dev:open-domain 19 19 9.1[2.1, 17.4] 0.0[0.0, 0.0] 114 dev:single-hop 200 200 68.3[63.2, 73.3] 42.5[36.0, 49.0] 56 test 1142 1142 62.1[59.9, 64.4] 35.3[32.5, 38.2] 84 test:multi-hop 208 208 53.9[49.2, 58.9] 20.7[15.4, 26.4] 180 test:temporal 231 231 53.4[48.3, 58.2] 20.3[15.2, 25.5] 64 test:open-domain 73 73 31.0[22.2, 41.6] 24.7[16.4, 35.6] 118 test:single-hop 630 630 71.5[68.8, 74.5] 46.8[43.0, 51.0] 56 ### locomo-all-cer-k20-qwen3.6-35b-a3b — partial k=20 · reader qwen3.6:35b-a3b · 1525 of 1536 answered · commit 12468828cfba9497ebbb927af840534bb7e3fce2 · run locomo-all-cer-k20-qwen3.6-35b-a3b questions n in set F1 EM tokens/q all 1525 1525 50.3[48.2, 52.4] 26.7[24.5, 28.9] 970 all:multi-hop 282 282 40.2[35.9, 44.4] 11.3[7.8, 15.2] 968 all:temporal 321 321 48.6[44.3, 52.8] 16.5[12.8, 20.9] 995 all:open-domain 92 92 14.9[9.1, 22.0] 9.8[4.3, 16.3] 906 all:single-hop 830 830 58.4[55.7, 61.2] 37.7[34.6, 40.8] 967 dev 383 383 50.4[46.6, 54.4] 25.1[20.9, 29.8] 996 dev:multi-hop 74 74 43.3[34.8, 51.2] 10.8[4.1, 18.9] 964 dev:temporal 90 90 52.7[44.8, 60.5] 17.8[11.1, 26.7] 1039 dev:open-domain 19 19 7.3[1.1, 16.1] 0.0[0.0, 0.0] 940 dev:single-hop 200 200 56.1[50.6, 61.8] 36.0[30.0, 42.5] 994 test 1142 1142 50.3[47.9, 52.6] 27.2[24.7, 29.9] 961 test:multi-hop 208 208 39.1[34.3, 43.7] 11.5[7.2, 15.9] 970 test:temporal 231 231 47.0[42.0, 52.1] 16.0[11.7, 21.2] 978 test:open-domain 73 73 16.9[9.8, 24.8] 12.3[5.5, 20.5] 897 test:single-hop 630 630 59.1[55.7, 62.3] 38.3[34.6, 42.1] 959 ## Where it loses The multi-hop answer gain does not clear its interval. Retrieval’s multi-hop ALL@20 roughly doubles — 18.8 to 37.5 — and the answer gain that follows it, 36.3 to 39.0, sits inside the bootstrap interval of the baseline. Two marginal intervals that overlap do not establish that there is no difference, and they do not establish that there is one. Finding the right turns is not the same problem as reasoning over them, and this benchmark says so. Open-domain answers are below the baseline. On the 92 open-domain questions the earlier four-term re-rank scores 14.9 token F1 against cosine’s 17.7, on intervals wide enough that the ordering is all the table establishes. It is the one row where a shipped configuration is worse than the thing it replaced, and it is in the table above rather than in a footnote. The full-question-set runs did not finish. The qwen3.6:35b-a3b runs are the only ones that answer all four question categories rather than multi-hop alone, and each stopped 11 questions short of its declared set. Their tables are above, dimmed, with both counts on them. They are not in the comparison, they carry the only open-domain figures on this page — which is why that loss is quoted from them and labelled — and the honest reading is that the whole-set answer picture on LoCoMo is not settled here yet. ## The other LoCoMo harness [hanzoai/semantic · bench/locomo](https://hanzo.ai/benchmarks/locomo-subject-scope) measures the same dataset and its numbers do not belong beside these. It uses lexical retrieval with no embedding model, a fixed extractive reader with no language model anywhere, and all 1,986 questions including the adversarial quarter, reported apart. Its absolute F1 is far below anything on this page for that reason, and comparing the two would be comparing readers, not memories. It has its own page and its own paper. ## The other benchmarks [MemoryAgentBench](https://hanzo.ai/benchmarks/memoryagentbench) · [LongMemEval](https://hanzo.ai/benchmarks/longmemeval) · [RepoBench-R](https://hanzo.ai/benchmarks/repobench-r) · [LoCoMo · subject scope](https://hanzo.ai/benchmarks/locomo-subject-scope) · [LoCoMo-Conv](https://hanzo.ai/benchmarks/locomo-conv) · [Fleet residency](https://hanzo.ai/benchmarks/fleet) · [Live agent footprint](https://hanzo.ai/benchmarks/goroutine) · [Sandbox cold start](https://hanzo.ai/benchmarks/sandbox) · [Inference vs llama.cpp](https://hanzo.ai/benchmarks/inference) · [GPQA-Diamond](https://hanzo.ai/benchmarks/gpqa) · [all of them, and the head-to-head](https://hanzo.ai/benchmarks) raw runs, traces, prompts and scoring: [hanzoai/cloud · bench/brain](https://github.com/hanzoai/cloud/tree/main/bench/brain) · protocol in `METHOD.md` · every table above regenerated by `node results.mjs`