Benchmarks / LoCoMo-Conv
LoCoMo-Conv
The same memories asked the way people ask them — dialog, implicit, counterfactual, composed. The dataset is not released, so what is here is the paper’s retrieval protocol run on the original questions the rewrites came from, and nothing more.
- dataset
- LoCoMo-Conv
- licence
- unreleased
- introduced by
- LoCoMo-Conv — Chang & Chen, NTU MiuLab, September 2026
- harness
- hanzoai/cloud · bench/brain
node results.mjsregenerates every table on this page from the raw runs
There is no score on this page
LoCoMo-Conv rewrites LoCoMo’s questions the way people actually ask them — dialog, implicit, counterfactual, composed — and it is the benchmark on this list we would most like to report. The data is not released. The paper’s one address answers 404. So no LoCoMo-Conv number appears here, no Hanzo configuration has been run on it, and nothing below is a result on that benchmark.
What is below is the paper’s retrieval protocol — all-MiniLM-L6-v2, k=10, verbatim containment — reproduced on the original LoCoMo questions the rewrites came from. Read as a diagnostic it says something useful about what a LoCoMo-Conv number would be measuring, which is why it is worth the page even though it scores nothing.
The embedder moves it more than most systems differ
the paper’s protocol · k=10 · verbatim containment · the original LoCoMo questions
| embedder | n | R@10, all | R@10, no adversarial | single-hop | multi-hop | temporal |
|---|---|---|---|---|---|---|
| all-MiniLM-L6-v2 (the paper’s) | 1977 | 42.9 | 45.5 | 52.3 | 26.7 | 49.4 |
| all-MiniLM-L6-v2, sentence-transformers | 1977 | 42.9 | 45.5 | 52.3 | 27.0 | 49.4 |
| zen-embedding-0.6b | 1977 | 59.7 | 63.5 | 70.5 | 38.1 | 74.0 |
Changing nothing but the embedding model moves R@10 from 42.9 to 59.7 — 16.8 points, on identical questions, an identical index unit and an identical k.
So does the unit of memory
one embedder · k=10 · the same questions · what changes is how the conversation is cut up
| memory unit | n | R@10, all | R@10, no adversarial |
|---|---|---|---|
| a single turn | 1982 | 42.8 | 45.4 |
| a 2-turn window | 1982 | 56.3 | 54.3 |
| a 3-turn window | 1982 | 62.5 | 60.7 |
| a 5-turn window | 1982 | 60.3 | 58.2 |
| a whole session | 1982 | 62.1 | 61.8 |
A single turn reaches 42.8; a three-turn window reaches 62.5 at three times the tokens — 19.7 points for a decision nobody reports.
What this is for
Both of those differences are larger than most published differences between named memory systems. A paper reporting R@10 on a conversational benchmark, ours included, is reporting the sum of three choices — the embedder, the unit and k — and usually names one of them. That is the reason every table on this site carries its embedder, its unit and its k in the line above it, and the reason a single R@10 quoted without them is not a comparison between systems.
For reference, and not comparable to anything here since the set is unreleased: LoCoMo-Conv reports a best retrieval R@10 with query facets and AnchorMem of dialog 0.754, implicit 0.524, counterfactual 0.732, composed 0.432.
The other benchmarks
LoCoMo · MemoryAgentBench · LongMemEval · RepoBench-R · LoCoMo · subject scope · Fleet residency · Live agent footprint · Sandbox cold start · Inference vs llama.cpp · GPQA-Diamond · all of them, and the head-to-head
raw runs and scoring: hanzoai/cloud · bench/brain · regenerated by node results.mjs