Benchmarks / LongMemEval
LongMemEval
Five hundred questions, each with its own haystack of chat sessions around 115k tokens. This is the cosine floor of that lane, measured across 4,846 sessions — the number a structured index would have to beat, and has not yet been asked to.
- dataset
- LongMemEval
- licence
- MIT
- introduced by
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory — Wu et al., 2024
- harness
- hanzoai/cloud · bench/brain
EMBED=all-minilm node longmemeval/retrieve.mjsregenerates every table on this page from the raw runs
What is measured
Five hundred questions, each with its own haystack of chat sessions about 115k tokens long. One or more sessions hold the evidence, and retrieval is scored as recall of those sessions against the benchmark’s own answer_session_ids. Every unique session is embedded turn by turn and ranked by its best turn — 4,846 sessions and 199,641 turns in all. ALL@5 is the share of questions whose every evidence session is inside the top five; ANY@5 is the share with at least one.
There is no Hanzo configuration in this table. It is the cosine floor of the lane and nothing else. It is on the site because it is the number a structured index would have to beat, and because the shape of the floor is itself informative about how much there is to win.
The floor
embedder all-minilm · turn-max ranking · retrieval p50 0.20 ms, p95 0.29 ms once the turns are embedded · run longmemeval/baseline-all-minilm.json
| question type | n | ALL@5 | ANY@5 | ALL@10 | ANY@10 | MRR |
|---|---|---|---|---|---|---|
| all questions | 500 | 85.8 | 97.4 | 93.4 | 98.8 | 0.903 |
| single-session-user | 70 | 95.7 | 95.7 | 97.1 | 97.1 | 0.852 |
| multi-session | 133 | 75.9 | 98.5 | 89.5 | 100.0 | 0.912 |
| single-session-preference | 30 | 96.7 | 96.7 | 96.7 | 96.7 | 0.803 |
| temporal-reasoning | 133 | 75.9 | 94.7 | 88.7 | 97.7 | 0.865 |
| knowledge-update | 78 | 96.2 | 100.0 | 98.7 | 100.0 | 0.964 |
| single-session-assistant | 56 | 100.0 | 100.0 | 100.0 | 100.0 | 1.000 |
How much headroom there is
Plain cosine already recovers every evidence session for 85.8% of questions in the top five, and at least one for 97.4%. So on this benchmark the headroom structure could claim is small, and it sits almost entirely in two question types: multi-session, where ALL@5 is 75.9, and temporal-reasoning, at 75.9. The single-session-assistant type is saturated at 100.0 and cannot move at all. A structured index that reported a large average gain here would be reporting something other than retrieval.
What this table is not
It is a retrieval floor, not a result about Hanzo, and it carries two caveats that a headline number would hide. It was run on the release the dataset card now marks deprecated in favour of a cleaned one. And session recall is not answer accuracy: systems reporting LongMemEval scores in the high eighties are usually reporting an end-to-end answer metric with a language-model judge, which is a different measurement from this column and not comparable to it.
The other benchmarks
LoCoMo · MemoryAgentBench · RepoBench-R · LoCoMo · subject scope · LoCoMo-Conv · Fleet residency · Live agent footprint · Sandbox cold start · Inference vs llama.cpp · GPQA-Diamond · all of them, and the head-to-head
raw run and scoring: hanzoai/cloud · bench/brain · regenerated by EMBED=all-minilm node longmemeval/retrieve.mjs