# LongMemEval benchmark — Hanzo AI

> The cosine retrieval floor on LongMemEval-S, measured across 4,846 sessions and 199,641 turns: the top five sessions already hold every evidence session for 85.8% of questions. No Hanzo row yet, and the page says so.

[Benchmarks](https://hanzo.ai/benchmarks) / LongMemEval

baseline only

# LongMemEval

Five hundred questions, each with its own haystack of chat sessions around 115k tokens. This is the cosine floor of that lane, measured across 4,846 sessions — the number a structured index would have to beat, and has not yet been asked to.

dataset

[LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval)

licence

MIT

introduced by

[LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://arxiv.org/abs/2410.10813) — Wu et al., 2024

harness

[hanzoai/cloud · bench/brain](https://github.com/hanzoai/cloud/tree/main/bench/brain/longmemeval)

`EMBED=all-minilm node longmemeval/retrieve.mjs`regenerates every table on this page from the raw runs

## What is measured

Five hundred questions, each with its own haystack of chat sessions about 115k tokens long. One or more sessions hold the evidence, and retrieval is scored as recall of those sessions against the benchmark’s own `answer_session_ids`. Every unique session is embedded turn by turn and ranked by its best turn — 4,846 sessions and 199,641 turns in all. ALL@5 is the share of questions whose every evidence session is inside the top five; ANY@5 is the share with at least one.

There is no Hanzo configuration in this table. It is the cosine floor of the lane and nothing else. It is on the site because it is the number a structured index would have to beat, and because the shape of the floor is itself informative about how much there is to win.

## The floor

embedder all-minilm · turn-max ranking · retrieval p50 0.20 ms, p95 0.29 ms once the turns are embedded · run `longmemeval/baseline-all-minilm.json`

question type

n

ALL@5

ANY@5

ALL@10

ANY@10

MRR

all questions

500

85.8

97.4

93.4

98.8

0.903

single-session-user

70

95.7

95.7

97.1

97.1

0.852

multi-session

133

75.9

98.5

89.5

100.0

0.912

single-session-preference

30

96.7

96.7

96.7

96.7

0.803

temporal-reasoning

133

75.9

94.7

88.7

97.7

0.865

knowledge-update

78

96.2

100.0

98.7

100.0

0.964

single-session-assistant

56

100.0

100.0

100.0

100.0

1.000

## How much headroom there is

Plain cosine already recovers every evidence session for 85.8% of questions in the top five, and at least one for 97.4%. So on this benchmark the headroom structure could claim is small, and it sits almost entirely in two question types: multi-session, where ALL@5 is 75.9, and temporal-reasoning, at 75.9. The single-session-assistant type is saturated at 100.0 and cannot move at all. A structured index that reported a large average gain here would be reporting something other than retrieval.

## What this table is not

It is a retrieval floor, not a result about Hanzo, and it carries two caveats that a headline number would hide. It was run on the release the dataset card now marks deprecated in favour of a cleaned one. And session recall is not answer accuracy: systems reporting LongMemEval scores in the high eighties are usually reporting an end-to-end answer metric with a language-model judge, which is a different measurement from this column and not comparable to it.

## The other benchmarks

[LoCoMo](https://hanzo.ai/benchmarks/locomo) · [MemoryAgentBench](https://hanzo.ai/benchmarks/memoryagentbench) · [RepoBench-R](https://hanzo.ai/benchmarks/repobench-r) · [LoCoMo · subject scope](https://hanzo.ai/benchmarks/locomo-subject-scope) · [LoCoMo-Conv](https://hanzo.ai/benchmarks/locomo-conv) · [Fleet residency](https://hanzo.ai/benchmarks/fleet) · [Live agent footprint](https://hanzo.ai/benchmarks/goroutine) · [Sandbox cold start](https://hanzo.ai/benchmarks/sandbox) · [Inference vs llama.cpp](https://hanzo.ai/benchmarks/inference) · [GPQA-Diamond](https://hanzo.ai/benchmarks/gpqa) · [all of them, and the head-to-head](https://hanzo.ai/benchmarks)

raw run and scoring: [hanzoai/cloud · bench/brain](https://github.com/hanzoai/cloud/tree/main/bench/brain/longmemeval) · regenerated by `EMBED=all-minilm node longmemeval/retrieve.mjs`