Hanzo Context

Benchmarks

Similarity finds what sounds related. Memory needs what is connected. These tables measure that difference on public benchmarks, with the split, the embedder, the reader and the metric named on every table, and every number regenerated from its raw run.

Protocol

Configurations are chosen on a declared development split and frozen at a commit before the test split runs once. LoCoMo: dev conversations 0–2, test 3–9; adversarial questions excluded, as Mem0 and Zep exclude them. MemoryAgentBench: dev 6k, test 32k / 64k / 262k. Readers: enso-flash (house),gemma4:31b (open, LoCoMo-Conv's own reader, run locally), and the strongest router model that answers. Temperature 0, one prompt file per benchmark shared by every row. Intervals are bootstrap 95% over questions. Retrieval recall is reported as ALL and ANY because a multi-hop question is only answerable when every turn it needs is present — and as a diagnostic, not the headline: a system can ground a correct answer on equivalent evidence the annotator did not mark, which the supported column counts.

Raw runs, traces and scoring: hanzoai/cloud · bench/brain. Papers: papers.hanzo.ai. Product: POST api.hanzo.ai/v1/memory/search.

LoCoMo · retrieval

Ten conversations, 5,882 turns, 1,540 questions with annotated evidence turns. Each row adds one candidate generator to the engine — BM25, the atomic fact index, canonical entities, the timeline, adjacency, one hop of typed expansion, a second retrieval hop asked with what the first resolved — and the rows below the working ones are the shapes that lost and shaped the design. The fact layer here is LoCoMo's own observation annotations, each citing its source turns.

LoCoMo · all ten conversations · zen-embedding-0.6b · k=20

facts LoCoMo's own observations · weights frozen on dev at 33d0f8749bd1 · ALL = every annotated turn in the top 20; ANY = at least one; R@20 = |retrieved ∩ gold| / |gold|

configuration	multi-hop ALL	multi-hop ANY	single-hop ALL	temporal ALL	open-domain ALL	R@20	nDCG@20	supported	candidates	tokens/q	p50 ms
semantic only	22.7	79.8	78.2	77.3	33.7	71.9	49.6	81.0	40	660	0.55
+lexical	22.7	78.0	82.5	78.8	37.0	74.6	53.2	83.3	52	665	0.61
+facts	36.9	88.3	82.3	82.6	42.4	78.2	59.6	86.4	71	787	0.83
+adjacency	39.0	88.7	84.2	83.5	44.6	79.7	61.4	87.8	74	782	0.83
+iterative hops	39.0	87.9	86.8	84.4	45.7	80.9	62.6	88.0	76	788	1.65
failed: global RRF	33.7	89.0	86.8	83.8	40.2	80.5	56.2	89.0	137	762	1.73

LoCoMo · all ten conversations · all-MiniLM-L6-v2 · k=10

facts LoCoMo's own observations · weights frozen on dev at 33d0f8749bd1 · ALL = every annotated turn in the top 10; ANY = at least one; R@10 = |retrieved ∩ gold| / |gold|

configuration	multi-hop ALL	multi-hop ANY	single-hop ALL	temporal ALL	open-domain ALL	R@10	nDCG@10	supported	candidates	tokens/q	p50 ms
semantic only	9.6	49.6	51.4	46.1	16.3	45.3	31.1	53.1	40	283	0.25
+lexical	13.8	58.9	67.2	62.0	23.9	59.2	43.7	67.6	54	301	0.31
+facts	20.9	73.4	73.1	72.0	25.0	66.7	52.0	75.6	73	364	0.42
+adjacency	21.6	74.5	74.6	72.6	27.2	67.7	53.3	76.3	77	361	0.42
+iterative hops	24.1	78.0	77.5	73.5	27.2	70.1	55.8	78.5	78	377	0.76
failed: global RRF	20.2	74.8	73.4	74.1	28.3	67.2	48.6	76.1	143	382	0.43

LoCoMo · test split, conversations 3–9 · zen-embedding-0.6b · k=20

facts LoCoMo's own observations · weights frozen on dev at 33d0f8749bd1 · ALL = every annotated turn in the top 20; ANY = at least one; R@20 = |retrieved ∩ gold| / |gold|

configuration	multi-hop ALL	multi-hop ANY	single-hop ALL	temporal ALL	open-domain ALL	R@20	nDCG@20	supported	candidates	tokens/q	p50 ms
semantic only	18.8	80.3	78.3	72.3	35.6	70.9	48.8	80.5	40	651	0.55
+lexical	21.2	78.4	82.8	73.6	38.4	73.9	52.9	83.2	53	656	0.62
+facts	35.1	88.5	83.0	78.8	43.8	77.9	58.7	86.4	71	771	1.15
+entities	35.1	88.5	83.0	78.8	43.8	77.9	58.7	86.4	71	771	1.10
+timeline	34.6	88.5	83.3	79.7	45.2	78.2	59.9	86.8	71	769	1.04
+adjacency	37.5	88.5	84.6	80.1	45.2	79.1	60.5	87.5	75	767	1.03
+typed graph	37.5	88.5	84.6	80.1	45.2	79.1	60.5	87.5	75	767	0.98
+iterative hops	37.5	88.0	87.7	81.4	45.2	80.7	61.6	87.8	77	771	1.67
failed: PRF	19.7	75.5	72.4	69.3	35.6	66.4	46.3	75.8	634	665	1.10
failed: chain search	37.5	88.0	83.8	78.8	45.2	78.5	56.7	86.3	79	764	1.64
failed: surface entities	35.6	88.0	83.0	78.8	43.8	77.9	58.2	86.2	82	773	0.98
failed: global RRF	32.2	88.5	86.7	80.1	38.4	79.6	55.5	88.4	139	751	1.20

LoCoMo-Conv

The same memories asked the way users ask: dialog, implicit, counterfactual, composed. The data is not yet released (the paper's one address answers 404), so no LoCoMo-Conv number appears here. What does: the paper's retrieval protocol (all-MiniLM-L6-v2, k=10, verbatim containment) reproduced on the original LoCoMo questions the rewrites came from, by memory unit.

conv-proxy-naive-rag-all-minilm-k10

k=10 · run conv-proxy-naive-rag-all-minilm-k10

row	n	recall	ALL	ANY	tokens/q
original	1977	42.9	39.0	47.7	262
original·cat2	320	49.4	46.3	52.5	253
original·no-adversarial	1531	45.5	40.7	51.6	265
original·cat3	89	26.3	16.9	39.3	250
original·cat1	281	26.7	9.6	49.8	270
original·cat4	841	52.3	51.5	53.2	269
original·cat5	446	33.9	33.2	34.5	250

conv-proxy-naive-rag-st-minilm-k10

k=10 · run conv-proxy-naive-rag-st-minilm-k10

row	n	recall	ALL	ANY	tokens/q
original	1977	42.9	39.0	47.9	262
original·cat2	320	49.4	46.3	52.5	253
original·no-adversarial	1531	45.5	40.7	51.7	265
original·cat3	89	26.3	16.9	39.3	251
original·cat1	281	27.0	9.6	50.5	270
original·cat4	841	52.3	51.5	53.2	269
original·cat5	446	33.9	33.2	34.5	250

conv-proxy-naive-rag-zenlm_zen-embedding-0.6b-k10

k=10 · run conv-proxy-naive-rag-zenlm_zen-embedding-0.6b-k10

row	n	recall	ALL	ANY	tokens/q
original	1977	59.7	54.9	65.6	315
original·cat2	320	74.0	70.9	76.9	332
original·no-adversarial	1531	63.5	57.6	70.9	319
original·cat3	89	40.4	33.7	49.4	286
original·cat1	281	38.1	14.9	68.7	311
original·cat4	841	70.5	69.3	71.7	320
original·cat5	446	46.4	45.5	47.3	302

conv-proxy-units-st-minilm-k10

run conv-proxy-units-st-minilm-k10

row	n	recall	tokens/q
turn·all	1982	42.8	279
turn·no-adversarial	1536	45.4	283
w2·all	1982	56.3	571
w2·no-adversarial	1536	54.3	567
w3·all	1982	62.5	829
w3·no-adversarial	1536	60.7	823
w5·all	1982	60.3	1258
w5·no-adversarial	1536	58.2	1240
session·all	1982	62.1	6855
session·no-adversarial	1536	61.8	6848

LoCoMo · answers

The same questions answered by a reader from the retrieved context. Token F1 and exact match with LoCoMo's own normalisation. Rows differ only in what was retrieved: cosine, the shipped re-rank, the engine, or the annotated gold turns themselves (the ceiling that separates retrieval's share of the error from the reader's).

locomo-all-cer-k20-gemma4-31b

k=20 · reader gemma4:31b · commit ddc076f5dd0506ce50428416c336e45c4a7ada73 · run locomo-all-cer-k20-gemma4-31b

row	n	supported	F1	EM	tokens/q
all	282	11.0	40.7[36.5, 44.9]	11.0[7.4, 14.9]	968
dev	74	10.8	44.5[36.3, 53.2]	10.8[4.1, 18.9]	964
test	208	11.1	39.3[34.8, 43.9]	11.1[7.2, 15.4]	970

locomo-all-context-k20-gemma4-31b

k=20 · reader gemma4:31b · commit ddc076f5dd0506ce50428416c336e45c4a7ada73 · run locomo-all-context-k20-gemma4-31b

row	n	supported	F1	EM	tokens/q
all	282	13.5	40.5[36.8, 44.3]	9.9[6.4, 13.5]	971
dev	74	13.5	44.7[36.5, 52.6]	9.5[2.7, 16.2]	984
test	208	13.5	39.0[34.8, 43.4]	10.1[6.3, 14.4]	967

locomo-all-oracle-k20-gemma4-31b

k=20 · reader gemma4:31b · commit ddc076f5dd0506ce50428416c336e45c4a7ada73 · run locomo-all-oracle-k20-gemma4-31b

row	n	supported	F1	EM	tokens/q
all	282	12.4	53.0[49.2, 57.0]	16.0[12.1, 19.9]	169
dev	74	13.5	59.2[50.7, 66.9]	13.5[6.8, 21.6]	135
test	208	12.0	50.8[46.7, 55.1]	16.8[12.5, 22.1]	180

locomo-all-single-k20-gemma4-31b

k=20 · reader gemma4:31b · commit ddc076f5dd0506ce50428416c336e45c4a7ada73 · run locomo-all-single-k20-gemma4-31b

row	n	supported	F1	EM	tokens/q
all	282	11.7	37.6[33.8, 42.0]	9.9[6.7, 13.8]	835
dev	74	12.2	41.1[33.1, 48.9]	6.8[1.4, 12.2]	831
test	208	11.5	36.3[31.9, 40.9]	11.1[6.7, 15.4]	836

MemoryAgentBench · Conflict Resolution

FactConsolidation: a haystack of numbered facts with later updates, single-hop and multi-hop questions, 6k to 262k tokens. Metric: substring exact match, the benchmark's own. This is the lane CAR reports on and the one vendor claims cite. The 6k haystacks are the development split; 32k, 64k and 262k are run once.

noreader

split dev · k=10 · reader none · commit ddc076f5dd · run mab-dev-noreader-none

row	n	substring EM
mh_6k	98	77.5
sh_6k	98	98.0

noreader

split test · k=10 · reader none · commit ced158ddb3 · run mab-test-noreader-none-v0

row	n	substring EM
mh_32k	100	32.0
mh_64k	99	31.3
mh_262k	98	25.5
sh_32k	98	98.0
sh_64k	99	92.9
sh_262k	95	97.9

Reference points

Numbers other systems publish, quoted with their protocol because the protocols differ. CAR (arXiv 2606.01435): MemoryAgentBench FactConsolidation 78.0 single-hop / 30.2 multi-hop pooled over 6k–262k with gpt-4o-mini, 82 / 27 at 262k; 94.8 / 51.5 pooled with gpt-4o, 93 / 41 at 262k. LoCoMo-Conv (arXiv 2609.03467): best reported retrieval R@10 with query facets, AnchorMem, dialog 0.754 · implicit 0.524 · counterfactual 0.732 · composed 0.432. Vendor pages: Mem0 92.5 and Zep 94.7 on LoCoMo under their own answer-accuracy protocols with LLM judges; Pith 68.0 substring EM on MemoryAgentBench multi-hop 262k. None of these are re-run here; a row of ours is comparable to one of theirs only where the split, the embedder, the reader and the metric match, and the page says where they do.

generated 2026-09-09T12:35:28.042Z