# RepoBench-R benchmark — Hanzo AI > Cross-file retrieval for code completion, scored apart from generation: typed links over imports, definitions and identifiers beat dense retrieval on both RepoBench-R settings — and the full combination costs a point on one of them. [Benchmarks](https://hanzo.ai/benchmarks) / RepoBench-R measured # RepoBench-R The retrieval half of RepoBench: a completion point needs one snippet from another file, and the imports, definitions and identifiers say which. Regular expressions over the source, no model reading anything. dataset [RepoBench-R](https://huggingface.co/datasets/tianyang/repobench-r) licence CC BY-NC-ND 4.0 introduced by [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://arxiv.org/abs/2306.03091) — Liu, Xu, McAuley, ICLR 2024 harness [hanzoai/cloud · bench/brain](https://github.com/hanzoai/cloud/tree/main/bench/brain) `node results.mjs`regenerates every table on this page from the raw runs ## What is measured A completion point needs one snippet from another file, and the file’s imports, definitions and identifiers say which. RepoBench-R gives about twelve candidates per item with one of them correct, in two settings: `cross-file-first`, the first cross-file line of a file, and `cross-file-random`, any of them. Python. Retrieval is scored on its own, apart from generation, which is the reason this benchmark is here — it isolates the part a memory is responsible for. Typed links are regular expressions over `import`, `def`, `class` and identifiers. No model reads anything and nothing reads the answer. On cross-file-first they alone reach 31.4 R@1 against dense retrieval’s 18.8, which is the whole finding: on code, the structure is in the source text and an embedding is the roundabout way to it. ## Results ### cross-file-first · dev split, 200 items dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item configuration R@1 R@3 R@5 MRR nDCG@5 easy R@1 hard R@1 candidates p50 ms dense only 24.0 56.5 75.0 44.9 50.5 28.0 20.0 13.8 0.13 BM25 only 25.5 50.0 71.5 43.5 48.1 28.0 23.0 13.8 0.80 typed links only (no model) 34.5 63.0 77.5 52.2 56.9 41.0 28.0 13.8 1.71 dense + BM25 22.5 55.0 71.5 43.8 48.3 25.0 20.0 13.8 1.35 dense + typed links 35.5 66.0 80.0 53.4 58.6 44.0 27.0 13.8 0.45 BM25 + typed links (no model) 34.5 63.0 77.5 52.2 56.9 41.0 28.0 13.8 1.08 full: dense + BM25 + typed links 37.0 67.0 79.0 55.0 59.4 43.0 31.0 13.8 0.85 ### cross-file-first · test split, 500 items dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item configuration R@1 R@3 R@5 MRR nDCG@5 easy R@1 hard R@1 candidates p50 ms dense only 18.8 51.4 72.8 40.8 46.1 24.8 12.8 11.5 0.14 BM25 only 19.4 50.8 67.6 40.4 44.2 23.6 15.2 11.5 0.78 typed links only (no model) 31.4 61.0 75.4 50.5 54.5 37.2 25.6 11.5 1.09 dense + BM25 21.2 51.6 73.8 42.3 47.6 25.2 17.2 11.5 0.92 dense + typed links 30.2 63.0 78.8 50.5 55.6 38.4 22.0 11.5 0.30 BM25 + typed links (no model) 31.4 61.0 75.4 50.5 54.5 37.2 25.6 11.5 0.75 full: dense + BM25 + typed links 33.2 63.4 79.4 52.3 57.2 40.0 26.4 11.5 0.69 ### cross-file-random · dev split, 200 items dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item configuration R@1 R@3 R@5 MRR nDCG@5 easy R@1 hard R@1 candidates p50 ms dense only 22.5 49.0 69.5 41.8 46.1 27.0 18.0 12.0 0.12 BM25 only 26.5 51.0 68.5 44.6 47.8 31.0 22.0 12.0 0.58 typed links only (no model) 34.0 58.0 76.5 51.2 55.3 38.0 30.0 12.0 0.81 dense + BM25 24.0 55.0 69.5 44.1 47.8 31.0 17.0 12.0 0.77 dense + typed links 26.5 59.0 79.5 47.1 53.3 33.0 20.0 12.0 0.34 BM25 + typed links (no model) 34.0 57.5 75.5 51.1 54.9 38.0 30.0 12.0 1.02 full: dense + BM25 + typed links 32.5 58.0 74.5 50.4 54.0 38.0 27.0 12.0 1.07 ### cross-file-random · test split, 500 items dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item configuration R@1 R@3 R@5 MRR nDCG@5 easy R@1 hard R@1 candidates p50 ms dense only 24.6 55.6 74.6 45.0 50.2 29.6 19.6 12.0 0.17 BM25 only 26.2 54.8 74.0 45.6 50.2 30.0 22.4 12.0 0.95 typed links only (no model) 29.6 63.2 79.0 49.9 55.3 35.2 24.0 12.0 0.92 dense + BM25 26.8 56.4 77.2 46.9 52.4 30.8 22.8 12.0 0.70 dense + typed links 31.2 64.6 77.8 51.0 55.8 35.6 26.8 12.0 0.26 BM25 + typed links (no model) 29.6 63.4 79.6 49.9 55.5 34.8 24.4 12.0 0.80 full: dense + BM25 + typed links 30.2 62.4 79.8 50.2 55.7 36.0 24.4 12.0 0.84 ## Read these as a comparison, not a result No configuration was frozen on dev before these test runs. Every other benchmark on this site picks its configuration on a declared development split and freezes it at a commit before the test split runs once; these tables did not. So they compare generators against each other and they do not deliver a held-out result, and both splits are printed for exactly that reason. On cross-file-random the two rows that tie for best on dev — typed links alone and BM25 plus typed links, both 34.0 R@1 — are not the row that leads on test, which is dense plus typed links at 31.2. A frozen comparison would have had to commit to one of them before seeing the test items, and would have picked the wrong one. The full combination is not the best row. On cross-file-random, dense plus BM25 plus typed links scores 30.2 R@1 while dense plus typed links alone scores 31.2. The third generator costs a point in that setting. It is in the table above rather than dropped, because a benchmark page that only shows the configuration that wins is a page that has chosen its rows after seeing them. ## The other benchmarks [LoCoMo](https://hanzo.ai/benchmarks/locomo) · [MemoryAgentBench](https://hanzo.ai/benchmarks/memoryagentbench) · [LongMemEval](https://hanzo.ai/benchmarks/longmemeval) · [LoCoMo · subject scope](https://hanzo.ai/benchmarks/locomo-subject-scope) · [LoCoMo-Conv](https://hanzo.ai/benchmarks/locomo-conv) · [Fleet residency](https://hanzo.ai/benchmarks/fleet) · [Live agent footprint](https://hanzo.ai/benchmarks/goroutine) · [Sandbox cold start](https://hanzo.ai/benchmarks/sandbox) · [Inference vs llama.cpp](https://hanzo.ai/benchmarks/inference) · [GPQA-Diamond](https://hanzo.ai/benchmarks/gpqa) · [all of them, and the head-to-head](https://hanzo.ai/benchmarks) raw runs and scoring: [hanzoai/cloud · bench/brain](https://github.com/hanzoai/cloud/tree/main/bench/brain) · regenerated by `node results.mjs`