RepoBench-R
The retrieval half of RepoBench: a completion point needs one snippet from another file, and the imports, definitions and identifiers say which. Regular expressions over the source, no model reading anything.
node results.mjs
regenerates every table on this page from the raw runs
- dataset
- RepoBench-R
- licence
- CC BY-NC-ND 4.0
- introduced by
- RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems — Liu, Xu, McAuley, ICLR 2024
- harness
- hanzoai/benchmarks · brain
What is measured
A completion point needs one snippet from another file, and the file’s imports, definitions and identifiers say which. RepoBench-R gives about twelve candidates per item with one of them correct, in two settings: cross-file-first, the first cross-file line of a file, and cross-file-random, any of them. Python. Retrieval is scored on its own, apart from generation, which is the reason this benchmark is here — it isolates the part a memory is responsible for.
What is measured
Typed links are regular expressions over import, def, class and identifiers. No model reads anything and nothing reads the answer. On cross-file-first they alone reach 31.4 R@1 against dense retrieval’s 18.8, which is the whole finding: on code, the structure is in the source text and an embedding is the roundabout way to it.
Results
dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item
Results
cross-file-first · dev split, 200 items
cross-file-first · test split, 500 items
dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item
| configuration | R@1 | R@3 | R@5 | MRR | nDCG@5 | easy R@1 | hard R@1 |
|---|---|---|---|---|---|---|---|
| dense only | 18.8 | 51.4 | 72.8 | 40.8 | 46.1 | 24.8 | 12.8 |
| typed links only (no model) | 31.4 | 61.0 | 75.4 | 50.5 | 54.5 | 37.2 | 25.6 |
| dense + typed links | 30.2 | 63.0 | 78.8 | 50.5 | 55.6 | 38.4 | 22.0 |
cross-file-random · dev split, 200 items
dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item
| configuration | R@1 | R@3 | R@5 | MRR | nDCG@5 | easy R@1 | hard R@1 |
|---|---|---|---|---|---|---|---|
| dense only | 22.5 | 49.0 | 69.5 | 41.8 | 46.1 | 27.0 | 18.0 |
| typed links only (no model) | 34.0 | 58.0 | 76.5 | 51.2 | 55.3 | 38.0 | 30.0 |
| dense + typed links | 26.5 | 59.0 | 79.5 | 47.1 | 53.3 | 33.0 | 20.0 |
cross-file-random · test split, 500 items
dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item
| configuration | R@1 | R@3 | R@5 | MRR | nDCG@5 | easy R@1 | hard R@1 |
|---|---|---|---|---|---|---|---|
| dense only | 24.6 | 55.6 | 74.6 | 45.0 | 50.2 | 29.6 | 19.6 |
| typed links only (no model) | 29.6 | 63.2 | 79.0 | 49.9 | 55.3 | 35.2 | 24.0 |
| dense + typed links | 31.2 | 64.6 | 77.8 | 51.0 | 55.8 | 35.6 | 26.8 |
| configuration | R@1 | R@3 | R@5 | MRR | nDCG@5 | easy R@1 | hard R@1 |
|---|---|---|---|---|---|---|---|
| dense only | 24.0 | 56.5 | 75.0 | 44.9 | 50.5 | 28.0 | 20.0 |
| typed links only (no model) | 34.5 | 63.0 | 77.5 | 52.2 | 56.9 | 41.0 | 28.0 |
| dense + typed links | 35.5 | 66.0 | 80.0 | 53.4 | 58.6 | 44.0 | 27.0 |
node results.mjs
PreviousCronQuestions · All Benchmarks Board · NextLoCoMo · subject scope
- Other benchmarks
LoCoMo · MemoryAgentBench · LongMemEval · CronQuestions · LoCoMo · subject scope · LoCoMo-Conv · Fleet residency · Live agent footprint · Sandbox cold start · Engine prefill on CUDA vs llama.cpp · GPQA-Diamond
raw runs and scoring: hanzoai/benchmarks · brain · regenerated by node results.mjs
Build on Hanzo Cloud
Hanzo AI Cloud