Benchmarks / RepoBench-R
measuredRepoBench-R
The retrieval half of RepoBench: a completion point needs one snippet from another file, and the imports, definitions and identifiers say which. Regular expressions over the source, no model reading anything.
- dataset
- RepoBench-R
- licence
- CC BY-NC-ND 4.0
- introduced by
- RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems — Liu, Xu, McAuley, ICLR 2024
- harness
- hanzoai/benchmarks · brain
node results.mjsregenerates every table on this page from the raw runs
What is measured
A completion point needs one snippet from another file, and the file’s imports, definitions and identifiers say which. RepoBench-R gives about twelve candidates per item with one of them correct, in two settings: cross-file-first, the first cross-file line of a file, and cross-file-random, any of them. Python. Retrieval is scored on its own, apart from generation, which is the reason this benchmark is here — it isolates the part a memory is responsible for.
Typed links are regular expressions over import, def, class and identifiers. No model reads anything and nothing reads the answer. On cross-file-first they alone reach 31.4 R@1 against dense retrieval’s 18.8, which is the whole finding: on code, the structure is in the source text and an embedding is the roundabout way to it.
Results
cross-file-first · dev split, 200 items
dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item
| configuration | R@1 | R@3 | R@5 | MRR | nDCG@5 | easy R@1 | hard R@1 |
|---|---|---|---|---|---|---|---|
| dense only | 24.0 | 56.5 | 75.0 | 44.9 | 50.5 | 28.0 | 20.0 |
| typed links only (no model) | 34.5 | 63.0 | 77.5 | 52.2 | 56.9 | 41.0 | 28.0 |
| dense + typed links | 35.5 | 66.0 | 80.0 | 53.4 | 58.6 | 44.0 | 27.0 |
cross-file-first · test split, 500 items
dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item
| configuration | R@1 | R@3 | R@5 | MRR | nDCG@5 | easy R@1 | hard R@1 |
|---|---|---|---|---|---|---|---|
| dense only | 18.8 | 51.4 | 72.8 | 40.8 | 46.1 | 24.8 | 12.8 |
| typed links only (no model) | 31.4 | 61.0 | 75.4 | 50.5 | 54.5 | 37.2 | 25.6 |
| dense + typed links | 30.2 | 63.0 | 78.8 | 50.5 | 55.6 | 38.4 | 22.0 |
cross-file-random · dev split, 200 items
dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item
| configuration | R@1 | R@3 | R@5 | MRR | nDCG@5 | easy R@1 | hard R@1 |
|---|---|---|---|---|---|---|---|
| dense only | 22.5 | 49.0 | 69.5 | 41.8 | 46.1 | 27.0 | 18.0 |
| typed links only (no model) | 34.0 | 58.0 | 76.5 | 51.2 | 55.3 | 38.0 | 30.0 |
| dense + typed links | 26.5 | 59.0 | 79.5 | 47.1 | 53.3 | 33.0 | 20.0 |
cross-file-random · test split, 500 items
dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item
| configuration | R@1 | R@3 | R@5 | MRR | nDCG@5 | easy R@1 | hard R@1 |
|---|---|---|---|---|---|---|---|
| dense only | 24.6 | 55.6 | 74.6 | 45.0 | 50.2 | 29.6 | 19.6 |
| typed links only (no model) | 29.6 | 63.2 | 79.0 | 49.9 | 55.3 | 35.2 | 24.0 |
| dense + typed links | 31.2 | 64.6 | 77.8 | 51.0 | 55.8 | 35.6 | 26.8 |
raw runs and scoring: hanzoai/benchmarks · brain · regenerated by node results.mjs