Benchmarks / RepoBench-R
RepoBench-R
The retrieval half of RepoBench: a completion point needs one snippet from another file, and the imports, definitions and identifiers say which. Regular expressions over the source, no model reading anything.
- dataset
- RepoBench-R
- licence
- CC BY-NC-ND 4.0
- introduced by
- RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems — Liu, Xu, McAuley, ICLR 2024
- harness
- hanzoai/cloud · bench/brain
node results.mjsregenerates every table on this page from the raw runs
What is measured
A completion point needs one snippet from another file, and the file’s imports, definitions and identifiers say which. RepoBench-R gives about twelve candidates per item with one of them correct, in two settings: cross-file-first, the first cross-file line of a file, and cross-file-random, any of them. Python. Retrieval is scored on its own, apart from generation, which is the reason this benchmark is here — it isolates the part a memory is responsible for.
Typed links are regular expressions over import, def, class and identifiers. No model reads anything and nothing reads the answer. On cross-file-first they alone reach 31.4 R@1 against dense retrieval’s 18.8, which is the whole finding: on code, the structure is in the source text and an embedding is the roundabout way to it.
Results
cross-file-first · dev split, 200 items
dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item
| configuration | R@1 | R@3 | R@5 | MRR | nDCG@5 | easy R@1 | hard R@1 | candidates | p50 ms |
|---|---|---|---|---|---|---|---|---|---|
| dense only | 24.0 | 56.5 | 75.0 | 44.9 | 50.5 | 28.0 | 20.0 | 13.8 | 0.13 |
| BM25 only | 25.5 | 50.0 | 71.5 | 43.5 | 48.1 | 28.0 | 23.0 | 13.8 | 0.80 |
| typed links only (no model) | 34.5 | 63.0 | 77.5 | 52.2 | 56.9 | 41.0 | 28.0 | 13.8 | 1.71 |
| dense + BM25 | 22.5 | 55.0 | 71.5 | 43.8 | 48.3 | 25.0 | 20.0 | 13.8 | 1.35 |
| dense + typed links | 35.5 | 66.0 | 80.0 | 53.4 | 58.6 | 44.0 | 27.0 | 13.8 | 0.45 |
| BM25 + typed links (no model) | 34.5 | 63.0 | 77.5 | 52.2 | 56.9 | 41.0 | 28.0 | 13.8 | 1.08 |
| full: dense + BM25 + typed links | 37.0 | 67.0 | 79.0 | 55.0 | 59.4 | 43.0 | 31.0 | 13.8 | 0.85 |
cross-file-first · test split, 500 items
dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item
| configuration | R@1 | R@3 | R@5 | MRR | nDCG@5 | easy R@1 | hard R@1 | candidates | p50 ms |
|---|---|---|---|---|---|---|---|---|---|
| dense only | 18.8 | 51.4 | 72.8 | 40.8 | 46.1 | 24.8 | 12.8 | 11.5 | 0.14 |
| BM25 only | 19.4 | 50.8 | 67.6 | 40.4 | 44.2 | 23.6 | 15.2 | 11.5 | 0.78 |
| typed links only (no model) | 31.4 | 61.0 | 75.4 | 50.5 | 54.5 | 37.2 | 25.6 | 11.5 | 1.09 |
| dense + BM25 | 21.2 | 51.6 | 73.8 | 42.3 | 47.6 | 25.2 | 17.2 | 11.5 | 0.92 |
| dense + typed links | 30.2 | 63.0 | 78.8 | 50.5 | 55.6 | 38.4 | 22.0 | 11.5 | 0.30 |
| BM25 + typed links (no model) | 31.4 | 61.0 | 75.4 | 50.5 | 54.5 | 37.2 | 25.6 | 11.5 | 0.75 |
| full: dense + BM25 + typed links | 33.2 | 63.4 | 79.4 | 52.3 | 57.2 | 40.0 | 26.4 | 11.5 | 0.69 |
cross-file-random · dev split, 200 items
dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item
| configuration | R@1 | R@3 | R@5 | MRR | nDCG@5 | easy R@1 | hard R@1 | candidates | p50 ms |
|---|---|---|---|---|---|---|---|---|---|
| dense only | 22.5 | 49.0 | 69.5 | 41.8 | 46.1 | 27.0 | 18.0 | 12.0 | 0.12 |
| BM25 only | 26.5 | 51.0 | 68.5 | 44.6 | 47.8 | 31.0 | 22.0 | 12.0 | 0.58 |
| typed links only (no model) | 34.0 | 58.0 | 76.5 | 51.2 | 55.3 | 38.0 | 30.0 | 12.0 | 0.81 |
| dense + BM25 | 24.0 | 55.0 | 69.5 | 44.1 | 47.8 | 31.0 | 17.0 | 12.0 | 0.77 |
| dense + typed links | 26.5 | 59.0 | 79.5 | 47.1 | 53.3 | 33.0 | 20.0 | 12.0 | 0.34 |
| BM25 + typed links (no model) | 34.0 | 57.5 | 75.5 | 51.1 | 54.9 | 38.0 | 30.0 | 12.0 | 1.02 |
| full: dense + BM25 + typed links | 32.5 | 58.0 | 74.5 | 50.4 | 54.0 | 38.0 | 27.0 | 12.0 | 1.07 |
cross-file-random · test split, 500 items
dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item
| configuration | R@1 | R@3 | R@5 | MRR | nDCG@5 | easy R@1 | hard R@1 | candidates | p50 ms |
|---|---|---|---|---|---|---|---|---|---|
| dense only | 24.6 | 55.6 | 74.6 | 45.0 | 50.2 | 29.6 | 19.6 | 12.0 | 0.17 |
| BM25 only | 26.2 | 54.8 | 74.0 | 45.6 | 50.2 | 30.0 | 22.4 | 12.0 | 0.95 |
| typed links only (no model) | 29.6 | 63.2 | 79.0 | 49.9 | 55.3 | 35.2 | 24.0 | 12.0 | 0.92 |
| dense + BM25 | 26.8 | 56.4 | 77.2 | 46.9 | 52.4 | 30.8 | 22.8 | 12.0 | 0.70 |
| dense + typed links | 31.2 | 64.6 | 77.8 | 51.0 | 55.8 | 35.6 | 26.8 | 12.0 | 0.26 |
| BM25 + typed links (no model) | 29.6 | 63.4 | 79.6 | 49.9 | 55.5 | 34.8 | 24.4 | 12.0 | 0.80 |
| full: dense + BM25 + typed links | 30.2 | 62.4 | 79.8 | 50.2 | 55.7 | 36.0 | 24.4 | 12.0 | 0.84 |
Read these as a comparison, not a result
No configuration was frozen on dev before these test runs. Every other benchmark on this site picks its configuration on a declared development split and freezes it at a commit before the test split runs once; these tables did not. So they compare generators against each other and they do not deliver a held-out result, and both splits are printed for exactly that reason. On cross-file-random the two rows that tie for best on dev — typed links alone and BM25 plus typed links, both 34.0 R@1 — are not the row that leads on test, which is dense plus typed links at 31.2. A frozen comparison would have had to commit to one of them before seeing the test items, and would have picked the wrong one.
The full combination is not the best row. On cross-file-random, dense plus BM25 plus typed links scores 30.2 R@1 while dense plus typed links alone scores 31.2. The third generator costs a point in that setting. It is in the table above rather than dropped, because a benchmark page that only shows the configuration that wins is a page that has chosen its rows after seeing them.
The other benchmarks
LoCoMo · MemoryAgentBench · LongMemEval · LoCoMo · subject scope · LoCoMo-Conv · Fleet residency · Live agent footprint · Sandbox cold start · Inference vs llama.cpp · GPQA-Diamond · all of them, and the head-to-head
raw runs and scoring: hanzoai/cloud · bench/brain · regenerated by node results.mjs