Benchmarks / Kai / RAG relevance
Kai · Decision harnessRAG relevance
Kai and Jev on the same 400 questions of RAG relevance, one run per Kai revision.
Latest
History
10 results across 2 metrics, newest first. Superseded runs stay, so a revision's progress can be read.
| Measured | Subject | Metric | Value | Baseline | Result | Status |
|---|---|---|---|---|---|---|
| Oct 3, 2026 | kai-1 · 0834a74f | Accuracy | 68.0% | Jev 62.0% | Win | Live · Hanzo-measured |
| Oct 3, 2026 | kai-1 · 0834a74f | Calibration error (ECE) | 0.028 | Jev 0.279 | Win | Live · Hanzo-measured |
| Sep 28, 2026 | Kai a8 · 0cfeef05 | Accuracy | 69.0% | Jev 62.0% | Win | Superseded · Hanzo-measured |
| Sep 28, 2026 | Kai a7 · 0834a74f | Accuracy | 67.5% | Jev 62.0% | Win | Superseded · Hanzo-measured |
| Sep 28, 2026 | Kai a6 · a211cc70 | Accuracy | 65.3% | Jev 62.0% | Win | Superseded · Hanzo-measured |
| Sep 28, 2026 | Kai a8 · 0cfeef05 | Calibration error (ECE) | 0.032 | Jev 0.279 | Win | Superseded · Hanzo-measured |
| Sep 28, 2026 | Kai a7 · 0834a74f | Calibration error (ECE) | 0.029 | Jev 0.279 | Win | Superseded · Hanzo-measured |
| Sep 28, 2026 | Kai a6 · a211cc70 | Calibration error (ECE) | 0.052 | Jev 0.279 | Win | Superseded · Hanzo-measured |
| Sep 27, 2026 | Kai a5 · df1c16a2 | Accuracy | 66.8% | Jev 62.0% | Win | Superseded · Hanzo-measured |
| Sep 27, 2026 | Kai a5 · df1c16a2 | Calibration error (ECE) | 0.037 | Jev 0.279 | Win | Superseded · Hanzo-measured |
Conditions
split: harness400 questions, 400 answereddataset dde140a63fc55 Kai revisions measuredJev: typesafe/jev-1.13-20260917, measured Sep 25, 2026Hardware
CUDA BF16
Source
research run app.rag_relevance-harness-kai-1.0834a74f-kai
Reproduce
bench score --harness . --kai results/kai/preds.json.gz --out results/scores.json # in hanzoai/benchmarks/decision @ 2c74e2d
More Kai benchmarks
AG News · DAIR Emotion · Banking77 · Support triage · Email spam · Phishing · Jailbreak · Toxicity · Model routing · Typed decisions · MASSIVE · AG News, validation split · Support triage, validation split · Typed decisions, validation split · HarmBench (benign prompts) · Choosing among many options · Joint decisions · Release gate against Jev · Latency of one decision
Every result for RAG relevance on the shelf · How these are measured