Benchmarks / Kai / Choosing among many options
Kai · CapabilityChoosing among many options
Accuracy as the number of listed options grows, the same cases put to both models. Past 160 options Kai shortlists by retrieval first.
Latest
History
7 results, newest first. Superseded runs stay, so a revision's progress can be read.
| Measured | Subject | Metric | Value | Baseline | Result | Status |
|---|---|---|---|---|---|---|
| Oct 3, 2026 | kai-1 · 0834a74f · 4 options | Accuracy | 96.8% | Jev 90.5% | Win | Live · Hanzo-measured |
| Oct 3, 2026 | kai-1 · 0834a74f · 16 options | Accuracy | 93.5% | Jev 85.3% | Win | Live · Hanzo-measured |
| Oct 3, 2026 | kai-1 · 0834a74f · 77 options | Accuracy | 90.8% | Jev 84.3% | Win | Live · Hanzo-measured |
| Oct 3, 2026 | kai-1 · 0834a74f · 150 options | Accuracy | 90.8% | Jev 94.0% | Loss | Live · Hanzo-measured |
| Oct 3, 2026 | kai-1 · 0834a74f · 1,000 options | Accuracy | 0.5% | Jev refused | Unscored | Live · Hanzo-measured |
| Oct 3, 2026 | kai-1 · 0834a74f · 10,000 options | Accuracy | 0.0% | Jev refused | Unscored | Live · Hanzo-measured |
| Oct 3, 2026 | kai-1 · 0834a74f · 100,000 options | Accuracy | 0.0% | Jev refused | Unscored | Live · Hanzo-measured |
Conditions
choice sizes: 4, 16, 77, 150, 1,000, 10,000, 100,000every option scored up to 160, shortlisted past itJev: typesafe/jev-1.13, measured Sep 27, 2026Hardware
CUDA F32 on dgx
Source
research run cap.cardinality-cases-kai-1-kai
Reproduce
bench cap --harness . --who kai,jev --kai <checkpoint> # in hanzoai/benchmarks/decision
More Kai benchmarks
AG News · DAIR Emotion · Banking77 · Support triage · Email spam · Phishing · Jailbreak · Toxicity · RAG relevance · Model routing · Typed decisions · MASSIVE · AG News, validation split · Support triage, validation split · Typed decisions, validation split · HarmBench (benign prompts) · Joint decisions · Release gate against Jev · Latency of one decision
Every result for Choosing among many options on the shelf · How these are measured