Hanzo AI
Try Hanzo

Benchmarks / Kai / Banking77

Kai · Decision harness

Banking77

Kai and Jev on the same 400 questions of Banking77, one run per Kai revision.

Latest

90.8%Accuracy · kai-1 · 0834a74fJev 83.5% · Win · Oct 3, 2026 · Hanzo-measured
0.074Calibration error (ECE) · kai-1 · 0834a74fJev 0.073 · Loss · Oct 3, 2026 · Hanzo-measured

History

10 results across 2 metrics, newest first. Superseded runs stay, so a revision's progress can be read.

82%88%94%09-2709-2810-03Kai a5 · df1c16a2: 90.8%, Sep 27, 2026Kai a8 · 0cfeef05: 89.8%, Sep 28, 2026Kai a7 · 0834a74f: 90.8%, Sep 28, 2026Kai a6 · a211cc70: 92.3%, Sep 28, 2026kai-1 · 0834a74f: 90.8%, Oct 3, 2026
MeasuredSubjectMetricValueBaselineResultStatus
Oct 3, 2026kai-1 · 0834a74fAccuracy90.8%Jev 83.5%WinLive · Hanzo-measured
Oct 3, 2026kai-1 · 0834a74fCalibration error (ECE)0.074Jev 0.073LossLive · Hanzo-measured
Sep 28, 2026Kai a8 · 0cfeef05Accuracy89.8%Jev 83.5%WinSuperseded · Hanzo-measured
Sep 28, 2026Kai a7 · 0834a74fAccuracy90.8%Jev 83.5%WinSuperseded · Hanzo-measured
Sep 28, 2026Kai a6 · a211cc70Accuracy92.3%Jev 83.5%WinSuperseded · Hanzo-measured
Sep 28, 2026Kai a8 · 0cfeef05Calibration error (ECE)0.085Jev 0.073LossSuperseded · Hanzo-measured
Sep 28, 2026Kai a7 · 0834a74fCalibration error (ECE)0.076Jev 0.073LossSuperseded · Hanzo-measured
Sep 28, 2026Kai a6 · a211cc70Calibration error (ECE)0.061Jev 0.073WinSuperseded · Hanzo-measured
Sep 27, 2026Kai a5 · df1c16a2Accuracy90.8%Jev 83.5%WinSuperseded · Hanzo-measured
Sep 27, 2026Kai a5 · df1c16a2Calibration error (ECE)0.065Jev 0.073WinSuperseded · Hanzo-measured

Conditions

split: harness400 questions, 400 answereddataset dde140a63fc55 Kai revisions measuredJev: typesafe/jev-1.13-20260917, measured Sep 25, 2026

Hardware

CUDA BF16

Source

research run jev.banking77_full-harness-kai-1.0834a74f-kai

Reproduce

bench score --harness . --kai results/kai/preds.json.gz --out results/scores.json # in hanzoai/benchmarks/decision @ 2c74e2d

More Kai benchmarks

AG News · DAIR Emotion · Support triage · Email spam · Phishing · Jailbreak · Toxicity · RAG relevance · Model routing · Typed decisions · MASSIVE · AG News, validation split · Support triage, validation split · Typed decisions, validation split · HarmBench (benign prompts) · Choosing among many options · Joint decisions · Release gate against Jev · Latency of one decision

Every result for Banking77 on the shelf · How these are measured

Build what’s next.

Try HanzoRead the docs