Benchmarks / GPQA-Diamond
measuredThe router on GPQA-Diamond
One set of 198 graduate-level questions, answered by Enso's router and by single models from other labs, every row a research run.
- dataset
- GPQA
- licence
- CC BY 4.0
- introduced by
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark — Rein, Hou, Stickland, Petty, Pang, Dirani, Michael, Bowman, COLM 2024
- harness
- api.hanzo.ai/v1/research
curl -s "https://api.hanzo.ai/v1/research/runs?org=hanzo&benchmark=enso.gpqa_diamond"prints the table on this page — transcribed from the research API, read when the page opens
What is measured
GPQA-Diamond is the hardest subset of GPQA: 198 multiple-choice questions in biology, physics and chemistry, written by domain PhDs and filtered to the ones that other PhDs get right and skilled non-experts get wrong even with the web open. It is a knowledge benchmark rather than an agentic one — one question, one answer, no tools — which is what makes it useful here: it isolates the choice of model from everything else a system does.
What the router adds is that the choice is per question. Enso reads each one and dispatches it, so a tier is not a model — it is a pool and a willingness to spend. Every row below answers the same 198 questions, which is why a score and what its run cost belong in one table, and why a score quoted without its cost says nothing about the system it came from.
What this does not establish
A benchmark score is not a capability. GPQA-Diamond is four-way multiple choice on settled science. It measures what a pool knows and rewards a router that recognises a hard question; it says nothing about whether the answer is reasoned, whether the model holds up over a long autonomous run, or whether it can use a tool. The routing overhead is real work this table does not score either.
A row is a run, as of its date. Every row went through enso-bench's harness at the tag and on the date it shows; a system is the router or a single model as that run met it, not necessarily a preset the catalog serves today. What a row cost is the tokens its run records at today's list price for its own model id, the router's included, so two rows are priced by one rule.
every figure a research run at api.hanzo.ai · the dataset: GPQA under CC BY 4.0