Benchmarks / CronQuestions
CronQuestions
Wikidata facts with start and end years, and 26,168 questions about who held what, and when. The graph API and Semantica’s temporal query engine, on the same facts and the same machine, with no model reading anything.
- dataset
- CronQuestions
- licence
- see the dataset repository
- introduced by
- Question Answering Over Temporal Knowledge Graphs — Saxena, Chakrabarti, Talukdar, ACL 2021
- harness
- hanzoai/benchmarks · cronquestions
bash cronquestions/run.sh && python3 cron.py site testregenerates every table on this page from the raw runs
What is measured
CronQuestions is 327,983 Wikidata facts — who held a position, played for a team, received an award — each with a start and an end year, and questions about them: who held a position in a given year, when someone held it, who held it before or after someone else, who held it first or last. The questions come annotated with their entities and years, so each is answered by a fixed plan of store calls and no model reads anything. What differs between rows is only the store the plan calls.
Both stores load the same facts and run in-process on one machine; nothing crosses a network. The Hanzo rows go through the graph’s own admission and its own resolution — what POST /v1/graph and POST /v1/graph/resolve do — asking every question at as_known now, as any caller does. Semantica is driven through its documented temporal query calls.
Results
test split, 4 question types, Hits@1 with bootstrap 95% intervals · hanzo-inc/cloud 1fffdf408f2e, apps/graph clean · semantica-agi/semantica 3910c33bc88c
| store | all | who held it in year T | when | before / after | first / last |
|---|---|---|---|---|---|
| control: the plan over dictionaries | 99.9[99.8, 99.9] | 100.0[100.0, 100.0] | 100.0[100.0, 100.0] | 98.6[98.0, 99.0] | 100.0[100.0, 100.0] |
| Semantica 0.7.0 | 99.9[99.8, 99.9] | 100.0[100.0, 100.0] | 100.0[100.0, 100.0] | 98.6[98.0, 99.0] | 100.0[100.0, 100.0] |
| Hanzo graph, through the API | 99.9[99.8, 99.9] | 100.0[100.0, 100.0] | 100.0[100.0, 100.0] | 98.6[98.0, 99.0] | 100.0[100.0, 100.0] |
| Hanzo graph, replayed write clock | 99.9[99.8, 99.9] | 100.0[100.0, 100.0] | 100.0[100.0, 100.0] | 98.6[98.0, 99.0] | 100.0[100.0, 100.0] |
cost of a question, and of loading the facts
| store | query p50 | query p99 | load | load average |
|---|---|---|---|---|
| Semantica 0.7.0 | 5,357 ms | 7,531 ms | 0.1 s | 11.0 |
| Hanzo graph, through the API | 0.063 ms | 2.4 ms | 116 s | 18.6 |
| Hanzo graph, replayed write clock | 0.058 ms | 2.1 ms | 145 s | 25.1 |
What it says
The graph API answers 99.9% of the questions, and Semantica 99.9% — each the control’s answer, question for question. 99.9% is the plan’s own ceiling: the before/after questions it misses, it misses with the facts in a dictionary too.
The difference is the price of a question. A point-in-time question about one entity costs the graph one indexed read of that entity’s relation, 0.063 ms at the median. Semantica’s temporal calls take no entity, so every question about one person reconstructs the whole graph at that year: 5,357 ms. That is about 85 thousand times the cost for the same answer.
What it found
The first run of this benchmark answered 70.0% through the API, and none of the questions asking who held a position in a given year. The graph’s as_of bounded when it had learned a fact, not when the fact was so: history filed today counted as learned today, so a question about 1995 found nothing. And ending one holder’s term ended the whole relation, erasing every other holder.
The graph now keeps the two instants apart — as_of for when the world was so, as_known for how much had been heard — a statement carries its own until, and a relation holds one value at a time or many, as the organization declares it. The replayed-clock row, which sets each fact’s write time to its own year, now answers the same as the API row: when a fact became known no longer decides what the world was.
What this table is not
It is not question answering from text. The questions arrive annotated, so this measures the store — whether it gives the right facts for a point in time, and at what cost — not a model’s reading of a question. 110 facts that start after today were refused at admission, which rejects a start more than five minutes ahead of the server clock; no covered question asks about them. The 3,832 time-join questions are not covered. Semantica loads in a fraction of a second because it builds no index; the graph’s load is admission, content addressing, its indexes and full-text search. The Hanzo timings were taken on a loaded machine, and each run records the load it had.
Deploy Frontier Intelligence with 10× Lower Costs
Run models, agents, and sandboxes on the world's fastest AI cloud. Instant GPU linking, 1,800+ tok/s speculative throughput, and unified management across all your infrastructure.
raw runs and scoring: hanzoai/benchmarks · cronquestions · regenerated by bash cronquestions/run.sh && python3 cron.py site test