Skip to content
Language: English (United States)

CronQuestions

Benchmarks / CronQuestions

measured

Wikidata facts with start and end years, and 26,168 questions about who held what, and when. The graph API and Semantica’s temporal query engine, on the same facts and the same machine, with no model reading anything.

Results
storequery p50
  • Semantica 0.7.05,357 ms
  • Hanzo graph, through the API0.063 ms
  • Hanzo graph, replayed write clock0.058 ms
bash cronquestions/run.sh && python3 cron.py site test

regenerates every table on this page from the raw runs

licence
see the dataset repository
introduced by
Question Answering Over Temporal Knowledge Graphs — Saxena, Chakrabarti, Talukdar, ACL 2021
harness
hanzoai/benchmarks · cronquestions

What is measured

CronQuestions is 327,983 Wikidata facts — who held a position, played for a team, received an award — each with a start and an end year, and questions about them: who held a position in a given year, when someone held it, who held it before or after someone else, who held it first or last. The questions come annotated with their entities and years, so each is answered by a fixed plan of store calls and no model reads anything. What differs between rows is only the store the plan calls.

Results

test split, 4 question types, Hits@1 with bootstrap 95% intervals · hanzo-inc/cloud 1fffdf408f2e, apps/graph clean · semantica-agi/semantica 3910c33bc88c

What it says

The graph API answers 99.9% of the questions, and Semantica 99.9% — each the control’s answer, question for question. 99.9% is the plan’s own ceiling: the before/after questions it misses, it misses with the facts in a dictionary too.

What this table is not

It is not question answering from text. The questions arrive annotated, so this measures the store — whether it gives the right facts for a point in time, and at what cost — not a model’s reading of a question. 110 facts that start after today were refused at admission, which rejects a start more than five minutes ahead of the server clock; no covered question asks about them. The 3,832 time-join questions are not covered. The Hanzo timings were taken on a loaded machine, and each run records the load it had.

bash cronquestions/run.sh && python3 cron.py site test

Build on Hanzo Cloud