Skip to content
Hanzo AI
Try Hanzo
Language: English (United States)

Engine prefill on CUDA vs llama.cpp

Benchmarks / Engine prefill on CUDA vs llama.cpp

measured

hanzo-engine against llama.cpp on CUDA, built against the same weights and run on the same box: prefill throughput at the prompt lengths where the whole 95% interval of the ratio lies above parity.

Start on Hanzo Cloud
Prefill on CUDA, cell by cell
hanzo t/sllama.cpp t/s
  • CUDA · Zen-1.7B · Q4_K_M · 5007961.37250.2
  • CUDA · Zen-1.7B · Q4_K_M · 5007474.86871.3
  • CUDA · Zen-4B · Q4_K_M · 20483774.43592.5
scripts/dossier.sh --backend cuda --model <same.gguf> -p 512,500,2048,4096 -n 128 -r 7

prints the table on this page — transcribed from the committed run files on branch bench/dossier, the pass of 2026-07-21

What is measured

hanzo-engine and llama.cpp, built against the same weights, run on the same box, alternating. Prefill is tokens per second at a given prompt length, and the ratio is ours over theirs, so above 1.000 is a win. The models are Zen-1.7B at Q4_K_M and Zen-4B at Q4_K_M; 7 repetitions per cell, as its run records, with the first discarded as warmup, a Student-t 95% interval on each engine, and the ratio's interval formed by adding the two relative intervals in quadrature.

Prefill on CUDA, cell by cell

3 cells from 2 runs on branch bench/dossier · engine 1.7.87 · each run pins its engine release, its llama.cpp build and its model digest, and is linked below

The work that produced it

Refutation-Driven Performance Engineering is the campaign that produced these kernels, written as a log of numbered hypotheses, each settled by a cheap decisive experiment. A long prefill is compute-bound, which is where that work landed.

scripts/dossier.sh --backend cuda --model <same.gguf> -p 512,500,2048,4096 -n 128 -r 7

PreviousSandbox cold start · All Benchmarks Board · NextGPQA-Diamond

  • Other benchmarks

LoCoMo · MemoryAgentBench · LongMemEval · CronQuestions · RepoBench-R · LoCoMo · subject scope · LoCoMo-Conv · Fleet residency · Live agent footprint · Sandbox cold start · GPQA-Diamond

the committed runs, in bench-runs: CUDA Zen-1.7B Q4_K_M · CUDA Zen-4B Q4_K_M

Build on Hanzo Cloud

Start on Hanzo CloudView Models & Pricing

Hanzo AI Cloud