Benchmarks / Inference vs llama.cpp

The engine against llama.cpp

Four GPU backends, two models, four prompt lengths, seven repetitions each with a 95% interval on every ratio. The standing is a surface rather than a number: prefill wins on CUDA at long prompts, reaches parity on Metal only at the longest, falls away on ROCm and collapses on Vulkan — and decode trails everywhere.

harness

hanzoai/engine · bench/dossier

our papers

Edge LLM Inference Across Commodity Accelerators, Refutation-Driven Performance Engineering

prints the table on this page — transcribed from the committed run files on branch bench/dossier, the pass of 2026-07-21

What is measured

hanzo-engine and llama.cpp, built against the same weights, run on the same box, alternating. Prefill is tokens per second at a given prompt length; decode is single-stream generation. The ratio is ours over theirs, so above 1.000 is a win. Four backends, two model sizes, four prompt lengths and a generation length — and every cell is 7 repetitions with the first discarded as warmup, a Student-t 95% interval on each engine, and the ratio's interval formed by adding the two relative intervals in quadrature.

The verdict column is the harness's and not a reading of ours: WIN only where the whole interval lies above 1.000, LOSS only where it lies below, PARITY wherever it straddles. A cell also carries a ! when either engine's coefficient of variation exceeded 5%, which is how the harness says the box was not quiet. A flagged cell is not a result, and the largest apparent win in the whole board is flagged.

hanzo-engine and llama.cpp, same weights, same box, ours over theirs — so above 1.000 is a win and below it is a loss. There is no single number for this and every version of this page that printed one was picking a cell. The ratio depends on how long the prompt is, and which way it depends on that differs by backend: on CUDA it rises to a win, on Metal it climbs from well behind to parity, on ROCm it falls further behind as the prompt grows, and on Vulkan it collapses.

ratio with its 95% interval · 7 repetitions, first discarded as warmup · Qwen3 at Q4_K_M · engine 1.7.87 against llama.cpp 81ff7ab · 6 committed runs on branch bench/dossier · pp500 is a deliberately ragged shape, not a rung of the ladder · ! is the harness's own flag for a cell where either engine's variance exceeded 5%

Where it wins. 3 cells in the board have an interval lying entirely above parity with no variance flag, and 2 of them are long prompts: CUDA Qwen3-1.7B at pp4096 is 1.088 [1.051, 1.125], CUDA Qwen3-4B at pp2048 is 1.051 [1.017, 1.085]. Those are the defensible wins — a long prefill is compute-bound and it is where the kernel work has landed.

The largest apparent win is the one not to quote. CUDA at pp512 reads 1.272 on the raw ratio, the largest number in the board — and the harness flags it, because llama.cpp's own samples varied by 19% on a box whose load average says it was not quiet. Its interval spans 1.029 to 1.515, which is another way of saying the run does not know. A flagged cell is not a result, and it is on this page only so that nobody finds it in the run file and wonders why it was left out.

Where it trails, plainly. Decode is behind on every backend measured — Metal 4B 0.938, Metal 1.7B 0.920, CUDA 4B 0.880, ROCm 1.7B 0.850, CUDA 1.7B 0.839, Vulkan 1.7B 0.604 — with Metal the closest at 0.938 and Vulkan the furthest at 0.604. Decode is bandwidth-bound and a memory-bound loop is where a portable kernel pays for its portability. On Metal, prefill climbs monotonically with the prompt — 0.826 at pp512, 0.893 at pp2048, 0.967 at pp4096 — and reaches parity only at the longest shape. The ragged pp500 column is a control rather than a rung of that ladder: it is not a power of two, so it catches a kernel that is quick only on the shapes it was tuned for, and Metal reads 0.839 there. ROCm goes the other way, 0.779 at pp512 down to 0.692 at pp4096. Vulkan is the open failure: 0.240 at pp4096, which is a quarter of llama.cpp's throughput and an attention-geometry problem rather than a tuning one.

every cell above is a row of board.json in a committed run directory — CUDA Qwen3-1.7B · CUDA Qwen3-4B · Metal Qwen3-1.7B · Metal Qwen3-4B · ROCm Qwen3-1.7B · Vulkan Qwen3-1.7B — each with its per-repetition samples, its quiet-gate log and the pinned model digest beside it. Rebuilt by scripts/dossier.sh --backend metal --model <same.gguf> -p 512,500,2048,4096 -n 128 -r 7 && python3 scripts/bench_stats.py <run-dir>

Every cell, and the throughputs behind it

26 cells · Qwen3 at Q4_K_M · engine 1.7.87 at a0c6d74 against llama.cpp 81ff7ab · model digest 72c5c3cb38fa · runs of 2026-07-21

The earlier pass, and why it is not the standing result

Table 1 of the cross-backend paper measured a different model — Qwen3-8B at Q8_0 — on three boxes, at a single prompt length, best-of-three with one warmup. It is a real measurement and it established the ordering the kernel work then acted on. It is not the standing result, for the reason the paper itself gives: these are best-of-three runs without a collected variance estimate, so a ratio near 1.00 cannot be told from noise.

Qwen3-8B-Q8_0 · single GPU · best-of-3 single-stream runs, one warmup · the pass of 2026-06-17

This table is where “decode at parity on all three” came from, and the board above does not reproduce it. Under seven repetitions with intervals, decode is a loss on every backend measured. Three things differ — a larger model, a heavier quantization, and a protocol with no variance estimate — and the honest statement is that the two do not contradict each other so much as the earlier one was never in a position to make the claim. Where a figure has an interval and a figure does not, the one with the interval stands.

A withdrawn pass. This page also carried a cooled, order-alternated Metal pass dated 2026-06-27, quoted to four decimal places, from which it drew a Metal prefill figure and a Metal decode figure that read as parity. Its run directory is in no repository and on no machine we can find, so nobody — including us — can check or reproduce it. It has been removed rather than left with a caveat: a number whose run has gone missing is not a weaker number, it is not a measurement.

Where it loses

Decode trails on every backend. Decode is bandwidth-bound, and a memory-bound loop is where a portable kernel pays for its portability: the same source compiles to four backends, and none of the four gets the hand-tuning a single-target kernel would. Prefill is compute-bound, which is where the work has landed and where the wins are.

Vulkan is an open failure, not a gap. It reaches a quarter of llama.cpp's prefill throughput at the longest prompt and gets worse as the prompt grows, which is the signature of attention geometry rather than of a slow kernel — the paper rules out submission chunking and the weight GEMM explicitly. ROCm has the same sign: it falls from 0.779 at pp512 to 0.692 at pp4096. A ratio that degrades with length is a different problem from one that is simply low.

Two models is not a sweep. The board is Qwen3 at 1.7B and 4B, one quantization. The model-size effect on decode is real and modest — Metal 0.92 to 0.94, CUDA 0.84 to 0.88 — and two points establish a direction rather than a trend. Nothing here says what happens at 70B, and the largest model this site quotes anywhere for this comparison is the 8B of the earlier table.

The work behind it

Refutation-Driven Performance Engineering is the campaign that produced these kernels, written as a refutation log rather than a result: numbered hypotheses, most of them plausible, each killed by a cheap decisive experiment. It is also the paper that retracted one of its own wins — an earlier ROCm decode figure was withdrawn when the llama.cpp build it beat turned out to be a weak one — which is the habit that makes the rest of it worth reading.

The other benchmarks

LoCoMo · MemoryAgentBench · LongMemEval · RepoBench-R · LoCoMo · subject scope · LoCoMo-Conv · Fleet residency · Live agent footprint · Sandbox cold start · GPQA-Diamond · all of them, and the head-to-head

the committed runs: dbc-metal-qwen3-1p7b-20260721-183614 · dbc-metal-qwen3-4b-20260721-181831 · spark-cuda-qwen3-1p7b-20260721-182202 · spark-cuda-qwen3-4b-20260721-183326 · evo-rocm-qwen3-1p7b-20260721-174808 · evo-vulkan-qwen3-1p7b-20260721-175906