Benchmarks / Sandbox cold start
Cold start, by primitive
A V8 context in 0.23 ms, an isolate measured at 0.59 ms rather than cited, a cold container in 150.8 ms and a pooled one in 37.5 ms — and the reason the fastest row is not the answer: an isolate runs JavaScript and cannot run pytest, pip, cargo or a shell.
node sandbox/sandbox.mjsprints the table on this page — transcribed from bench/README.md, the pass of 2026-09-07
What is measured
The time from asking for a sandbox to having one that will execute code. Three primitives answer that request in three different ways, and they are not interchangeable: a V8 context is a new heap inside a process that is already running, a cold container is an image started from nothing, and a pooled container is one that was started earlier and kept. Timings are the median of the pass.
Results
M-series laptop · the pass of 2026-09-07 · median of the pass
| primitive | measured here | published elsewhere | what it can run |
|---|---|---|---|
| V8 context | 0.23 ms | — | JavaScript, sharing the heap |
| V8 isolate, isolated-vm 7.0.1 | 2.79 ms, 1.2 MB | JavaScript, its own heap | |
| container, cold | 150.8 ms | E2B under 200 ms · Modal ~1 s · Cloudflare 1–3 s | anything with a shell |
| container, pooled | 37.5 ms | — | anything with a shell |
The isolate row used to be the context row, and that was wrong in our favour. A vm.createContext makes a fresh global inside an isolate that is already running; it is cheap because it shares that heap, which is the one thing an isolate does not do. Reporting the context against a published isolate figure and calling it seventeenfold compared two different primitives. Installing the library the 2.79 ms is attributed to and running it on this laptop puts an isolate at 0.59 ms — still under the figure cited at us, by 4.7× rather than 17×, and a measurement rather than an argument. The megabyte is V8's too: a fresh isolate's heap measures 1.00 MiB here, close to the 1.2 MB cited, because that is what any isolate costs including one of ours.
The row that matters for an agent that needs a shell is the cold container, and it comes in under the figures E2B, Modal and Cloudflare publish, with a pooled one at 37.5 ms. Those three were taken by other people on other hardware under protocols we did not run — they are the vendors' own claims, quoted, not a matched measurement.
Why the fastest row is not the answer
An isolate runs JavaScript. It cannot run pytest, pip, cargo or a shell, which is most of what a coding agent is actually asked to do. Putting 0.23 ms and 150.8 ms in one column invites the reading that the first is the same service six hundred times faster, and it is not the same service. The comparison is per workload: for a JavaScript transform the isolate is the right primitive and the container is waste, and for anything that installs a package the isolate cannot take the job at any latency.
That is also why the table above carries a column for what each primitive can run. A latency table without it is the kind of chart that wins an argument and loses the deployment.
What this does not measure
Cold start is not throughput. This is the time to the first instruction, on an idle machine, with the image already local. It says nothing about what happens when a thousand sandboxes are asked for at once, and nothing about pulling an image that is not cached — both of which dominate in production and neither of which is on this page.
One machine, one day. A single pass on one laptop, reproducible by one command. Read it as a floor that has been established rather than as a service level, and note that the published figures in the right-hand column were taken by other people on other hardware under protocols we did not run — they are the vendors’ own claims, quoted, not a matched measurement.
It is one of three costs, not the cost. A sandbox is what an agent needs to run a command. What it costs to keep an agent that is doing nothing is 477 bytes and a 0.034 ms resume, and what a running agent holds is 601 bytes of heap and 187 ns to spawn. The three are separate measurements with separate commands and they do not add up into a single number.
The other benchmarks
LoCoMo · MemoryAgentBench · LongMemEval · RepoBench-R · LoCoMo · subject scope · LoCoMo-Conv · Fleet residency · Live agent footprint · Inference vs llama.cpp · GPQA-Diamond · all of them, and the head-to-head
harness and the committed table: hanzoai/cloud · bench/sandbox · sandbox.mjs needs a container runtime; colima start is enough on macOS