# Sandbox cold start benchmarks: what the published numbers leave out

Published sandbox start times differ in where the clock runs, where it stops, the shape and the percentile. Time all four yourself.

**Runtime times every operation from outside each minute, as a customer, and
publishes the samples and the code ([speed](/docs/speed)).** Every sandbox
provider publishes a start time, and they rarely measure the same thing. One
quotes a fresh microVM. Another resumes a paused one. One stops the clock when
the machine boots. Another stops it when your code runs. This post covers the
four choices that move a published figure, then gives you a script that times
any provider the same way.

## Why can't you compare published start times directly?

Each provider answers a different question, and the page rarely says which.
Most providers' pages do not say where the figure was timed from. They do not say whether
the network is in it, which machine size was started, or whether it is a median or a best
case. Each of those choices can move a figure more than the gap between two providers.
The four sections below cover them one at a time.

## Where does the clock run: on the server or on your machine?

On the server, a create is timed from the moment the request reaches the API
until the API answers. Your code times from before the request leaves until
the answer is back. That adds DNS, TLS and one round trip each way, on every
call. A figure under one round trip across the country can only have been
timed on the provider's side. Put it beside a server column, never beside
what your own code sees.

## Where does the clock stop: running, or your code's first result?

A sandbox is "running" when its machine is up. It is useful once your first
command has come back. Between the two, a second request starts the process
inside the machine, and the language runtime loads. A provider that stops at
"running" or "boot" is quoting the shorter number. For an agent, the second
number is what the user waits for.

## Which shape was started?

Many providers keep a few machines of their most common size started ahead of time
(a [warm pool](/glossary/warm-pool)). They hand one over when a create asks for
exactly that shape. Ask for anything else, and the machine is started on demand.
The pool's start and the on-demand start are both true start times for the same product. A benchmark that uses only the default shape says nothing about the
shape your agent actually asks for. A custom image is a third path.

## Which percentile is shown?

Medians hide the slow starts that users remember. A figure such as "under one
second" with no percentile could be a median or a best case. A
benchmark with fewer than about 50 samples cannot show the p99 at all
([what p95 latency means](/glossary/p95-latency)).

## What else moves the number?

- **First time or repeat.** The first [snapshot](/glossary/sandbox-snapshot)
  of a sandbox writes its whole memory. Later ones are quicker. Say which
  one you timed.
- **One at a time or many at once.** A sequential run is the easy case. A hundred creates
  in the same second queue behind each other, which a single create does not
  face. Time the concurrency you will run in production.
- **The date.** Start times move with every release. On 15 October a change
  in how snapshots are taken cut a repeat snapshot of a 4 GiB sandbox from
  1.96 s to 1.19 s at the median ([changelog](/changelog/2026-10-15)). A
  benchmark without a date and a version is a rumor.
- **Errors.** A run that drops its failed starts reports a median that is
  better than it should be. Count every attempt.

## How do you benchmark sandbox start times yourself?

Time the step your agent waits for. Time it from where your agent runs, for
the shape the agent asks for, and report the tail. This script does that for
Runtime. It runs 30 sequential starts of the default shape, then 30 of a shape
with no warm spare. Each start is timed through its first Python result, and
the script reports the median, p95 and p99. To compare like with like, swap the
two calls inside `time` for another provider's create and first command.

```ts
import { Sandbox } from "withruntime";

const at = (sorted: number[], q: number) =>
  Math.round(sorted[Math.min(sorted.length - 1, Math.ceil(q * sorted.length) - 1)]!);

async function time(label: string, shape: { vcpu: number; memoryMiB: number }, runs = 30) {
  const running: number[] = [];
  const firstResult: number[] = [];
  let failed = 0;
  for (let i = 0; i < runs; i++) {
    const started = performance.now();
    try {
      await using sbx = await Sandbox.create(shape);
      running.push(performance.now() - started);
      await sbx.exec(["python3", "-c", "pass"], { check: true });
      firstResult.push(performance.now() - started);
    } catch {
      failed++;
    }
  }
  for (const [step, xs] of [
    ["running", running],
    ["first result", firstResult],
  ] as const) {
    const s = [...xs].sort((a, b) => a - b);
    console.log(
      `${label}, ${step}: p50 ${at(s, 0.5)} ms, p95 ${at(s, 0.95)} ms, p99 ${at(s, 0.99)} ms (${s.length} ok, ${failed} failed)`,
    );
  }
}

await time("default shape", { vcpu: 2, memoryMiB: 4096 });
await time("no warm spare", { vcpu: 2, memoryMiB: 2048 });
```

```python
import math
import time

from withruntime import Sandbox


def at(xs: list[float], q: float) -> int:
    s = sorted(xs)
    return round(s[min(len(s) - 1, math.ceil(q * len(s)) - 1)])


def bench(label: str, vcpu: int, memory_mib: int, runs: int = 30) -> None:
    running: list[float] = []
    first_result: list[float] = []
    failed = 0
    for _ in range(runs):
        started = time.perf_counter()
        try:
            with Sandbox.create(vcpu=vcpu, memory_mib=memory_mib) as sbx:
                running.append((time.perf_counter() - started) * 1000)
                sbx.exec(["python3", "-c", "pass"], check=True)
                first_result.append((time.perf_counter() - started) * 1000)
        except Exception:
            failed += 1
    for step, xs in (("running", running), ("first result", first_result)):
        print(
            f"{label}, {step}: p50 {at(xs, 0.5)} ms, p95 {at(xs, 0.95)} ms, "
            f"p99 {at(xs, 0.99)} ms ({len(xs)} ok, {failed} failed)"
        )


bench("default shape", 2, 4096)
bench("no warm spare", 2, 2048)
```

Percentiles here are nearest-rank, as in Runtime's published measurements. Run the
script from the server your agent loop runs on, not from a laptop, unless the
loop runs on users' laptops. Thirty runs give a fair median and p95. For a
p99, you need a few hundred runs.

## Where are Runtime's full results?

The [speed guide](/docs/speed) covers every operation Runtime times from
outside, each minute, through the public API: the API's own time and the
client's, at p50, p95 and p99, with sample counts, the raw samples and the
code. It publishes an operation's figures once that operation is within twice
its physical limit.

**Runtime gives your agents Linux sandboxes that bill only measured use; try it
at [withruntime.com](https://withruntime.com) and time it yourself with the
script above.** To see what start times add up to in an agent loop, read
[cold starts in agent loops](/blog/cold-starts-in-agent-loops). For other
questions to ask a provider, see the
[provider checklist](/blog/agent-sandbox-provider-checklist).