On Runtime each Inspect sample gets its own microVM, built from the Dockerfile or compose file the eval already has.
Inspect AI, the eval framework from the UK AI Security Institute and Meridian
Labs, runs tool calls in a sandbox: local, or docker on your own machine. A
wide agentic eval on Docker is limited by that one machine. With Runtime
(withruntime.com) each sample is a machine with its own kernel, an account with
credit runs 100 sandboxes at the same time, and a sandbox that
waits on the model costs $0.03125 an hour at 2 vCPU, 4 GiB. inspect-ai
0.3.285 was the current PyPI release on 25 October 2026.
Switch an eval to Runtime
Terminalpip install "withruntime[inspect-ai]"inspect eval task.py --sandbox runtimeInstalling the package registers runtime with Inspect through an entry
point, next to docker and local. Nothing needs importing. A task sets the
sandbox like this:
Pythonfrom inspect_ai import Task, taskfrom inspect_ai.dataset import Samplefrom inspect_ai.scorer import includesfrom inspect_ai.solver import generate, use_toolsfrom inspect_ai.tool import bash, python@taskdef ctf_warmup(): return Task( dataset=[Sample(input="Find the flag in /workspace/flag.txt and print it.", target="FLAG{", files={"/workspace/flag.txt": "FLAG{runtime}"})], solver=[use_tools(bash(), python()), generate()], scorer=includes(), sandbox=("runtime", "Dockerfile"), )sandbox="runtime" alone uses Runtime's base image, Ubuntu with Python,
Node.js and git. Naming a Dockerfile or a single-service compose.yaml, as
for the Docker sandbox, builds it once as a Runtime image. Every
sample starts from that image. Runtime rebuilds it only when the Dockerfile or
its build context changes.
How many samples run at once?
Inspect starts as many sandboxes as --max-sandboxes allows. Without credit,
an account runs four sandboxes at the same time. By default, the
sandbox asks Inspect for that many. With credit, raise the limit:
Terminalinspect eval task.py --sandbox runtime --max-sandboxes 64 --max-connections 64--max-connections limits calls to the model. Raise both limits together.
Otherwise, the model becomes the bottleneck.
Which settings carry over from Docker?
| Docker sandbox setting | On Runtime |
|---|---|
Dockerfile, compose build or image |
Built once as a Runtime image |
compose cpus, mem_limit |
The sandbox's vCPU and memory |
working_dir, environment, user |
Carried over |
network_mode: none |
No internet |
| Commands as root (Inspect's default) | Run through the sandbox user's passwordless sudo |
inspect sandbox cleanup docker |
inspect sandbox cleanup runtime |
Other options, such as a ready Runtime image or labels, go in
RuntimeSandboxEnvironmentConfig or under x-runtime on the compose service
(every option).
A compose file with more than one service is not supported yet. An eval with a sidecar, such as a separate victim host, still needs Docker.
Why a microVM for agentic evals?
Capability evals ask models to break out, escalate and exfiltrate on purpose.
A container shares the host's kernel, so a kernel exploit inside one reaches
the machine (container escape). A Firecracker
microVM has its own kernel behind a hypervisor, and a sample with
network_mode: none has no route out.
Related
Sources
- Inspect AI: Sandboxing, read 25 October 2026
- inspect-ai on PyPI, version 0.3.285,
inspect eval --helpread 25 October 2026
On 1 October 2026, with a key on production, inspect-ai 0.3.275 passed 47 of 47 live checks on this sandbox, including a Dockerfile eval.
Facts on this page were checked on 25 October 2026.