# How to run visual regression tests with stable screenshots

Make and compare screenshot baselines in the same pinned sandbox image. The OS, fonts and browser then stay the same between runs.

**On Runtime, the machine that made a baseline can be rebuilt exactly, by
version number.** Each build of a custom image creates a new numbered version.
A sandbox created from `visual-tests@3` gets the files of that version and
nothing newer. This holds on every run until you delete that version. Every run
is a Firecracker microVM created from the image, on the same Ubuntu 24.04 base.
It is billed on the CPU it uses. A 2 vCPU, 4 GiB sandbox costs $0.08 an hour when
fully busy ([pricing](/docs/pricing), 26 October 2026).

## Why screenshot tests fail on the wrong machine

Playwright's `toHaveScreenshot()` saves reference screenshots on the first
run and compares later runs against them. Its documentation warns that
"browser rendering can vary based on the host OS, version, settings, hardware,
power source (battery vs. power adapter), headless mode, and other factors".
It advises running the tests "in the same environment where the baseline
screenshots were generated"
([Playwright visual comparisons](https://playwright.dev/docs/test-snapshots)).

A baseline made on a developer's laptop and compared on a CI runner fails on
anti-aliasing and font hinting, not on real changes. The fix is one environment
for both jobs. Make baselines in the sandbox, and compare in the sandbox.

## Build the image once

Put the fonts and the browser in an image. Use the Playwright version that your
`package.json` pins, so the browser build matches the test runner:

```ts check
import { Runtime } from "withruntime";

const runtime = new Runtime();
const image = await runtime.images.build({
  name: "visual-tests",
  recipe: {
    apt: ["fonts-noto-core", "fonts-noto-color-emoji"],
    env: { PLAYWRIGHT_BROWSERS_PATH: "/opt/ms-playwright" },
    commands: [
      "npm init -y && npm install --no-fund --no-audit playwright@1.64.1",
      "npx playwright install --with-deps chromium",
    ],
  },
});
console.log(`pin this: visual-tests@${image.version}`);
```

```python check
from withruntime import Runtime

runtime = Runtime()
image = runtime.images.build(
    name="visual-tests",
    recipe={
        "apt": ["fonts-noto-core", "fonts-noto-color-emoji"],
        "env": {"PLAYWRIGHT_BROWSERS_PATH": "/opt/ms-playwright"},
        "commands": [
            "npm init -y && npm install --no-fund --no-audit playwright@1.64.1",
            "npx playwright install --with-deps chromium",
        ],
    },
)
print(f"pin this: visual-tests@{image['version']}")
```

Recipe commands run as root. `PLAYWRIGHT_BROWSERS_PATH` puts the browser where
the sandbox's own user can find it. Playwright 1.64.1 was the current release on 25 October 2026. Record the version that the build prints in your CI config.

## Compare, and bring the diffs back

Each run uploads the project and runs the suite in the pinned image. On a
failure, it downloads Playwright's `test-results` folder, which holds the
expected, actual and diff images:

```ts check
import { Runtime } from "withruntime";

const runtime = new Runtime();
const IMAGE = "visual-tests@3"; // the version your baselines were made with

export async function compareScreens(update = false) {
  await using sbx = await runtime.sandboxes.create({
    image: IMAGE,
    diskMiB: 8192,
    timeoutSeconds: 1800,
  });
  await sbx.files.upload("./", "/workspace/site");
  await sbx.exec("npm ci", { cwd: "/workspace/site", check: true, timeoutMs: 600_000 });
  const run = await sbx.exec(`npx playwright test${update ? " --update-snapshots" : ""}`, {
    cwd: "/workspace/site",
    timeoutMs: 1_200_000,
    onStdout: (text) => process.stdout.write(text),
  });
  if (update)
    await sbx.files.download("/workspace/site/tests", "./tests"); // the new baselines
  else if (run.exitCode !== 0)
    await sbx.files.download("/workspace/site/test-results", "./test-results");
  return run.exitCode === 0;
}
```

```python check
from withruntime import Runtime

runtime = Runtime()
IMAGE = "visual-tests@3"  # the version your baselines were made with


def compare_screens(update: bool = False) -> bool:
    with runtime.sandboxes.create(image=IMAGE, disk_mib=8192, timeout_seconds=1800) as sbx:
        sbx.files.upload(".", "/workspace/site")
        sbx.exec("npm ci", cwd="/workspace/site", check=True, timeout_ms=600_000)
        flag = " --update-snapshots" if update else ""
        run = sbx.exec(f"npx playwright test{flag}", cwd="/workspace/site", timeout_ms=1_200_000)
        if update:
            sbx.files.download("/workspace/site/tests", "./tests")  # the new baselines
        elif run.exit_code != 0:
            sbx.files.download("/workspace/site/test-results", "./test-results")
        return run.exit_code == 0
```

Run it with `update` once to make the baselines. Commit them. Then run it
without `update` on every pull request. Leave `node_modules` out of the upload
with your own ignore step if it is large. The sandbox installs its own copy.
Playwright's `maxDiffPixels` option, in the config or per assertion, sets how
many pixels may differ before a test fails.

## Let a reviewer see the page itself

A diff image shows what changed. A reviewer often wants to click around the page
too. Start the site in the same sandbox, and share its port as a private HTTPS
preview:

```ts check
import { Sandbox } from "withruntime";

await using sbx = await Sandbox.create({ image: "visual-tests@3", diskMiB: 8192 });
await sbx.files.upload("./", "/workspace/site");
await sbx.spawn("cd site && npm ci && npm run dev -- --port 3000");
const preview = await sbx.previews.create(3000);
console.log(preview.urlWithToken); // a link that carries its token, for the reviewer
```

The preview is private by default, under `runtimehost.com`. Calling
`previews.rotate(3000)` refuses every token issued so far.

## Screens that are not web pages

For an app that draws its own window, or a page Playwright cannot drive, the
sandbox's desktop takes screenshots of the whole screen at a size you set:

```ts check
import { writeFile } from "node:fs/promises";
import { Sandbox } from "withruntime";

await using sbx = await Sandbox.create({ diskMiB: 8192 });
await sbx.desktop.start({ width: 1280, height: 800 });
await sbx.desktop.open("https://example.com");
await writeFile("screen.png", await sbx.desktop.screenshot());
```

Compare the PNG against its baseline with the diff tool you already use. The
desktop is ready on every server. The first desktop start in a sandbox installs
nothing and takes about a second.

## What stable screenshots need

| Need                                  | How Runtime covers it                                                  |
| ------------------------------------- | ---------------------------------------------------------------------- |
| One environment for baseline and test | A versioned image, pinned with `name@version`                          |
| The same fonts every time             | `apt` fonts in the image recipe                                        |
| The same browser build                | Installed at build time, not at test time                              |
| Diffs to look at                      | `files.download` brings back `test-results`                            |
| A live look for reviewers             | `previews.create(port)`, private with a token                          |
| Whole-screen captures                 | The desktop's `screenshot()` at a fixed width and height               |
| Parallel runs                         | Up to 100 sandboxes at the same time, on a paid account |

## What it costs

Take 400 visual test runs a month. Each run keeps a 2 vCPU, 4 GiB sandbox
running for 5 minutes, install included. The browser uses 1 vCPU on average.
The image is stored at 2 GB:

```
CPU:    400 × 5 min / 60 × 1 vCPU × $0.025     = $0.83
Memory: 400 × 5 min / 60 × 4 GiB × $0.0075     = $1.00
Image:  2 GB × $0.08                           = $0.16
Total:                                           $1.99
```

A stored image is billed on its accounted storage at $0.08 per GB per 30-day month.
Building one is free. An account without credit stores its first three images
free ([pricing](/docs/pricing#snapshots-images-and-volumes)). Every account gets
100 free hours of a 1 GB machine in its first 30 days, with no card required.

## Start

```bash no-run
npx withruntime sandboxes run --keep -- node --version
```

The first run prints a link to approve in your browser. Then
`runtime images build` and `runtime images versions visual-tests` manage the
image from a terminal ([custom images](/docs/images)).

Related: [Playwright in a cloud sandbox](/integrations/playwright),
[end-to-end testing in sandboxes](/use-cases/end-to-end-testing),
[preview agent-built apps](/use-cases/preview-agent-built-apps),
[a browser automation agent](/use-cases/browser-automation-agent).

## Sources

Checked 26 October 2026.

- [Playwright, visual comparisons](https://playwright.dev/docs/test-snapshots)

A stored image is charged on its whole file
([image storage pricing](/docs/pricing#snapshots-images-and-volumes)).

Facts on this page were checked on 26 October 2026.