# How to run Hugging Face Transformers models on CPU in a sandbox

Install PyTorch's CPU wheel and `transformers`, cache the model in an image, and run the pipeline with the internet off.

**On Runtime you pay for the CPU the model uses, and the model files can be
baked in once.** Inference that keeps 4 vCPUs busy costs $0.10 an hour for
the CPU, plus $0.0075 per GiB-hour of memory, and a sandbox made from an image
with the model cached starts with nothing to download. Store a gated model's
token as a Runtime secret, which keeps it out of the sandbox. transformers
5.20.2 and PyTorch 2.14.1 were current on 25 October 2026.

## Install the CPU build of PyTorch

PyTorch's default wheel on PyPI is built for NVIDIA GPUs and pulls in CUDA
libraries. Its CPU index serves a much smaller wheel. Use that wheel here, as
the Transformers install guide shows:

| Wheel for Python 3.12 on x86-64 Linux, torch 2.14.1   | Size   |
| ----------------------------------------------------- | ------ |
| `torch` from PyPI, before its `nvidia-*` dependencies | 555 MB |
| `torch` from `download.pytorch.org/whl/cpu`           | 196 MB |

In a test on 25 October 2026, a virtual environment with the CPU wheel and
transformers 5.20.2 took 1.2 GB of disk. The default 4 GiB sandbox has about
2.5 GiB free. Request `diskMiB: 8192` before you add a model.

## Run a pipeline

```ts check
import { Sandbox } from "withruntime";

const classify = `
from transformers import pipeline
clf = pipeline("text-classification", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
for row in clf(["The sandbox ran out of disk.", "The release shipped on time."]):
    print(row["label"], round(row["score"], 4))
`;

await using sbx = await Sandbox.create({
  vcpu: 2,
  memoryMiB: 4096,
  diskMiB: 8192,
  timeoutSeconds: 1800,
});
const slow = { check: true, timeoutMs: 900_000 } as const;
await sbx.exec(
  "pip install -q --no-cache-dir torch==2.14.1 --index-url https://download.pytorch.org/whl/cpu",
  slow,
);
await sbx.exec("pip install -q --no-cache-dir transformers==5.20.2", slow);

await sbx.files.write("/workspace/classify.py", classify);
const run = await sbx.exec(["python3", "/workspace/classify.py"], slow);
console.log(run.stdout); // NEGATIVE 0.9998 for the first sentence
```

```python check
from withruntime import Sandbox

CLASSIFY = """
from transformers import pipeline
clf = pipeline("text-classification", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
for row in clf(["The sandbox ran out of disk.", "The release shipped on time."]):
    print(row["label"], round(row["score"], 4))
"""

with Sandbox.create(vcpu=2, memory_mib=4096, disk_mib=8192, timeout_seconds=1800) as sbx:
    sbx.exec("pip install -q --no-cache-dir torch==2.14.1 --index-url https://download.pytorch.org/whl/cpu",
             check=True, timeout_ms=900_000)
    sbx.exec("pip install -q --no-cache-dir transformers==5.20.2", check=True, timeout_ms=900_000)
    sbx.files.write("/workspace/classify.py", CLASSIFY)
    print(sbx.exec(["python3", "/workspace/classify.py"], check=True, timeout_ms=900_000).stdout)
```

The first run downloads the model from the Hugging Face Hub. For this model,
the pipeline fetched 256 MB: the safetensors weights and tokenizer. It did not
fetch the TensorFlow and Rust copies that the repository also holds.

## Bake the model into an image

Download the model during the image build and set `HF_HOME` so the cache sits
in the image. Every sandbox made from it starts ready:

```ts check
import { Runtime } from "withruntime";

const model = "distilbert/distilbert-base-uncased-finetuned-sst-2-english";
const runtime = new Runtime();
await runtime.images.build(
  {
    name: "sst2-cpu",
    recipe: {
      env: { HF_HOME: "/opt/hf" },
      commands: [
        "pip install --no-cache-dir torch==2.14.1 --index-url https://download.pytorch.org/whl/cpu",
        "pip install --no-cache-dir transformers==5.20.2",
        `python3 -c "from transformers import pipeline; pipeline('text-classification', model='${model}')"`,
        "chown -R 1000:1000 /opt/hf",
      ],
    },
    build: { diskMiB: 8192 },
  },
  { onLog: (line) => console.log(line.text) },
);

await using sbx = await runtime.sandboxes.create({
  image: "sst2-cpu",
  network: { internet: false },
});
const run = await sbx.exec(
  [
    "python3",
    "-c",
    `from transformers import pipeline; print(pipeline('text-classification', model='${model}')('fine'))`,
  ],
  { env: { HF_HUB_OFFLINE: "1" }, timeoutMs: 300_000 },
);
console.log(run.stdout);
```

- **`HF_HOME`:** where `huggingface_hub` keeps its cache and token. Pointing
  it at `/opt/hf` in the recipe's `env` puts the downloaded model in the image.
- **`HF_HUB_OFFLINE=1`:** no calls to the Hub; files come only from the cache.
  With the internet off as well, the model cannot fetch code or weights at run
  time. In a test on 25 October 2026, the cached pipeline ran offline.
- **The build machine:** its scratch disk is 4 GiB by default. `build.diskMiB`
  raises it, up to 32 GiB. An image can be up to 20 GiB.

A stored image is charged at $0.08 per GB per 30-day month on its accounted storage,
so an ordinary 2 GB image costs $0.16 a month to keep. Building it is free.

## Gated and private models

Models such as Llama and Gemma need an access token. Store it as a secret for
`huggingface.co`. Every sandbox then has `HF_TOKEN` set to a placeholder.
`huggingface_hub` sends the placeholder as it would the real token. On its way
to the Hub, the host's proxy replaces the placeholder with the real value:

```python check
import os
from withruntime import Runtime

runtime = Runtime()
runtime.secrets.set("HF_TOKEN", value=os.environ["HF_TOKEN"], hosts=["huggingface.co"])
```

`HF_TOKEN` is the variable `huggingface_hub` reads for the Hub's access token.
Accept the model's license on its Hub page first, with the account that owns
the token.

## How big a model fits

Weights in 32-bit floats take 4 bytes per parameter, so memory, not the CPU,
decides what loads:

| Model size        | Weights in float32 | Sandbox to ask for       |
| ----------------- | ------------------ | ------------------------ |
| 66 M (DistilBERT) | about 0.27 GB      | The default 4 GiB        |
| 0.5 B             | about 2 GB         | `memoryMiB: 8192`        |
| 1.5 B             | about 6 GB         | `memoryMiB: 16384`, paid |

Runtime runs on CPUs only, with no GPUs. The models that suit a sandbox are
encoders, embedding models, classifiers and small generators. For more
throughput, raise `vcpu`. CPU is billed on what the model uses. A batch that
finishes sooner on more vCPUs costs about the same CPU.

## Related

- [Run a Gradio demo in a sandbox](/integrations/gradio)
- [Run untrusted Python code safely](/languages/python)
- [Turn off sandbox internet](/how-to/turn-off-sandbox-internet)
- [Custom images](/docs/images)
- [Secrets sandboxes never see](/docs/security#secrets-sandboxes-never-see)

## Sources

Checked 25 October 2026.

- [Transformers installation](https://huggingface.co/docs/transformers/installation):
  the CPU-only install from `https://download.pytorch.org/whl/cpu`, Python
  3.10 and later, PyTorch 2.6 and later
- [PyTorch CPU wheel index](https://download.pytorch.org/whl/cpu/torch/):
  `torch-2.14.1+cpu-cp312-cp312-manylinux_2_28_x86_64.whl`, 196,253,677 bytes;
  [torch 2.14.1 on PyPI](https://pypi.org/project/torch/2.14.1/): the cp312
  x86-64 wheel is 554,618,695 bytes and requires `nvidia-*` packages on Linux
- [huggingface_hub environment variables](https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables):
  `HF_HOME`, `HF_TOKEN`, `HF_HUB_OFFLINE`
- [distilbert-base-uncased-finetuned-sst-2-english](https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english):
  `model.safetensors`, 267,832,558 bytes
- [transformers on PyPI](https://pypi.org/project/transformers/): version 5.20.2
- A local test on 25 October 2026: torch 2.14.1+cpu and transformers 5.20.2
  in a 1.2 GB virtual environment; the pipeline cached 256 MB, labelled
  "The sandbox ran out of disk." NEGATIVE (0.9998), and ran again with
  `HF_HUB_OFFLINE=1`

A stored image is charged on its whole file
([image storage pricing](/docs/pricing#snapshots-images-and-volumes)).

Facts on this page were checked on 25 October 2026.