Skip to content
docs

Sandbox lifecycle

A sandbox microVM does not have to stay running. Pause snapshots its memory and disk and frees its concurrency slot. Resume loads the snapshot back into the same vm_id. Fork loads it into several new VMs at once, so you can try N approaches from one warmed-up state. A VM also suspends itself when idle and wakes on the next request. The state machine behind this is in microVM lifecycle.

Sandbox lifecycleSchedule, idle-suspend with transparent resume, explicit pause and resume, and copy-on-write forks.

1. POST /v1/runtime/schedule

AgentSpec: image, isolation, limits, egress, secrets, and optional idleTimeoutSecs.

server/main.gohandlers/runtime.gohandlers/runtime.go
You want Call Quota slot
Stop paying for an idle VM, keep its state POST /v1/runtime/vms/{id}/pause Freed
Get it back, same vm_id POST /v1/runtime/vms/{id}/resume Reserved again (402 if exhausted)
N copies of a warm VM POST /v1/runtime/vms/{id}/fork N reserved up front, all or nothing
Pause automatically idleTimeoutSecs on the schedule request Freed on suspend, reserved on wake

Idle suspend

Section titled “Idle suspend”

Set idleTimeoutSecs on the schedule request (--idle-timeout N on the CLI; 0 or unset means never). When a sandbox sees no exec, tool call or file request for that long, it is snapshotted and its compute and quota slot are released. The scheduler sweeps every 10 s (DefaultIdleSweepInterval in services/runtime-scheduler/internal/handlers/idle_sweeper.go) on activity the runtime manager reports with its 30 s node heartbeat (INTERVAL in services/runtime-manager/src/scheduler_heartbeat.rs), so suspension can trail the timeout by up to about 40 s.

The next exec or file request (REST, CLI or SDK) resumes the VM transparently. The caller sees a slower first response while the restored guest re-identifies and reconnects, not an error. In the committed benchmark (docs/benchmarks/runtime-latency.md, nested virtualization on a laptop) a restored VM answered its first exec after 2,491 ms (p50). A VM paused by an operator (POST /pause) is never auto-resumed: exec answers 409.

Forks

Section titled “Forks”

Forks share the parent snapshot’s memory copy-on-write, so N forks cost one snapshot’s memory plus whatever each child writes. Each fork gets its own private copy of the captured disk, its own vm_id, certificate and agent identity. The per-call cap defaults to 100; set PAL_MAX_FORK_COUNT (ceiling 1000) to change it.

Snapshots

Section titled “Snapshots”

A snapshot captures guest memory, the vmstate and the writable disk. The disk is copied while the VM is paused for the memory snapshot, so memory and disk describe the same instant. It is stored as three artifacts:

Artifact Contents
snapshot Firecracker vmstate
mem Guest memory, as large as the VM’s RAM
rootfs The captured disk

Resume, idle wake and each fork copy the snapshot’s rootfs to a per-VM writable path (or into the jail). Files written before the snapshot are still there, and writes after it stay private to that VM. A legacy snapshot taken before disks were captured has no rootfs; it restores onto the base image, which does not match its memory, so re-snapshot long-lived VMs.

Snapshots belong to the tenant that created them. Restoring another tenant’s snapshot returns NOT_FOUND (ADR 0025). A restored VM re-identifies with a fresh identity instead of reusing the source VM’s.

Where snapshots are stored

Section titled “Where snapshots are stored”

The runtime manager keeps the canonical copy on local disk and optionally mirrors it to S3 or MinIO. Reads go to the local tier first and fall back to S3.

Env var (runtime manager) Default Purpose
SNAPSHOT_DIR /var/lib/pal/snapshots Local tier root, laid out <agent_version_id>/<vm_id>/<snapshot_id>/
S3_ENDPOINT + S3_BUCKET unset (S3 tier off) Enable the mirror under snapshots/<agent_version_id>/<vm_id>/<snapshot_id>/
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY unset S3 credentials

An S3 upload failure is logged and does not fail the snapshot; the local copy is already safe. Retention runs after every write: the newest 3 snapshots per VM are kept and anything older than 7 days is deleted.

Cost and latency

Section titled “Cost and latency”

Each snapshot stores mem plus a full copy of the disk, and every restore or fork copies the disk again. The copy reflinks on XFS or btrfs and byte-copies elsewhere (see Operations). On the jailed path forks share the memory file through a hard link, so only the disk is copied per fork.

These numbers come from docs/benchmarks/runtime-latency.md (run 2026-10-29, 20 iterations): a Lima nested-virtualization guest on an Apple M4 Max, 1 vCPU, 128 MiB VM, jailed. Figures are p50. They are preliminary dev-laptop figures, not production numbers.

Step Indicative
Snapshot RPC (pause guest, write memory and disk, resume) 702 ms
Restore (Spawn RPC) 131 ms
Fork spawn (N=1) 151 ms
First exec after restore 2,491 ms
First exec after fork (N=1) 2,722 ms
Stop 198 ms

After a restore the manager holds the first exec until the restored harness has re-identified and sent a heartbeat (PAL_HARNESS_READY_TIMEOUT_MS, default 5 s), so a caller makes one call instead of retrying. Bare-metal KVM has not been measured.

REST

Section titled “REST”
POST /v1/runtime/vms/{id}/pause
-> { "vmId": "...", "snapshotId": "...", "state": "paused" }


POST /v1/runtime/vms/{id}/resume          # 402 quota exhausted, 409 VM not paused
-> { "vmId": "...", "snapshotId": "...", "state": "running" }


POST /v1/runtime/vms/{id}/fork
{ "count": 3 }
-> { "sourceVmId": "...", "snapshotId": "...",
     "forks": [ { "vmId": "...", "state": "running" },
                { "vmId": "...", "state": "failed", "error": "..." } ] }

GET /v1/runtime/vms/{id} also returns snapshotId and parentVmId (set on forks), and state may be paused. Restoring onto a fresh TAP device (resume, idle wake, fork; not broker mode) uses snapshot-load network_overrides, which needs Firecracker 1.12 or later with the jailer (tested on 1.15).

Streaming exec

Section titled “Streaming exec”

POST /v1/runtime/vms/{id}/exec buffers output by default. Send Accept: text/event-stream to receive stdout, stderr, then exit events (or error) as they happen. Exec is execve-style: command is the executable and argv its arguments. Pass /bin/sh with ["-c", ...] for a shell line.

Files

Section titled “Files”

Read and write files under the VM’s /workspace. These calls run through the in-guest tool runner, so paths are confined to /workspace (escapes return 400) and files are accessed as the workload user. Content is UTF-8 text, at most 1 MiB: writes above that return 413, and larger reads are cut and flagged with X-Pal-Truncated: true. Firecracker VMs only; other backends return 503.

GET /v1/runtime/vms/{id}/files?path=src            -> { path, entries: [{name, type, size}], truncated }
GET /v1/runtime/vms/{id}/files/content?path=a.txt  -> raw text
PUT /v1/runtime/vms/{id}/files/content?path=a.txt  # body = file bytes -> { path, bytesWritten }

From code and the CLI

Section titled “From code and the CLI”

The Go SDK exposes PauseVM, ResumeVM, ForkVM, ListFiles, ReadFile and WriteFile.

const { snapshotId } = await client.runtime.pause(vmId);
await client.runtime.resume(vmId);                        // same vm id
const { forks } = await client.runtime.fork(vmId, { count: 3 });
for (const f of forks) if (f.error) console.warn(f.vmId, f.error);


await client.runtime.writeFile(vmId, "job.py", "print('hi')");
const { content } = await client.runtime.readFile(vmId, "job.py");
const { entries } = await client.runtime.listFiles(vmId);


for await (const ev of client.runtime.execStream(vmId, {
  command: "/bin/sh",
  argv: ["-c", "pip install -q requests && python job.py"],
})) {
  if (ev.type === "exit") console.log("exit", ev.exitCode);
  else process.stdout.write(ev.data);                     // stdout | stderr
}

The dashboard VM detail page (/runtime/{vm}) has Pause, Resume and Fork actions. See also Isolation and CLI.