microVMs on your machine

macOS has no KVM, so on a Mac the default scheduler is a stub that spawns nothing. This page gives an Apple Silicon Mac (M3 or newer, macOS 15 or newer) a real /dev/kvm through a Lima Linux guest with nested virtualization. The runtime manager runs inside the guest with RUNTIME_BACKEND=firecracker, so you can schedule, exec into, pause, resume and fork real Firecracker microVMs from your laptop. It is for development; it is not a production topology.

Caution

Provisioning fails with FATAL: /dev/kvm missing on an M1/M2 or a pre-macOS-15 host. There, use the KVM CI workflow (.github/workflows/microvm-integration.yml), a remote Linux host, or the Docker backend (RUNTIME_BACKEND=docker), which stays the zero-setup macOS path.

Real or stub

	Real Firecracker (this page)	Stub scheduler (default)
Isolation	Firecracker microVM on KVM	None: nothing is spawned
Placement	Real VMs you can exec into, pause, resume and fork	Faked: a generated vm id on node node-stub, zone az-stub
Setup	Lima guest, runtime manager built inside it	None

Without PAL_SCHEDULER_GRPC_ADDR the control plane uses a stub scheduler client. Without PAL_DEFAULT_MANAGER_ADDR the scheduler keeps a log-only dialer. Both are fine for dashboard work and both are rejected in production. For the topology around this, see Deployment topology.

Set up

You need Lima (brew install lima). The repo must live under your home directory, which the guest mounts read-only at the same path. The Lima template (infra/lima/firecracker-dev.yaml) uses vmType: vz with nestedVirtualization: true, 4 CPUs, 8 GiB RAM and a 30 GiB disk, and installs the pinned Firecracker v1.15.1 release plus the jailer.

brew install lima
make lima-up          # create/start "firecracker-dev"; check /dev/kvm, firecracker, jailer
make lima-e2e         # build harness, manager and image in the VM; run the jailed lifecycle test
make lima-broker-e2e  # broker egress mode (ADR 0026): no NIC, vsock only, credential injection

Target	What it does
make lima-up	Creates the VM if absent, starts it if stopped, does nothing if running. Fails fast on M1/M2 or macOS older than 15.
make lima-image	Builds vmlinux and rootfs.ext4 with the current harness as PID 1. Cached in the VM at ~/.cache/pal-lima/image; rebuilt only when the harness binary or build-image.sh changes.
make lima-e2e	lima-image, then builds the runtime manager and runs infra/firecracker/integration-test.sh jailed: boot, snapshot, stop, restore, fork x3. Saves a timing table to ~/.cache/pal-lima/timing/lifecycle-timing.md in the VM.
make lima-broker-e2e	Same build, then infra/firecracker/broker-e2e.sh. Needs outbound HTTPS to httpbin.org.
make lima-measurement-e2e	Software measurement (ADR 0031) end to end, jailed.
make lima-bench	Latency bench: cold boot, exec, pause, resume, fork, broker.
make lima-runtime	Same build, then a long-lived jailed runtime manager in the foreground (Ctrl-C stops it) on :52054 in the guest. Lima forwards it to the Mac’s localhost:52054 and it registers itself with the host scheduler, so pal dev can use it. Dev only: plaintext gRPC, no service token.
make lima-down	Stops the VM.
make lima-clean	Drops the VM’s cached cargo target and microVM image.

Both e2e targets fail if a run leaves a firecracker process, snapshot directory or jail behind. If another run holds the manager port (:50054, broker :51054) they wait up to LIMA_PORT_WAIT_SECS (default 1800) instead of killing it. Builds refuse to start when the VM disk is over LIMA_DISK_MAX_PCT (default 85%); make lima-clean frees the caches. Build output goes to ~/.cache/pal-lima in the VM, never into the repo.

Check it worked:

• make lima-e2e ends with ALL ASSERTIONS PASSED, including the fork step, and prints the timing table.
• Inside the guest (limactl shell firecracker-dev), /dev/kvm exists and firecracker --version prints v1.15.1.

The fork step runs only when FIRECRACKER_JAILER_PATH points at the jailer (/usr/local/bin/jailer in the Lima guest). make lima-e2e sets it. Without it the test prints SKIP: fork test SKIPPED and still passes.

Smoke-boot one microVM

To see a Firecracker guest boot without PAL in the way, run this inside the guest with the official Firecracker CI artifacts:

mkdir -p ~/fc-demo && cd ~/fc-demo
curl -sO https://spec.ccfc.min.s3.amazonaws.com/firecracker-ci/v1.15/aarch64/vmlinux-5.10.245
curl -sO https://spec.ccfc.min.s3.amazonaws.com/firecracker-ci/v1.15/aarch64/ubuntu-22.04.ext4
cat > vm.json <<'EOF'
{
  "boot-source": {"kernel_image_path": "vmlinux-5.10.245", "boot_args": "keep_bootcon console=ttyS0 reboot=k panic=1 pci=off"},
  "drives": [{"drive_id": "rootfs", "path_on_host": "ubuntu-22.04.ext4", "is_root_device": true, "is_read_only": false}],
  "machine-config": {"vcpu_count": 2, "mem_size_mib": 512}
}
EOF
firecracker --no-api --config-file vm.json   # boots to a root login on ttyS0

It reaches a root login on ttyS0. Clean up with limactl stop firecracker-dev && limactl delete firecracker-dev.

Point the stack at it

make lima-runtime is the quickest wiring. If you run the manager yourself, set these on the Mac, assuming the manager is reachable at localhost:50054:

Env var	Set on	Value
PAL_SCHEDULER_GRPC_ADDR	control plane	localhost:50055
PAL_DEFAULT_MANAGER_ADDR	scheduler	localhost:50054
PAL_NODE_ADDR_<NODE>	scheduler	host:port, only when the scheduler picks a named node whose address it cannot discover

Dev setups without a shared service token warn and allow unauthenticated gRPC. Production requires PAL_GRPC_SERVICE_TOKEN on every hop. The other backends are covered in Isolation.

Check it worked: a scheduled VM’s node is not node-stub, and GET /v1/runtime/vms/{id} shows it running.

Troubleshooting

Symptom	Cause and fix
FATAL: /dev/kvm missing	M1/M2 or older than macOS 15. Use CI, a remote Linux host, or RUNTIME_BACKEND=docker.
Build refuses to start: disk over the limit	Run make lima-clean, or raise LIMA_DISK_MAX_PCT.
An e2e target waits on :50054	Another manager run holds the port. It waits up to LIMA_PORT_WAIT_SECS; check ss -ltnp | grep 50054 in the guest.
First exec after a restore takes seconds	The manager holds it until the restored harness re-identifies and heartbeats, and the guest is slow under nested virtualization. See Sandbox lifecycle.
Node shows node-stub	PAL_SCHEDULER_GRPC_ADDR is unset on the control plane, so the stub client fakes placement.

Limits

• Nested virtualization is for development. Lima timings are indicative, not bare-metal numbers.
• The artifacts above are aarch64 (Apple Silicon) only.
• The repo mount is read-only, so build outputs land inside the guest.