PAL Runtime is a set of real processes joined by real calls. Callers sit outside, a control plane owns tenants, runs and budgets, and a data plane in your VPC executes steps and runs sandboxes. The microVM sits inside the data plane, and the data stores are shared. The overview is How PAL Runtime works. The picture of the control-plane and data-plane split is on Production deployment, and the run and sandbox sequences are on Request lifecycle and microVM lifecycle.
| Component |
Language / port |
Role and boundary |
| SDKs |
TypeScript, Python 3.11+, Go |
Typed clients for the REST API. Untrusted caller: every request carries a Bearer JWT or API key that the control plane validates. Models are addressed by capability (auto, reasoning-large), never by vendor name. |
pal CLI |
Go (Cobra) |
Static binary over the same REST API: runs, sandboxes (pal vm ...), onboarding. Authenticates like the SDKs. |
| Dashboard |
TypeScript (Next.js), :3001 in dev |
Operator UI over the same REST endpoints: agents, runs and their journal waterfall, sandboxes, budgets. |
| LLM providers |
Anthropic, OpenAI (external HTTPS) |
Reached directly by the control plane’s failover chain (default) or through the model router. Calls carry an Idempotency-Key. Tenant provider keys are AES-256-GCM encrypted at rest and never logged. |
| Internet |
External |
Whatever the sandbox’s egress rules allow. Everything else is denied and audited. Private and metadata ranges (including 169.254.169.254) are refused even when a hostname resolves to them. |
| Component |
Language / port |
Role and boundary |
| Gateway |
Rust (Axum), :8443 TLS |
Edge ingress. Terminates TLS, validates API keys by asking the control plane (cached 60 s by key hash), calls the control plane over gRPC with x-pal-service-token. Refuses API-key auth when introspection is not configured. Without PAL_TLS_CERT and PAL_TLS_KEY it refuses to start in production; in dev it serves plaintext and warns. |
| Control plane |
Go, :8080 HTTP (REST, SSE), :50051 gRPC |
Owns agents, runs, budgets and tenants. Gates every run (auth, budget, isolation), runs shared-tier agents inline, journals every step to Postgres. Only callers presenting the service token may set a tenant_id on :50051. Refuses to start in production without the token. |
| Model router |
Rust, :50053 gRPC |
ModelService: Complete, CompleteStream, Embed, Tokenize. Maps a capability to a concrete provider model, with caching. It carries tenant provider keys, so it refuses to start in production without the token. |
Control plane invariants:
- Hard-fail budgets block a run with HTTP 402 before any compute or tokens are spent.
- Every tenant-scoped query runs through
WithTenant so Postgres RLS can enforce it.
- A crashed run is re-driven from its journal by a 30 s recovery sweep.
The control plane uses the model router only when PAL_USE_MODEL_ROUTER is on (default off), and any router error falls through to the direct provider chain. Workflow-engine llm_call steps always go through the router.
| Component |
Language / port |
Role and boundary |
| Data-plane agent |
Go |
Runs in your VPC. Opens a RunStream to the control plane, receives assignments, drives each on the local workflow engine. It dials out; the control plane never connects in. Each plane is bound to one tenant. With no plane connected, the run executes inline in the control plane. |
| Workflow engine |
Go, :50052 gRPC |
Durable step executor: llm_call via the model router, tool_call via the runtime manager, child_run in-engine. Each attempt is replay-checked first. CancelRun, ResumeRun and SignalRun filter by the verified tenant_id and return NotFound on mismatch. |
| Runtime scheduler |
Go, :50055 gRPC, :8085 REST |
Places sandboxes on nodes (warm pool, capacity, fair share) and owns lifecycle: pause, resume, fork and the idle-suspend sweeper. Reserves tenant quota atomically (all or nothing for forks). |
| Runtime manager |
Rust (tonic), :50054 gRPC (mTLS), one per node |
The only process that touches microVMs: spawn, stop, snapshot and restore, exec, ExecTool. Serves RuntimeHarness for the guests on its node. |
| Host egress broker |
Rust, inside the runtime manager, opt-in |
Per-VM proxy on the host end of vsock port 3128. A new trust boundary: host code parsing guest bytes. Request heads are capped at 64 KiB and error strings returned to the guest are static. |
| Harness |
Rust (musl, static), PID 1 in the VM, :50056 exec in the guest |
Init process inside every microVM: supervises the workload, serves exec and tool calls, vends short-lived secrets, enforces egress. The last in-VM boundary. |
| Untrusted code |
Anything |
Sees /workspace, the proxy env, and nothing that identifies the control plane. In broker mode it has no network device. |
Invariants by component:
| Component |
Invariant |
| Workflow engine |
Every side effect carries the key run_id:step_id:attempt. A missing router or manager address fails the step with a typed error, never a made-up result. |
| Runtime scheduler |
A PAUSED VM holds no concurrent-VM slot. Lifecycle changes on one VM serialize on a per-VM lock; only the leader replica mutates. |
| Runtime manager |
A VM owned by another tenant looks absent (NOT_FOUND). Untrusted code runs in Firecracker or Kata, never a bare pod. |
| Egress broker |
Opt-in (PAL_EGRESS_MODE=broker, Firecracker only; default is guest). HTTPS is terminated only for hosts whose rule injects a credential. The credential never enters the VM. |
| Harness |
Security audits are logged locally before forwarding. Tool file operations are confined to /workspace. |
| Store |
Port |
Role |
| Postgres 16 with pgvector |
:5432 |
System of record: tenants, agents, runs, journal_events, run_locks, budgets and usage rollups. Every tenant table has a tenant_isolation RLS policy. journal_events is append-only, keyed (run_id, seq). |
| Redis |
:6379 |
Caching, rate limiting and session event pub/sub for SSE. Internal only. |
| S3 or MinIO |
:9000 |
Agent bundles, attachments and the optional second tier of microVM snapshots. Uploads are best-effort; the local copy on the node is authoritative. |
| From |
To |
Call |
| SDK |
Gateway |
HTTPS :8443 |
| SDK, CLI, dashboard |
Control plane |
REST /v1 |
| Gateway |
Control plane |
gRPC :50051 |
| Control plane |
Model router |
Complete, behind a flag |
| Data-plane agent |
Control plane |
RunStream, dialed out |
| Data-plane agent |
Workflow engine |
ExecuteRun |
| Workflow engine |
Model router |
llm_call |
| Workflow engine |
Runtime manager |
ExecTool |
| Control plane |
Scheduler |
Schedule, Pause, Resume, Fork |
| Control plane |
Runtime manager |
Exec and files |
| Scheduler |
Runtime manager |
Spawn, Stop, Snapshot |
| Runtime manager |
Control plane |
Report and secret relay |
| Harness |
Runtime manager |
Heartbeat and Report up; Exec on :50056 down |
| Workload |
Harness |
:3128. Guest mode proxies out over the TAP device; broker mode forwards over vsock 3128 to the broker. |
| Control plane, workflow engine, scheduler |
Postgres |
SQL with RLS, journal |
One per node, listening on :50054. Callers cross the authz interceptor; guests reach only RuntimeHarness.
| Part |
Role and boundary |
Authz interceptor (authz.rs) |
Per-call authorization for RuntimeManager (ADR 0025). The TLS handshake cannot tell a guest from the scheduler, because every per-VM certificate is signed by the trusted CA. A peer presenting a VM certificate (OU=pal-vm) is refused; everyone else needs the service token (constant-time compare). Token unset: production refuses to start, dev warns and allows. |
RuntimeManager service |
Spawn (fresh or restore_snapshot_id), Stop, Logs, Exec (bidi stream), ExecTool, Stats, Snapshot. ExecTool resolves the VM by vm_id or run_id. Tool output buffered in manager memory is capped at 1 MiB; tool timeout defaults to 30 s, capped at 900 s. |
RuntimeHarness service |
What guests call: a 5 s heartbeat stream (also how the manager learns the guest’s dial-back address), secret vending and the report stream. Each method binds the certificate CN or SAN to the request’s vm_id, pinned to this incarnation’s certificate. VendSecret resolves only URIs declared in the VM’s own spawn record. |
| Handle registry |
Every live sandbox on the node: tenant, backend, declared secrets, certificate pin and exec token. The source of truth for ownership checks. |
| Isolation backends |
One RuntimeBackend trait (schedule, cancel, snapshot, restore, exec) with Firecracker, Kata, Kubernetes, Wasmtime and Docker implementations. An isolation class the backend cannot satisfy is refused, never downgraded. Only Firecracker VMs run the in-guest tool registry; ExecTool elsewhere returns TOOL_STATUS_UNAVAILABLE. |
| Warm pool |
Pre-booted handles keyed by (isolation class, bundle digest, tenant), published to the scheduler for warm-start scoring. The tenant is in the key, so a handle booted with one tenant’s env and certificate is never handed to another. |
| Snapshot store |
vmstate and mem files under SNAPSHOT_DIR, mirrored to S3 when configured. mem is made read-only so forks can hard-link it. S3 failures are warn-logged, never fatal. |
| Reaper |
Polls backend liveness and deregisters workloads that exited, so finished VMs stop counting against quota. It reaps only on an affirmative “not running”; an error keeps the handle. |
| Scheduler heartbeat |
Posts node capacity and warm-pool inventory to the scheduler’s REST gateway (:8085) every 30 s. |
| Secret resolver |
Resolves pal.secret/... URIs at request time through the control-plane relay (POST /v1/runtime/secrets/resolve), or from env in dev. The relay authenticates with the X-Pal-Runtime-Token header; the control plane fails closed without the token. |
| Report forwarder |
Forwards each harness report (logs, OTLP traces, metrics, audit events) to POST /v1/runtime/report. A forward failure never crashes the workload; failed audit forwards log at ERROR. |
| Egress broker (broker mode) |
Per-VM listener on the Firecracker vsock socket for guest port 3128: allowlist, upstream dial from the host, credential injection. At most 128 concurrent connections per VM; 10 s limits on head read, TLS, DNS and connect; 300 s idle close. Production refuses broker mode without the relay resolver. |
The harness starts as PID 1 and is the last trust boundary around the workload. The workload reaches it only through the proxy port, the secrets socket and its own stdio.
| Part |
Role and boundary |
| Init and boot env |
Mounts /run/pal, sets rlimits, reads spawn.json (workload command, env, egress rules, workload uid and gid) from the read-only cert drive. Unknown spawn.json versions are rejected. In production it refuses to start without a workload uid. |
| Supervisor |
Spawns the workload as the workload uid, tails stdio, restarts on crash, injects the proxy env. Every PAL_* key is withheld from the child except a short allowlist of non-secret identifiers. |
| Signals and reaper |
PID 1 duties: reaps orphaned zombies, drains on SIGTERM, prepares for snapshots on SIGUSR1. |
| Heartbeat |
Bidi stream to the manager every 5 s with resource usage, has_active_exec and last_activity_unix_ms; the scheduler’s idle sweeper reads the last two. mTLS with this incarnation’s certificate. |
| Activity tracker |
Atomic count of running workload and exec children, plus when the last one exited. Idle means no child alive. |
| Report stream |
Client-streaming RuntimeHarness.Report to the manager, with reconnect. Security audits (secret_vend, egress deny, secret_access_denied, egress_preflight) are logged locally first and never dropped on a full buffer. |
| Exec server |
:50056, bearer token. Serves one-shot or TTY commands and the __pal_tool_call__ envelope for tool calls. Rejects calls without this incarnation’s token, so a token frozen into a snapshot is refused after restore. |
| Tool runner |
Five tools: shell_exec, http_fetch (through the egress proxy), read_file, write_file, list_dir. File tools are confined to /workspace via openat2 RESOLVE_BENEATH, under the workload’s fsuid. Every captured output is capped at 1 MiB and marked truncated. |
| Secrets socket |
/run/pal/secrets.sock. Vends short-lived secrets fetched from the manager’s VendSecret. The peer uid comes from the kernel (SO_PEERCRED) and must be the workload uid; denials emit secret_access_denied. |
| Egress proxy (guest mode) |
127.0.0.1:3128, an in-guest CONNECT allowlist proxy. It enforces only with the image’s iptables REDIRECT rule, which the harness checks at boot. In production, declared rules with no REDIRECT mean the workload is not started. |
| Egress forwarder (broker mode) |
127.0.0.1:3128. Bridges each proxy connection byte for byte to vsock CID 2, port 3128. Nothing in the guest enforces anything; the VM has no network device and the host broker decides. |
| Re-identify |
A restored guest wakes up believing it is the source VM. This mints the new incarnation’s vm_id, certificate and agent identity before anything else runs. |
The workflow engine asks the runtime manager, which hands a JSON envelope to the harness. The happy path is on Request lifecycle. The details that matter here:
- The manager sends
Exec __pal_tool_call__ with { tool, args, idempotency_key, timeout_sec } in argv[0], plus this incarnation’s bearer token. On broker VMs the same gRPC rides host-initiated vsock.
- The exec server intercepts the sentinel before its generic exec audit, so arguments are never logged. The manager audits the tool name, duration and status, never the arguments.
shell_exec runs /bin/sh -c as the workload uid. read_file uses openat2 with RESOLVE_BENEATH, so no .. or symlink escape.- The runner writes
{ ok: true, result } to stdout before the done frame. The status is typed; there is no string matching.
| Case |
Result |
Tool ran and failed, or the name is unknown (for example read_file outside /workspace) |
The runner returns { ok: false, error }; the manager returns TOOL_STATUS_ERROR and the step fails with the manager’s detail. |
| VM is not Firecracker, or has not heartbeated yet |
TOOL_STATUS_UNAVAILABLE. Never a made-up success; the engine fails the step wrapping ErrRuntimeManagerUnavailable. |
| VM belongs to another tenant |
The mismatch is audited under pal_audit as exec_tool DENIED and the manager returns NOT_FOUND, the same answer as a VM that does not exist (ADR 0025). |
The workload’s request leaves the VM over vsock. The host broker decides, dials and adds the credential, and the VM never holds it. The picture and the rules are on Egress.
| Case |
What happens |
Allowed, credential injected (for example api.github.com) |
See the steps below. |
Allowed, no injection (for example pypi.org) |
The rule has no inject list, so TLS is not terminated. The broker resolves once, refuses private and metadata answers, dials the checked address and answers 200 Connection Established. From there it is an opaque TLS tunnel; the broker only splices bytes. |
| Not allowlisted |
Policy.evaluate returns the deny reason. The workload gets 403 Forbidden with an X-Pal-Reason header carrying a static reason. Nothing is dialed. The deny is always logged locally and reaches the shared audit channel at a burst of 20, then 5 per second per VM. |
| Plain HTTP to a credential host |
The broker returns 403 plain-HTTP injection refused, because the credential would cross the network in cleartext. Only PAL_EGRESS_PLAIN_INJECT=1 allows it, and only to port 80. |
| Host mismatch |
After TLS termination the workload sends a request with a different Host:. The check runs before any secret is resolved or anything is dialed; the broker audits an egress_inject deny and returns 403. The credential never rides a request addressed elsewhere. |
Injected request, step by step:
- The workload sends
CONNECT api.github.com:443 through HTTPS_PROXY=http://127.0.0.1:3128.
- The forwarder splices bytes to vsock CID 2, port 3128. The VM has no network device.
- The broker reads the head (at most 64 KiB, 10 s). A slow or silent guest is dropped.
Policy.evaluate checks the port (PAL_EGRESS_ALLOWED_PORTS, default 443), rejects private or metadata IP literals, and takes the first matching rule.- The broker answers
200 Connection Established. This rule injects, so it terminates TLS with a leaf from the per-VM ephemeral CA. Nothing is dialed yet.
- The workload sends its real request inside TLS, with a placeholder or no
Authorization header.
- Preflight: not
TRACE or CONNECT, and the target and exactly one Host: header must name the approved host.
- The broker resolves the
pal.secret/... reference on the host at request time, scoped to the VM’s tenant. The reference comes from the manager’s spawn record, not the guest.
- The value is held for one header write, then dropped. It is never logged, traced or audited.
- The broker dials upstream over TLS with the header injected: DNS checked against private ranges, upstream verified against native roots, same-named header replaced,
Connection: close.
- It audits
egress_inject with header names only.
- The response is spliced back, rate-limited by the rule’s
rate_bps. Idle connections close after 300 s.