Skip to content

Security model

PAL Runtime runs agents for many tenants on shared infrastructure, and some of that code is untrusted. The security model is four trust boundaries: edge authentication, tenant isolation in the database, a service token on every internal gRPC port, and the sandbox around the workload. Each boundary has its own credential, and each one refuses to start or refuses the request when its credential is missing in production. This page lists what is enforced today, and which protections are only designs.

Security trust boundariesWhat a caller must present at each hop, from the internet to code inside a microVM. Step through the boundaries in order.
External calleruntrustedGateway:8443Control plane:50051Model router:50053Postgres RLStenant_isolationWorkflow engine:50052Scheduler:50055Secret relay/v1/runtime/secrets/resolveRuntime manager:50054Egress brokerhostInternetHarnessPID 1Workloaduser codeSecrets socketSO_PEERCRED
Swipe sideways to pan.

1. The edge

TLS plus a JWT or an API key. API keys are introspected by the control plane and cached for 60 s keyed by a SHA-256 of the key; any error fails closed.

  • Untrusted
  • Untrusted code
  • Trusted host
  • microVM wall
  • Refused
Protected From Control When the control is missing
One tenant’s rows Another tenant Row-level security on every tenant-scoped table (USING and WITH CHECK) Production control plane refuses to start without PAL_APP_DB_PASSWORD, unless you set PAL_RLS_ENFORCE=0
Endpoints An API key with too few rights Per-endpoint scopes (agents:write, runs:execute, and so on); a missing scope gets 403 On by default in production, off in dev. PAL_AUTHZ_ENFORCE=0 opts out with a warning
Internal gRPC ports (:50051 to :50055) A caller that is not a PAL service Shared service token (x-pal-service-token, set from PAL_GRPC_SERVICE_TOKEN), checked in constant time before the tenant is read; mutual TLS when PAL_GRPC_TLS_CA, _CERT, _KEY are set (ADR 0043) Control plane refuses to start in production without the token. Without TLS material it refuses too, unless PAL_GRPC_INSECURE=1 (loud warning). Dev falls back to plaintext with a warning
Provider keys and connector tokens Anyone reading the database AES-256-GCM at rest with PAL_CREDENTIAL_KEY Control plane refuses to start in production without it. Dev stores plaintext and warns
The host and other tenants A compromised workload MicroVM per workload, egress allowlist, no credentials inside in broker mode (Isolation, Egress) Untrusted and hostile classes are refused, never downgraded to a bare pod
Another run’s audit stream A compromised microVM Step events and exit reports are bound to that VM’s own run; a report with another run_id gets 403 Rejected
Spend A runaway agent Budgets; a hard-fail budget returns 402 before the run starts If the budget cannot be read, run create returns 503
Private agents A cross-tenant A2A caller The agent card and invoke return 404 unless the agent is public or owned by the caller Rejected

Two more token gates fail closed. The API-key introspection endpoint the gateway calls returns 403 in production when PAL_GRPC_SERVICE_TOKEN is unset. The personal /v1/signals endpoint returns 401 whenever PAL_SIGNAL_TOKEN is unset.

For the exact behavior of scopes and row-level security, see Environment variables. make rls-validate checks the policies against a live database, and a catalog test fails CI when a new tenant table has neither a policy nor an explicit exemption.

These are enforced by the platform, so a prompt injection cannot talk past them.

  • Budgets. Per-agent cost and runs per day, plus a tenant-wide daily cap. See Budgets.
  • Egress allowlist. Declared per spawn. In broker mode the VM has no network device. In the default mode the allowlist binds only when the VM image installs the iptables REDIRECT rule; in production the harness refuses to start the workload without it.
  • Human approval. Workflow approval nodes block until an operator grants the request. Nested approvals inside child runs fail today rather than wait.
  • Confidence gate. Low-confidence tool and connector steps divert to approval. Off by default (PAL_CONFIDENCE_GATE).
  • Signed receipts. Proof of what a run executed. See Receipts.
  • At rest. Connector credentials and LLM provider keys are encrypted at the application layer. Other Postgres columns use plain Postgres storage, and disk encryption is the host’s job. Object-store encryption for bundles and snapshots is a bucket setting on your storage; PAL does not set it.
  • In transit. The gateway terminates TLS with rustls default protocol versions (TLS 1.2 and 1.3). Internal gRPC carries the service token and, when configured, mutual TLS. The data-plane tunnel to the control plane is server-authenticated TLS and does not use client certificates.
  • Browser session. Dashboard auth sets an HttpOnly session cookie. The JWT is also kept in localStorage for client-side API calls, so an XSS bug on the dashboard can read it.
  • Bridge proxy. The dashboard’s /api/bridge/... route adds the messaging bridges’ token server-side, so it requires a valid session cookie, only reaches the session’s own tenant, and refuses writes that are not same-origin application/json (CSRF). The always-on dashboard binds 127.0.0.1 unless PAL_DASHBOARD_HOST says otherwise.
  • Privacy levels (strict, paranoid) are not implemented. Today there is one level: data is encrypted at rest and in transit, and logs include inputs and outputs.
  • PII redaction in inputs, outputs and traces is not implemented.
  • Bring your own key (AWS KMS, Google Cloud KMS, Azure Key Vault) is not available.
  • Audit log is partial. Run events are in journal_events, and A2A card access is tenant-scoped. There is no immutable audit log of secret access, role changes or dashboard actions, and no SIEM export.
  • The Firecracker root filesystem is writable. Only the cert and spawn drive is mounted read-only. Do not assume a workload cannot modify its own rootfs.
  • Broker egress is opt-in. The default is guest mode, which depends on the image installing the iptables REDIRECT rule.
  • Forced isolation for marketplace and LLM-generated bundles is policy in ADR 0002, not enforced at schedule time: a request for trusted is honoured as sent.
  • Launch measurement is host-asserted. It records what the manager booted. It does not prove what ran inside the VM. See Identity and secrets.

Every control-plane HTTP request and gRPC call carries a span with pal.tenant_id, pal.run_id and pal.step_id. govulncheck, cargo audit and npm audit run on every pull request.

You want to know Read
How a sandbox reaches the network, and why it holds no credential Egress
How workloads get secrets and identity Identity and secrets
How to prove what a run did Receipts
Which isolation class to pick Isolation