Identity and secrets

Agents need API keys, and keys are what an attacker wants. A spec holds only a reference to a secret. The value is resolved at the last moment, handed to a process that proves it is the workload, and kept in memory for a short TTL. Each spawn also gets a signed instance identity, so a request to the secret relay can be tied to one VM, one tenant and one run. If even a short-lived value inside the VM is too much, use broker mode, where the credential never enters it.

Identity and secret vendingThe spec holds a reference, never a value. A secret is vended at the last moment to a peer the kernel has authenticated.

Vend a secretDenied peerRelay not configured

Swipe sideways to pan.FitShow steps

⏮◀▶ Play▶

Scrub through steps1 / 5

1. A reference is declared

The spec names the secret; the manager carries only the reference into the spawn.

• Config
• Trusted host code
• Inside the VM
• Control plane

Who proves what

Hop	Credential	Checked by
Workload to the harness secrets socket	Kernel-attested uid and pid (SO_PEERCRED)	The harness, against PAL_WORKLOAD_UID
Harness to the runtime manager	Per-VM client certificate whose identity must match the vm_id	The manager, before it vends
Manager to the control-plane relay	Pre-shared token in X-Pal-Runtime-Token, and optionally the instance token	The control plane
Relay to the stored secret	The tenant on the VM’s own record	The control plane

Instance identity

At schedule time the control plane mints a short-lived signed token for the instance and injects it into the VM’s environment. It is a JWT with typ: agent-instance (so a normal user JWT is rejected), the agent_instance_id, tenant_id and run_id. The token is signed with Ed25519 when PAL_AGENT_IDENTITY_ED25519_SEED is set, and with HS256 otherwise. The default lifetime is one hour (PAL_AGENT_IDENTITY_TTL). The instance id is what observability uses as agent_instance_id, so traces from two runs of one agent do not collide.

With the Ed25519 seed set, the public key is published, with no auth, at:

GET /.well-known/pal-agent-identity

Without the seed the endpoint reports only HS256 and never emits the secret.

Caution

The identity is a token signed by the control plane’s key, not a keypair held inside the VM. Set the Ed25519 seed in any real deployment, or a verifier has no public key to check against.

Secret references

In a spec you reference a secret, you do not embed it. The raw value never appears in the image, logs, traces or run state. From demo 02:

secrets:
  - env_name: USER_AGENT
    secret_uri: pal.secret://__tenant__/key/scraper-user-agent

The relay resolves refs of the form pal.secret/llm/<provider>, pal.secret/connector/<install_id>/<config_key> and pal.secret/connector/<install_id>/oauth. An unknown ref and a ref that belongs to another tenant return the same not found, so the endpoint is not an oracle for what exists.

Vending

The workload does not ship with secrets. It requests them at runtime from the harness’s socket, /run/pal/secrets.sock. The harness authenticates every peer with SO_PEERCRED and vends only to the workload uid the manager injects as PAL_WORKLOAD_UID (and to its own uid). Vended secrets are cached with a 300-second TTL. An unauthorized attempt emits a secret_access_denied audit event. If PAL_WORKLOAD_UID is unset, the harness warns once and allows the request, which is the dev path. In production the manager must inject it, and the harness refuses to start without a workload identity.

The harness asks the manager, and the manager asks the control plane over POST /v1/runtime/secrets/resolve:

• Fail closed. The relay needs PAL_RUNTIME_SECRET_TOKEN on both the control plane and the manager. Unset on the control plane, every call returns 403. The token is hashed with SHA-256 before a constant-time compare.
• Not reachable with a user JWT. The route accepts the service token, not an end-user session.
• Brute-force limit. Authentication failures are rate-limited per remote IP, at 10 failures per minute. Beyond that, the relay answers 429.
• VM binding. The relay checks that the vm_id exists, belongs to the claimed tenant and is not terminated or failed. All three failures return the same 404 body. When an instance token is presented, its agent_instance_id must match that vm_id and tenant, and a bad token gets 403.
• Per-VM certificate. The manager checks that the client certificate’s identity matches the vm_id on a vend, so one VM cannot ask for another’s secrets.
• Audit. Each request writes a secret_resolve audit row with the ref names and count. Values are never logged.
• The manager never holds the decryption key. PAL_CREDENTIAL_KEY lives on the control plane. A compromised manager node cannot decrypt a tenant’s secrets by itself.

Without a manager, the harness can read secrets from its own environment when PAL_ALLOW_SECRET_STUB=1. Never set it in production.

Caution

Vending still puts a short-lived value inside the VM, so a compromised workload can read it while it is valid. Pair secrets with the right isolation class: a workload that loads internet packages and holds a secret should be untrusted, with an egress allowlist.

Broker mode: credentials never enter the VM

In broker mode (PAL_EGRESS_MODE=broker, Firecracker only, opt-in) the VM has no network device. The host broker resolves the secret ref and sets the header on the way out. The workload never holds the value. See Egress.

Software measurement of the launched VM

Each Firecracker VM gets a SHA-256 measurement over its kernel, rootfs, harness, boot arguments and spawn contract (ADR 0031). Secrets and per-VM identity are excluded, so two VMs from the same images and spec match. PAL_MEASUREMENT_GATE on the runtime manager is off, warn (the default) or enforce, which refuses a VM whose measurement matches no reference.

Caution

The measurement is host-asserted: the manager hashes the files it is about to boot and records what it launched. It does not prove what ran inside the VM, and a compromised host could forge it. Hardware attestation (ADR 0023) needs confidential-compute hardware and is unverified here. The value appears as softwareMeasurement in Receipts for microVM runs.

Encryption of stored credentials

Provider keys and connector tokens are AES-256-GCM encrypted at rest with PAL_CREDENTIAL_KEY. The control plane refuses to start in production without it. See Environment variables.