Skip to content
docs

Changelog

This page lists what shipped in PAL Runtime since the 0.2.0 release, with the one breaking change first. Read Breaking before you upgrade, then scan New to see what is in your checkout. The full history is in CHANGELOG.md.

Breaking

Section titled “Breaking”

  • Firecracker guests no longer get caller env on the kernel cmdline. spawn.json is the only channel for caller env; the cmdline carries just the manager-owned PAL_ENV, PAL_MANAGER_ADDR and PAL_WORKLOAD_UID. Harness builds that predate spawn.json are unsupported for caller env. The old path leaked values, such as an instance token, into console logs.

  • Upgrading to chart 0.3.0. Ingress is split into an API host and an app host. The chart refuses to render with ingress on unless controlPlane.trustedProxies (and networkPolicy.ingressFrom, when network policy is on) is set, and in production the control plane refuses to start without PAL_CALLBACK_SIGNING_SECRET. open and invite sign-up need Redis, and OAuth sign-up follows PAL_SIGNUP_MODE. Migrations are forward-only. See Upgrading.

Renamed

Section titled “Renamed”

The product is now PAL, the Personalized Agentic Layer, with two products: PAL for you and PAL Runtime for builders (ADR 0032). Docs, dashboard copy, widget and CLI help say PAL, the CLI binary is pal, and variables are spelled PAL_*. Wire identifiers follow: x-pal-service-token, pal.secret/, pal.v1.

New since 0.2.0

Section titled “New since 0.2.0”
  • Promotion. Freeze an immutable version, promote it through sandbox, staging and prod behind server-side gates, approve prod as a human, canary it and roll it back (pal freeze, promote, approve, rollback).
  • Agent identity and policy. Every agent is a principal with a signed identity and a default-deny policy enforced at each tool, connector, A2A and sandbox dispatch, with a hash-chained audit (pal agents policy).
  • Console. Dashboard v3: Overview, the agent lifecycle tabs, Reliability and SLOs, Incidents, Debug, Promotions and Sandboxes.
  • Observability and debugging. Run trace trees, per-agent SLOs, sampled production evals, anomaly incidents, and debug and support agents whose fixes need your approval.
  • A2A v0.3 and MCP. Spec cards and JSON-RPC tasks, an MCP client with OAuth 2.1, and a PAL MCP server for API keys with the mcp scope.
  • REST API route index. Generated from the code and checked in CI, so the reference lists every route.
  • Batches and run callbacks. POST /v1/batches fans one request out into child runs with a concurrency cap and an Idempotency-Key; runs and batches take a signed callbackUrl, delivered at most once.
  • Self-consistency is the default confidence estimator when the confidence gate is on; PAL_CONFIDENCE_ESTIMATOR=verbalization opts out.
  • Billing (off by default). Stripe plans, checkout, portal and dunning behind PAL_STRIPE_SECRET_KEY; a failed payment falls back to free limits after a grace period and never deletes data.
  • Paired devices. /v1/bridges enrolls a device with a one-time code and issues a device token scoped to signals; /v1/signals is tenant-scoped when called with a credential.
  • Demo mode. make demo or pal demo. Docker only, no API key. Each visitor gets a private workspace with recorded answers, deleted after 30 minutes.
  • Knowledge base and handoff. Upload docs for cited answers, and hand a chat to a human with a Slack ping.
  • Webhook triggers. A signed public URL per agent. GitHub, Sentry, PagerDuty or any HMAC sender starts a run; duplicates are deduplicated.
  • Accounts and sign-up. Gated sign-up, email verification, password reset, invite codes, captcha, data export and workspace deletion.
  • Budgets, enforced. Tokens per day, per-run caps inside workflows, per-tool limits, and fail-closed 503 when a hard-fail budget cannot be read.
  • Capability names. Every capability and optimize target, and the model it resolves to per provider.
  • Connector actions. Slack thread replies, GitHub PR comments, Sentry issues, Calendar events, PagerDuty, and a recorded test mode.
  • Egress DLP. A per-tenant policy the egress broker applies to TLS-terminated and plain-HTTP egress: card numbers, secret-key prefixes, email and phone thresholds. A guard rail and audit trail, not the boundary.
  • Host-asserted measurement (ADR 0031). Every Firecracker cold spawn records a SHA-256 over kernel, rootfs, harness, cmdline and spawn contract; signed receipts carry it as softwareMeasurement. It proves what the host launched, not what ran inside; hardware attestation is not done.
  • Secure by default (ADR 0042). API-key scope enforcement is on in production and every authenticated GET route now carries a scope; PAL_AUTHZ_ENFORCE=0 opts out with a warning. Row-level security is on in production when PAL_APP_DB_PASSWORD is set, and startup is fatal without it unless PAL_RLS_ENFORCE=0.
  • Mutual TLS on internal gRPC (ADR 0043). Every internal gRPC server now requires a client certificate (PAL_GRPC_TLS_CA/_CERT/_KEY, optional SPIFFE allowlist) in addition to the service token, and production refuses to start without it unless PAL_GRPC_INSECURE=1. The customer-VPC data-plane tunnel is the exception: server-auth TLS with a bootstrap token and session JWT.
  • Multi-region, active-passive (ADR 0046). PAL_REGION, PAL_READ_ONLY and PAL_FENCE_EPOCH fence a standby region; data-plane agents fail over across PAL_CONTROL_PLANE_ENDPOINTS in order and fail back on PAL_CONTROL_PLANE_FAILBACK_INTERVAL. See the region-failover runbook in the repo.
  • Examples and test lanes. Daily digest, Telegram assistant, voice receptionist and data analyst, each with a fixture path that needs no provider key; Playwright dashboard specs; a CI lane that runs them on every PR.
  • Telegram and WhatsApp replies. The surface gateway now posts the run’s output back to the chat; before, the answer was dropped.
  • Disk snapshots. Snapshots capture the VM disk; restore and fork stage from it.
  • Real microVMs on a Mac. make lima-up, then make lima-e2e: Firecracker on KVM inside a Lima guest, on Apple Silicon M3+.
  • End-to-end tests. make test-e2e against a live stack, with a spend policy and make e2e-clean.

Fixed

Section titled “Fixed”
  • Receipt verification detects an edited payload. POST /v1/runs/receipts/verify used to compare only the signature string.
  • Fork and restore clone the VM disk copy-on-write (FICLONE) on XFS and btrfs instead of byte-copying it.
  • Workflow connector nodes can call Sentry and PagerDuty; a workflow whose JSON held a backslash no longer silently runs as a plain LLM call; the resumed node after an approval park is journaled.

API keys and first owner

Section titled “API keys and first owner”
  • New API keys start with pal_live_. Existing hlx_live_ keys keep working.
  • API keys record the creating user, accept an optional expiresAt, and can be rotated with POST /v1/api-keys/{id}/rotate.
  • Self-hosts can create the first owner with PAL_BOOTSTRAP_OWNER_EMAIL (see Accounts and teams).