Production deployment

PAL Runtime splits into a control plane that decides what should run (dashboard, scheduling, budgets) and a data plane where agent code executes with your API keys and your data. You run the control plane yourself and put a data plane in each cloud account or VPC that should hold the workloads, installed with Helm or a generated Terraform cluster. There is no managed cloud endpoint yet, so today you deploy both planes. For the internals of the split, see Deployment topology.

Design time on a laptop, the same bundle in the cloudOne immutable agent bundle moves from a laptop stack to a data plane in your VPC. Pick local or cloud and step through.

Design time (local)Ship to cloud

Swipe sideways to pan.FitShow steps

⏮◀▶ Play▶

Scrub through steps1 / 4

1. One command

pal dev boots the whole stack; pal doctor reports what is missing.

The control plane never runs user code. Specs, identity and trace context go down; heartbeats, step events and audit come back, over server-authenticated TLS with a bootstrap token and session JWT (no client certificates). Prompts and data stay in your VPC.

Pick a topology

Topology	Control plane	Data plane	Status
Fully managed	Hosted for you	Hosted for you	Not offered yet
Hybrid (recommended)	Your platform account	In each workload’s cloud	Shipped
Fully self-hosted	Your cluster	Your cluster	Shipped; you run Postgres, Redis and Kubernetes

Hybrid, step by step

1. Deploy the pal-data-plane chart into the workload cluster (Helm).
2. Register it with POST /v1/data-planes or Deployments > Register data plane. This mints a one-time 32-byte bootstrap token. Store it immediately.
3. The data plane dials out to the control plane’s :50051, sends the bootstrap token in the Register RPC, and receives a session JWT (1 h TTL, rotated through RefreshToken).
4. The data plane opens a persistent RunStream bidirectional RPC. The control plane pushes run assignments; the agent reports status and completion. Only run metadata crosses the boundary.

Check it worked: the plane’s heartbeat keeps arriving (every 30 s) and the data_planes.last_heartbeat column advances. If it does not, start with the data plane disconnected runbook.

The tunnel RPCs are defined in packages/proto/pal/v1/dataplane.proto and served on the control plane’s :50051 listener.

RPC	Direction	Purpose
Register	agent to control plane	Exchange the bootstrap token for a session JWT
Heartbeat	agent to control plane	Liveness every 30 s; learns drain orders
ReportMetrics	agent to control plane	CPU, memory and active-run pressure every 60 s
RefreshToken	agent to control plane	Rotate the session JWT before the 1 h expiry
RunStream (bidi)	both	Control plane pushes assignments; agent reports accepted and completed

Each tenant may hold up to PAL_DP_MAX_STREAMS_PER_TENANT concurrent RunStream connections (default 10). Excess connections get gRPC RESOURCE_EXHAUSTED.

Helm

First login. A fresh production install has sign-up closed and no users. Set controlPlane.env.PAL_BOOTSTRAP_OWNER_EMAIL on the first helm install, then read the one-time password link from the control-plane log (kubectl -n pal logs deploy/pal-control-plane | grep -A2 "first-owner bootstrap"). See Accounts and teams.

The repo ships three charts under infra/helm/.

Chart	Installs
pal	Everything in one release, with values-dev.yaml, values-staging.yaml and values-prod.yaml
pal-control-plane	The control plane half of a split topology
pal-data-plane	Workflow engine, runtime manager and the agent that tunnels out to your control plane; see its README

The rest of this section documents pal: gateway, control plane, model router, workflow engine, runtime manager, runtime scheduler, surface gateway and dashboard, with in-cluster Postgres, S3-compatible storage and a Redis subchart for development. The chart defaults PAL_ENV=production, so the production startup guards are on.

The install, step by step, with a values file that renders as written: Deploy to your cloud.

Production settings

• Autoscaling and disruption. <component>.autoscaling.enabled adds a CPU-based HPA (needs metrics-server) for the control plane, gateway, model router, workflow engine and dashboard; values-prod.yaml turns it on. Components with more than one replica get a PodDisruptionBudget.
• NetworkPolicy. networkPolicy.enabled=true denies all traffic in the release, then allows only the service-to-service flows the platform needs, DNS, and scrapes of /metrics from networkPolicy.metricsFrom. Only the control plane, model router, workflow engine, runtime scheduler, runtime manager, surface gateway and backup job may leave the cluster; use networkPolicy.extraEgress for managed Postgres or Redis on private IPs. Your CNI must enforce policy (Calico, Cilium).
• Secrets you must supply. secrets.jwtSecret, credentialKey, receiptSecret, callbackSigningSecret, grpcServiceToken, schedulerNodeToken, appDbPassword, and with managed stores postgresPassword and minioPassword (the S3 secret key). Pass them with --set or an overlay you keep out of git.
• Internal mTLS. values-prod.yaml sets mtls.enabled, and the chart issues every service and runtime-manager certificate through cert-manager, which must be installed first. To bring your own PKI, set mtls.certManager.enabled=false and mount the certificates with controlPlane.extraVolumes and extraEnv.
• Ingress. Two hosts: the API host routes /v1, /auth, /.well-known, /mcp and /dl to the control plane, and the app host serves the dashboard. With ingress.enabled, the chart refuses to render without controlPlane.trustedProxies (the ingress controller’s pod CIDR) and, with NetworkPolicy on, networkPolicy.ingressFrom. The gateway is not on the ingress.
• Backups. backup.enabled=true adds a CronJob that writes a gzipped pg_dump to s3://<backup.s3Bucket>/postgres/ and prunes dumps older than backup.retentionDays. Restore procedure and drill are in Operations. Managed Postgres with point-in-time recovery is the recommended production path.
• Scheduler probe. The scheduler’s readiness probe uses /healthz by default because /readyz stays unready until a runtime manager node registers.

In-cluster stores versus managed

PAL needs Postgres with pgvector, an S3-compatible object store and Redis. The chart can run them for you, which suits kind or a laptop. Production should use managed services.

Store	Dev, kind, evaluation (default)	Production (values-prod.yaml)
Postgres	postgresql.enabled=true: one pgvector/pgvector StatefulSet on a PVC; password is secrets.postgresPassword	postgresql.enabled=false and externalDatabase.host (plus port, sslmode). User and database names still come from postgresql.auth. Provision pgvector and the pal_app role yourself.
Object store	minio.enabled=true: one SeaweedFS StatefulSet on a PVC; one identity from minio.auth.rootUser and secrets.minioPassword; a post-install Job creates minio.defaultBuckets	minio.enabled=false and externalS3.endpoint, accessKeyId, bucket, region (must match the bucket’s region); the secret key is secrets.minioPassword
Redis	Bitnami subchart (redis.enabled)	Same subchart for now

With the in-cluster stores off, helm template fails unless externalDatabase.host and externalS3.endpoint are set, so a production install cannot silently fall back to a single-pod database.

Dev stores are not production storage

They are single replicas with no replication, point-in-time recovery or lifecycle rules. Image tags are pinned in values.yaml (postgresql.image, minio.image). Earlier chart versions used Bitnami subcharts for Postgres and MinIO; the new StatefulSets keep the old names but use different selectors, which Kubernetes will not patch in place. To move an existing release, take a dump (docs/runbooks/db-restore.md), uninstall, install, and restore.

Terraform

pal infra install --cloud aws|gcp|azure generates Terraform for a data-plane cluster. It only writes files; it does not apply them. --tenant-id (or PAL_TENANT_ID) and --control-plane-endpoint (an https:// URL) are required. Other flags include --region, --cluster-name, --existing-cluster, --instance-type, --node-count and --output-dir. The modules live in infra/terraform/ and are not published to the Terraform Registry.

pal infra install --cloud aws --region us-east-1 --tenant-id <tenant-id> \
  --control-plane-endpoint https://<your control plane host>   # writes pal-data-plane-aws/
cd pal-data-plane-aws
terraform init
terraform plan     # review before applying
terraform apply

Multi-region (active-passive)

Opt-in and flag-gated (ADR 0046). A standby region runs the same chart read-only (PAL_READ_ONLY=1: HTTP 503 with Retry-After, gRPC UNAVAILABLE) against a cross-region Postgres read replica and a replicated bucket. A fence epoch in the pal_region_fence table (PAL_FENCE_EPOCH, PAL_REGION) stops a stale region from writing runs, locks or journal rows after a failover. Data-plane agents take an ordered list in PAL_CONTROL_PLANE_ENDPOINTS (primary first) and probe for failback every PAL_CONTROL_PLANE_FAILBACK_INTERVAL (default 60 s).

Targets are an RPO of 1 minute for Postgres and 15 minutes for S3 bundles, and an RTO of 30 minutes. Promotion is manual; automated promotion is future work. Follow the region-failover runbook.

Tenant isolation and monitoring

Each tenant gets its own Kubernetes namespace, pal-t-<tenant_id>, for network isolation, per-tenant resource quotas, and separate service accounts and secrets.

Services emit OTel traces to any backend. The control plane also serves unauthenticated Prometheus metrics on its own listener, :9464/metrics (PAL_METRICS_ADDR), so scrape it only from inside the cluster. Alert rules and wiring are in Operations and Observability.

Database migrations

The control plane manages its Postgres schema with golang-migrate; migrations are embedded SQL in the binary. Only the control plane runs them. Other services read and write tables but never change the schema.

• Fresh database. Migration 0001 creates the full schema on first boot.
• Existing database. 0001 is fully IF NOT EXISTS; running it records version 1 in schema_migrations with no DDL changes.
• New changes. Sequential pairs (0002_*.up.sql, 0002_*.down.sql). Down migrations are required after the baseline.

Next: Upgrading.