Operations

This is what you do after helm install: scrape the metrics, wire the alerts, keep backups you have actually restored, and upgrade without losing data. The install itself is in Production deployment. Every setting named here is in Environment variables.

Before you start

• Supply the secrets the production guards require (secrets.jwtSecret, credentialKey, receiptSecret, grpcServiceToken and the rest listed under Production settings). The production checklist shows which process needs which.
• values-prod.yaml turns the in-cluster Postgres and object store off. Set externalDatabase.host and externalS3.endpoint, or helm template fails on purpose.
• Set networkPolicy.enabled=true (needs a CNI that enforces policy, such as Calico or Cilium) and list your Prometheus namespace under networkPolicy.metricsFrom. The default allows the monitoring namespace.

Metrics

Two Prometheus endpoints exist.

Endpoint	Port	Notes
Control plane /metrics	9464 (PAL_METRICS_ADDR)	Its own listener, not the public :8080 API. Unauthenticated, and labels carry agent names: scrape it from Prometheus, never through ingress.
Runtime scheduler /metrics	8085

The chart annotates the control-plane pod with prometheus.io/port: 9464, and the NetworkPolicy allows both scrape paths from networkPolicy.metricsFrom.

kubectl -n pal port-forward deploy/pal-control-plane 9464:9464 &
curl -s localhost:9464/metrics | grep -E '^(pal_|http_server_request_duration_seconds_count)' | head

Check it worked: you see pal_run_total (see Observability), pal_run_budget_blocks_total, and the http_server_request_duration_seconds series the 5xx alert uses. Step-level series (duration, retries, replay skips, budget-check errors, onboarding steps) appear as the matching events happen. Deployment and CronJob names are prefixed with your release name; adjust them.

The gateway emits traces only. The model router and runtime manager expose Prometheus metrics only when PAL_METRICS_ADDR is set. Gateway latency lives in your tracing backend; see Observability and SLOs and error budgets.

Alert groups

Rules are in infra/monitoring/prometheus/alerts.yml. Two Grafana dashboards (platform-overview.json, data-plane-runtime.json) are in infra/monitoring/grafana/. Every alert has a runbook: annotation pointing at a file in docs/runbooks.

Group	Alerts	Needs
pal-scheduler	SchedulerDown, SchedulerNoLeader, SchedulerScheduleErrorRateHigh, SchedulerQuotaRejectionSurge, SchedulerNoRegisteredNodes	scrape job pal-runtime-scheduler on :8085
pal-liveness	ControlPlaneDown, ControlPlaneNotReady, GatewayDown, ModelRouterDown	jobs pal-control-plane, pal-gateway, pal-model-router, and a blackbox probe pal-control-plane-readyz on /readyz
pal-postgres	PostgresExporterDown, PostgresConnectionSaturation, DataPlaneHeartbeatStale, CronScheduleOverdue	postgres_exporter with postgres-exporter-queries.yaml
pal-instrumented-slos	ControlPlane5xxRateHigh (above 5% for 5 m), ControlPlane403RateElevated, RunFailureRateHigh (above 20% for 10 m), BudgetDenied402Surge	control-plane job scraping :9464

The job names must match the ones in the rules. Gateway and model-router p99 rules are parked, commented out, in a “needs instrumentation” group. Do not enable a rule before its series exists: a rule over a missing series is silently always green. RunFailureRateHigh counts inline-executor runs only; microVM-tier runs are not counted yet.

Backups and the restore drill

In production, use managed Postgres (RDS, Cloud SQL, Azure Database) with automated backups and point-in-time recovery, with postgresql.enabled=false and externalDatabase.*. The in-cluster Postgres is one pod on a PVC for dev and evaluation. If you self-host it, enable the logical backup as a floor:

# values: backup.enabled=true  (schedule "0 2 * * *", s3Bucket pal-backups, retentionDays 14)
kubectl create job --from=cronjob/pal-postgres-backup backup-manual-1 -n pal   # dump on demand
aws s3 ls s3://pal-backups/postgres/                                        # newest object should be under 24 h old

The CronJob (<release>-postgres-backup) runs pg_dump, gzips it to s3://<bucket>/postgres/pal-<ts>.sql.gz, and prunes dumps older than retentionDays. A failed or empty dump fails the pod before anything is uploaded or pruned. With backup.useChartMinio=true (the default) it uploads to the release’s own object store; for real durability point it at external S3 with backup.existingSecret.

A backup you have never restored is a hope. The drill needs only Docker and touches no live database:

aws s3 cp s3://pal-backups/postgres/<newest>.sql.gz ./latest.sql.gz
scripts/db-restore-drill.sh ./latest.sql.gz     # restores into a throwaway pgvector container
scripts/db-restore-drill.sh --self-test         # exercises the script itself (CI runs this)

The drill restores with ON_ERROR_STOP, so a corrupt gzip fails, and requires the tables in DRILL_EXPECT_TABLES (default agents runs journal_events) to exist. Empty tables pass; set DRILL_REQUIRE_ROWS=1 when you know the source has data. DRILL_PG_IMAGE (default pgvector/pgvector:pg16) must match the server’s major version. Exit 0 means the backup restores; a failure means your backups are not usable until it passes. Run it monthly and after any schema or major-version change.

For a real restore, stop writes (scale the control plane to 0), restore into a new database, re-point the release, and scale back up. A dump contains no roles: recreate pal_app and its password, then run DATABASE_URL=... scripts/check-app-grants.sh before you rely on RLS enforcement. The full procedure for both topologies is db-restore.md.

Upgrading

Version upgrades, the 0.3.0 requirements, rollback and the move off the Bitnami-based Postgres and object store: Upgrading.

Kiosk and demo deployments

make demo (or pal demo) runs the stack in kiosk mode on Docker alone: every visitor gets a private throwaway workspace, a fixture LLM and no sign-up. The tour is in Try it: the demo.

make demo             # then open http://localhost:3001/kiosk
make demo-down        # stop it and delete its data
pal demo --no-open    # same stack from the CLI, without opening a browser
pal demo --down

For a public demo host the guard is strict. PAL_KIOSK=1 refuses to start unless PAL_RLS_ENFORCE=1, PAL_AUTHZ_ENFORCE=1 and a non-default JWT_SECRET are set, and under a production-like PAL_ENV it also needs PAL_KIOSK_ALLOW_PROD=1. Caps on workspace lifetime, concurrent workspaces and new workspaces per IP per hour are the PAL_KIOSK_* variables. Do not enable kiosk mode on a deployment that holds real tenants.

Node filesystem for snapshots and forks

Pause, resume and fork stage each child’s disk from the snapshot. On XFS (formatted with reflink=1) or btrfs the runtime manager clones it copy-on-write (FICLONE), so a fork costs metadata, not a copy of the disk. On ext4 or tmpfs it falls back to a full byte copy and logs a warning with disk_staging="full_copy", the bytes copied and the time taken. Everything still works, but fork cost grows with disk size.

Once your nodes’ snapshot directory (/var/lib/pal/snapshots) and jail root are on XFS or btrfs, set PAL_REQUIRE_REFLINK=1 on the runtime manager to turn the fallback into an error. It is off in every shipped values file.

Runbooks

On-call runbooks live in the repository, each as symptom, what fired it, triage, remediation, escalate.

Runbook	Use it when
control-plane 5xx	ControlPlane5xxRateHigh fires
run failure spike	RunFailureRateHigh fires
budget 402 surge	BudgetDenied402Surge fires
data plane disconnected	DataPlaneHeartbeatStale fires
database saturation	PostgresConnectionSaturation fires
scheduler not firing	CronScheduleOverdue fires
gateway latency	Gateway latency regresses in your tracing backend
enabling RLS enforcement	Turning row-level security enforcement on
staged rollout of GA features	Enabling flag-gated features in stages
region failover	Promoting a standby region

Limits

• The logical backup has no point-in-time recovery: you lose every write since the last dump. It is a floor, not a recovery objective.
• The scheduler’s readiness probe defaults to /healthz because /readyz stays unready until a runtime manager registers.
• HPAs are CPU-based and need metrics-server.
• Connector calls ignore HTTPS_PROXY / HTTP_PROXY. The control plane dials vendor APIs directly, so its SSRF guard can check the real destination IP. If your deployment sends all egress through a proxy, connector calls won’t use it. Allow the control plane direct egress to the vendor hosts, for example with networkPolicy.extraEgress, or connectors will fail.
• Jira must be Atlassian Cloud (*.atlassian.net). Jira Data Center, Jira Server and custom domains are refused for now. The same allowlist applies to Salesforce: the instance URL must be under salesforce.com or force.com.
• Sign-up that is open or invite in production needs a reachable Redis. The control plane refuses to start without one, because a missing sign-up rate limiter fails closed. Behind an ingress, set PAL_TRUSTED_PROXIES to its CIDRs. Without it, every client shares one rate-limit bucket.