Observability

Every run produces one OpenTelemetry trace: a timeline of each step with its timing, model, tokens and cost, tagged with tenant, run and step ids. It works with any OTel backend (Grafana, Datadog, Jaeger) and is a no-op when switched off. Spans, metrics, journal rows and receipts share the same identifiers, so you answer “what is this agent doing now” and “what did it do last Tuesday” with one run id.

Trace propagationOne trace per spawn: both entry points stamp the same attributes, and the trace context follows the run into the microVM.

An HTTP requestA gRPC callInto the microVMExporter unset

Swipe sideways to pan.FitShow steps

⏮◀▶ Play▶

Scrub through steps1 / 5

1. The gateway opens the first span

gateway.request is tagged with tenant_id and joins any inbound W3C traceparent.

• PAL Runtime services
• Trusted host code
• Inside the VM
• Collector

Turn it on

Export is gated by environment. Set an endpoint and traces flow; leave both unset and tracing is a no-op with no overhead.

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318   # control plane and gateway export OTLP/HTTP
# or
PAL_OTEL_ENABLED=1   # uses the default localhost endpoint

The model router and runtime manager export OTLP/gRPC, so point them at the collector’s gRPC port (4317). Service names are pal.control-plane, pal-gateway, pal-model-router and pal-runtime-manager.

W3C traceparent is always honoured, even with no exporter: inbound trace context is forwarded correctly either way.

One trace per run

Spans from every entry point carry the same attribute keys, stamped by one helper (EnrichSpan in services/control-plane/internal/middleware/span.go) so HTTP, gRPC and step spans never drift. traceparent propagates at every boundary:

Hop	Span	Attributes
Gateway	gateway.request	tenant_id
Control plane	run dispatch	pal.run_id
Model router	route	step_id, model_used, tokens_in, tokens_out, cost_usd
Runtime scheduler	schedule	tenant_id, vm_id
Runtime manager to harness	trace context propagated	traceparent

On the microVM tier the per-VM isolation class is recorded as a container or pod label, and the trace context reaches the harness (the per-spawn identity is in Identity and secrets). A durable resume starts a new trace tagged with the same run id.

LLM steps carry the OTel GenAI semantic-convention attributes, including reasoning tokens and cache tokens, not only input and output counts. Per-step cost attribution works in any compatible backend.

OTel span attribute contract

Attribute	Key	Set by
Tenant	pal.tenant_id	HTTP enrichment middleware, gRPC tracing interceptor
User	pal.user_id	Same
Run	pal.run_id	Same, plus the inline executor
Step	pal.step_id	Inline executor, per step
VM id	vm_id	Scheduler spans
Model, cost, tokens	model_used, cost_usd, tokens_in, tokens_out	Model-router completion span

Run metrics

The control plane registers its OTel instruments on the pal.runtime meter (services/control-plane/internal/middleware/metrics.go). They are no-ops when no MeterProvider is set.

Instrument	Type	Attributes	Meaning
pal.run.step.duration	Histogram (ms)	agent_name, node_type, tier, outcome	Wall-clock time of one workflow step, including retry backoff. outcome is ok, failed or retried.
pal.run.step.retries	Counter	agent_name, node_type, tier	Attempts beyond the first.
pal.run.replay.skips	Counter	agent_name	CompletedStep cache hits during crash-resume: one node skipped instead of re-run.
pal.run.budget.blocks	Counter	agent_name	Runs denied by a hard-fail budget (HTTP 402).
pal.run.total	Counter	agent_name, tier, status	Runs dispatched through the inline executor. tier is currently always shared. status is succeeded or failed; workflow runs whose steps failed count as succeeded, a known gap.
pal.budget.check.errors	Counter	budget_check (tenant, agent, run, tool), agent_name	Budget lookups that errored and failed closed.
pal.onboarding.step.total	Counter	onboarding_step, onboarding_event	Steps of the agentic onboarding flow.

Attribute names above are shown without their pal. prefix; on the wire each is pal.<name>. In Prometheus the series are pal_*, which the shipped alert rules query.

Anomaly detection

The runtime watches the live event stream and journals an anomaly_detected event with one kind: runaway_loop, excessive_steps, retry_storm, cost_spike or token_spike. It appears in the run waterfall.

Peer-service health sweep

A background loop in the control plane TCP-probes its peers every 60 s (PAL_HEALTH_SWEEP_INTERVAL, a Go duration; "0" or "off" disables it). After 3 consecutive failures it marks a peer DOWN and sends one alert to the operator’s notification channel, then one more on recovery. Only state transitions notify, so there are no alert storms.

A peer is probed when its address variable is set:

Peer	Address variable
model-router	PAL_MODEL_ROUTER_ADDR (only when PAL_USE_MODEL_ROUTER=1 or the address is non-default)
runtime-scheduler	PAL_SCHEDULER_GRPC_ADDR
runtime-manager	PAL_DEFAULT_MANAGER_ADDR
workflow-engine	PAL_WORKFLOW_ENGINE_ADDR (optional)

Read the current snapshot with GET /v1/system/health (JWT):

{
  "services": [
    {
      "name": "runtime-manager",
      "addr": "localhost:50054",
      "up": false,
      "consecutiveFailures": 5,
      "lastChecked": "2026-07-23T10:00:00Z",
      "lastTransition": "2026-07-23T09:57:00Z"
    }
  ]
}

Runtime endpoints

GET /v1/runtime/metrics returns live per-VM stats for the caller’s tenant as a vmMetricsDTO array: vmId, state, node, az, isolationClass, promMetrics (raw Prometheus text from the harness) and timestamps. Per-VM detail is GET /v1/runtime/vms/{id}; the live log stream is GET /v1/runtime/vms/{id}/logs (SSE). The dashboard /runtime page renders all three.

Gateway and model-router traces

The gateway emits one span per HTTP request (gateway.request, tagged with tenant_id) over OTLP/HTTP. The model router emits one span per routing call tagged with tenant_id, run_id, step_id, model_used, tokens_in, tokens_out, cost_usd and escalated, over OTLP/gRPC. Both join the caller’s distributed trace through inbound traceparent.

The gateway has no Prometheus metrics yet

The model router and runtime manager expose Prometheus metrics only when PAL_METRICS_ADDR is set (ports 9465 and 9466 in ADR 0041). The gateway emits traces only, so its latency SLO lives in your tracing backend. Alert rules that need a missing series are parked in the pal-TODO-needs-instrumentation group in infra/monitoring/prometheus/alerts.yml.

Find a run, its logs and its audit trail

One run is one query in your trace backend:

# Jaeger:    service=pal.control-plane pal.run_id=<run_id>
# Tempo:     { span.pal.run_id = "<run_id>" }
# Honeycomb: WHERE pal.run_id = "<run_id>"

VM logs and the runtime audit trail (spawn, terminate, secret vend, egress deny, 403s) are plain API calls:

pal vm logs <vm-id>                                    # or GET /v1/runtime/vms/{id}/logs (SSE)
curl "$API/v1/runtime/audit?limit=50" -H "Authorization: Bearer $TOKEN"

Logs are stored in runtime_vm_logs with a retention sweep (PAL_RUNTIME_LOG_RETENTION_DAYS, see Environment variables). Audit events are also attached to the trace as span events.

Alerts, dashboards and runbooks

Monitoring artifacts live in infra/monitoring/.

Group	Alerts	Source
pal-scheduler	SchedulerDown, SchedulerNoLeader, SchedulerScheduleErrorRateHigh, SchedulerQuotaRejectionSurge, SchedulerNoRegisteredNodes	runtime-scheduler :8085/metrics
pal-liveness	ControlPlaneDown, ControlPlaneNotReady, GatewayDown, ModelRouterDown	up scrape and blackbox /readyz
pal-postgres	PostgresExporterDown, PostgresConnectionSaturation, DataPlaneHeartbeatStale, CronScheduleOverdue	postgres_exporter with custom queries
pal-instrumented-slos	ControlPlane5xxRateHigh, ControlPlane403RateElevated, RunFailureRateHigh, BudgetDenied402Surge	control plane :9464

Each alert’s runbook: annotation links a file in docs/runbooks/. Grafana dashboards are grafana/platform-overview.json and grafana/data-plane-runtime.json. How to wire all of this is in Operations.

SLOs and error budgets

Ten SLOs over a 28-day window are defined once in the Helm values (monitoring.slos: SLI as bad over total, objective, job, runbook). Recording rules and alerts are generated from that list (ADR 0041).

SLO	Objective
Control-plane API availability / latency (under 1 s)	99.9% / 99%
Run success	98%
Scheduling availability / latency (under 1 s)	99.5% / 99%
Model-router errors	99.5%
microVM spawn availability	99.5%
microVM cold boot (under 5 s) / snapshot restore (under 2 s)	95% / 99%
Egress-broker errors	99.5%

Alerts burn on two windows at once: page at 14.4x over 1 h and 5 m, or 6x over 6 h and 30 m; ticket at 3x over 1 d and 2 h, or 1x over 3 d and 6 h. Both windows must exceed the threshold, so a burst that already ended does not page. A policy deny at the egress broker counts as good; only platform faults burn that SLO.

monitoring.alerts.enabled renders a PrometheusRule (needs the prometheus-operator CRDs; on in values-prod.yaml) and monitoring.dashboards.enabled renders a Grafana dashboard. Each alert links its docs/runbooks/slo-*.md runbook. The targets were chosen without production history, so review them after the first full window.