Observability

One OTel trace per spawn, GenAI token telemetry, real-time anomaly detection — wired through standard OpenTelemetry.

One trace per spawn

Every run opens a single OTel trace. Spans from every entry point use the same attribute keys, defined in internal/middleware/span.go and stamped by a single EnrichSpan helper so they never drift between HTTP and gRPC code paths:

lantern.tenant_id   # on every span, both tiers
lantern.user_id
lantern.run_id
lantern.step_id     # per-step spans in the inline executor

On the microVM tier the manager and harness additionally stamp vm_id, isolation_class, and agent_instance_id (the per-spawn Ed25519 identity — see Identity & secrets). A durable resume after a crash re-joins the same trace_id — the full lifecycle is one coherent timeline.

Enabling OTel

Export is env-gated. Set the endpoint and traces flow; leave it unset and tracing is a no-op (zero overhead):

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
# or
LANTERN_OTEL_ENABLED=1   # uses default localhost endpoint

GenAI semantic conventions

LLM steps are annotated with OTel GenAI semantic-convention attributes — including reasoning tokens and cache tokens, not just plain input/output counts. Per-step cost attribution and model-usage breakdowns work out of the box with any OTel-compatible backend.

OTel span attribute contract

Every span emitted anywhere in the stack uses these keys, set via internal/middleware.EnrichSpan. They are no-ops when telemetry is disabled.

The five lantern.run.* metrics

The inline executor emits five OTel metric instruments via the lantern.runtime meter (internal/middleware/metrics.go). They are no-ops when the global MeterProvider is unset (the default no-op provider is safe to import with zero overhead).

Real-time anomaly detection

The runtime watches the live event stream for pathological shapes — a tool-call loop, a step retrying without progress — and surfaces them in real time. This is the early-warning layer for runaway runs.

Peer-service health sweep

A background loop in the control-plane TCP-probes its peer services every 60 s (LANTERN_HEALTH_SWEEP_INTERVAL; set "0" or "off" to disable). After 3 consecutive failures it declares a peer DOWN and sends one self-chat alert. It sends one more when the peer recovers. No alert storms — only state transitions fire notifications.

Services probed when their address env var is set:

• model-router — LANTERN_MODEL_ROUTER_ADDR (only when LANTERN_USE_MODEL_ROUTER=1 or addr is non-default)
• runtime-scheduler — LANTERN_SCHEDULER_GRPC_ADDR
• runtime-manager — LANTERN_DEFAULT_MANAGER_ADDR
• workflow-engine — LANTERN_WORKFLOW_ENGINE_ADDR (optional)

Read the current snapshot:

GET /v1/system/health    # JWT-authed

{
  "services": [
    {
      "name": "runtime-manager",
      "addr": "localhost:50054",
      "up": false,
      "consecutiveFailures": 5,
      "lastChecked": "2026-07-23T10:00:00Z",
      "lastTransition": "2026-07-23T09:57:00Z"
    }
  ]
}

Metrics endpoint

Per-VM live stats for the caller's tenant:

GET /v1/runtime/metrics

Returns a vmMetricsDTO array with vmId, state, node, az, isolationClass, promMetrics (raw Prometheus text from the harness), and timestamps. Per-instance detail: GET /v1/runtime/vms/{id}. Live log stream: GET /v1/runtime/vms/{id}/logs (SSE). The dashboard runtime page renders all three.

Gateway and model-router traces

The gateway emits one span per HTTP request (gateway.request, tagged with tenant_id) via OTLP/HTTP. The model-router emits one span per routing call tagged with tenant_id, run_id, step_id, model_used, tokens_in/out, cost_usd, and escalated via OTLP/gRPC. Both honour inbound W3C traceparent, so spans join the caller's distributed trace automatically.

Prometheus alerts, dashboards, runbooks

Production monitoring artifacts live in infra/monitoring/:

Eight operator runbooks cover every active alert plus the DB restore procedure, linked from each alert's runbook: annotation in alerts.yml. Grafana dashboards: grafana/platform-overview.json and grafana/data-plane-runtime.json.