Agent Runtime
Every Lantern run executes in one of two tiers — shared (inline executor inside the control-plane) or microVM (W12 Kubernetes/Firecracker/Kata stack) — declared in the agent version's manifest.isolation at publish time, not overridable by the caller.
The runtime, end to end
A run arrives from any entry point, passes the gates, executes in its tier, checkpoints to the journal, streams live, and ends with proof. Every cell is clickable.
- Never a bare pod
- Deny-default egress
- Secrets vended, never baked in
- VM reports bound to their own run
- microvm_unavailable, not a quiet fallback
Two tiers, one journal
The journal is the runtime's flight recorder: every step of every run is written down before it executes, in one place, regardless of which tier ran it. That one habit is what makes crash recovery, the dashboard waterfall, and signed receipts possible.
Concretely: both tiers write to the same journal_events table. The run waterfall, Ed25519 receipts, and crash-replay are tier-agnostic — there is no second event store.
| Shared tier | MicroVM tier | |
|---|---|---|
| Declared as | "shared" (default, or absent) | "microvm" |
| Executor | Goroutine inside control-plane (executeRunInlineSync) | Scheduler → manager → Firecracker / Kata / K8s Job |
| Isolation | Same OS process — trust first-party code | Separate kernel (gVisor) or hypervisor (Kata) |
| Latency to first token | ~50–200 ms | ~150 ms warm / ~1.5 s cold-boot |
| Egress | Unrestricted (trusted code) | Harness allowlist; deny-default; iptables REDIRECT required in prod |
| Crash resume | 30 s recovery sweep + CompletedStep journal replay | VM lifecycle; recovery sweep re-schedules (≤ 3 attempts) |
| Secret delivery | Resolved inline at step time, never logged | Short-TTL JWT over vsock; args stripped from audit |
| Use case | Loop agents, bridge replies, dashboard runs, trusted workflows | User-supplied code, exec tools, untrusted packages |
| Downgrade safety | N/A | Never falls back to shared — failure is explicit (microvm_unavailable) |
How routing works
When POST /v1/runs arrives, the control-plane reads manifest.isolation from the resolved agent version and dispatches to either executeRunInline (shared) or scheduleAgentSpec (microVM). The caller supplies only the input; the tier comes from the manifest.
Unknown values in manifest.isolation are rejected at agent-version publish time with HTTP 400 — a typo fails at deploy, not at run time.
microvm_unavailable. A VM that exits unexpectedly produces microvm_exit; exhausting the 3-attempt resume limit produces microvm_resume_exhausted. None of these ever fall back to the shared tier — the isolation declaration is a security boundary.Shared tier
The shared tier is a goroutine inside the control-plane. Every live Lantern run today executes here: loop agents, bridge replies, dashboard runs, cron-triggered runs, and sessions. It drives either the plain-LLM tool-use loop (for agents with no workflow JSONB) or the workflow interpreter (for agents with a graph saved in the visual editor). Crash-resume is handled by the recovery sweep — see Durable execution.
Entry points: POST /v1/runs, POST /v1/sessions/{id}/messages, cron scheduler, loop agent tick, bridge-triggered run.
MicroVM tier
The microVM tier is required for agents that run user-supplied code, exec arbitrary tools, or load untrusted packages. Declare it in the manifest:
manifest:
isolation: microvm # routes this agent version to the W12 stack
image_digest: …@sha256:…
limits: { vcpu: "250m", memory: "128Mi", timeout: "60s" }
egress_rules: [{ host: "api.openai.com" }]
idempotent: trueThe in-guest tool runner (shipped 2026-07-23) gives the harness a typed tool registry — shell_exec and http_fetch — so workflow-graph agents can route to the microVM tier as a real step executor.
Service-health sweep
A background loop TCP-probes peer services every 60 s (LANTERN_HEALTH_SWEEP_INTERVAL). After 3 consecutive failures it declares the peer DOWN and texts the owner's self-chat — once on transition, no storms. Read the current snapshot:
GET /v1/system/health # JWT-authed{
"services": [
{
"name": "runtime-manager",
"addr": "localhost:50054",
"up": false,
"consecutiveFailures": 5,
"lastChecked": "2026-07-23T10:00:00Z"
}
]
}Operating it in production
| Capability | Mechanism | Knobs / views |
|---|---|---|
| Checkpointing | Every step journaled to journal_events before it runs (both tiers) · Firecracker VM snapshots persisted to S3 (ADR 0007) | Durable execution |
| Auto-restart | Recovery sweep steals expired run leases and re-drives from the last completed step · microVM runs re-scheduled with LANTERN_RESUME=1, ≤3 attempts · scheduler is HA via leader election | LANTERN_RECOVERY_INTERVAL (default 30 s) |
| Throttling | Budget gate blocks over-budget runs with 402 · per-tenant concurrent-VM hard cap (gRPC ResourceExhausted) · spawn-storm guard returns 429 · per-step timeout | LANTERN_SPAWN_RATE_PER_MIN (default 120) · LANTERN_SPAWN_BURST · Budgets |
| Job monitoring | VM list / detail / audit trail / live SSE logs / per-VM metrics / cluster + quota views | GET /v1/runtime/vms · /metrics · /cluster · /audit · lantern vm list|get|logs|stop|exec · dashboard /runtime |
| Telemetry | Five lantern.run.* OTel metrics · scheduler Prometheus scrape · alert rules + Grafana dashboards + 8 runbooks in infra/monitoring/ | scheduler :8085/metrics · Observability |
| Traceability | W3C trace context propagated control-plane → scheduler → manager → in-VM harness · one correlated identity chain (tenant · run · step · instance) · Ed25519 receipts | Receipts · Observability |
System architecture
The technical view — how the control-plane, scheduler, runtime-manager, and in-VM harness collaborate across both tiers on the same journal_events substrate.
What makes it different
lantern.run.* metrics, and a shared span-attribute contract from HTTP entry to harness exit.In this section
- Headless agent quickstart — write your first
agent.yamland run it end-to-end in ~15 minutes - Isolation classes — the decision tree from
trustedtohostile, and the fail-closed gate - Durable execution — exactly-once under crash: journal, replay, per-step retry, idempotency keys
- Token streaming —
message_delta/message_completed/message_errorcontract and the SDK async iterator - Sessions & memory — interactive multi-turn sessions on the shared tier, and LLM-distilled long-term memory
- Observability — OTel span attributes, the five
lantern.run.*metrics, service-health sweep - Identity & secrets — per-instance Ed25519 keys and short-TTL secret vending (microVM tier)
- Verifiable receipts — signed, offline-verifiable proof of what ran