LanternDOCS

Agent Runtime

Every Lantern run executes in one of two tiers — shared (inline executor inside the control-plane) or microVM (W12 Kubernetes/Firecracker/Kata stack) — declared in the agent version's manifest.isolation at publish time, not overridable by the caller.

In plain termsThe runtime is the engine room — the part of Lantern that actually executes your agents. The one decision you make is trust: agents running code you wrote go in the fast shared lane, and agents running code you don't fully trust (user uploads, arbitrary shell commands, packages from the internet) go in the microVM lane, where each run gets its own tiny disposable virtual machine that can't touch anything else. Everything downstream — crash recovery, live streaming, receipts — works the same in both lanes.

The runtime, end to end

A run arrives from any entry point, passes the gates, executes in its tier, checkpoints to the journal, streams live, and ends with proof. Every cell is clickable.

Entry points · every one becomes the same kind of run
Fail-closed by designwhen a guarantee can't be met, the run fails loudly — never downgrades silently
  • Never a bare pod
  • Deny-default egress
  • Secrets vended, never baked in
  • VM reports bound to their own run
  • microvm_unavailable, not a quiet fallback
The life of your run read top to bottom: gated → executed → tracked → proven 1 · THE GATES — before anything runs Your request POST /v1/runs Who are you? auth · everything tenant-fenced Can you afford it? over budget → HTTP 402, no spend How sandboxed? the manifest decides: shared or microvm — caller can't override 2 · EXECUTION — one of two tiers SHARED TIER — your own code, fast first token in ~50–200 ms · tokens stream live • workflow graph or plain LLM loop — every step journaled • risky steps can wait for a human — parked at zero compute • crash-safe: resumes at the last completed step MICROVM TIER — untrusted code, walled in its own virtual machine per workload • separate kernel · internet access only by allowlist • tools run inside the VM: shell_exec · http_fetch • tier down? the run fails loudly — never quietly downgraded 3 · STATUS — where your run is right now queued running succeeded failed waiting cancelled a crash re-drives within 30 s — finished work is never re-billed 4 · YOUR EVIDENCE — what you hold afterward journal_events — every step on the record, streamed live Ed25519 receipt — signed proof anyone can verify, no account needed OTel traces — tenant · run · step on every span Shared tier MicroVM tier Durable execution — crash-safe resume Verifiable receipts
The same story in full detail: POST /v1/runs through auth, budget, and isolation gates to the tier that executes, the status machine, and a signed receipt — every box is real code.

Two tiers, one journal

The journal is the runtime's flight recorder: every step of every run is written down before it executes, in one place, regardless of which tier ran it. That one habit is what makes crash recovery, the dashboard waterfall, and signed receipts possible.

Concretely: both tiers write to the same journal_events table. The run waterfall, Ed25519 receipts, and crash-replay are tier-agnostic — there is no second event store.

Shared tierMicroVM tier
Declared as"shared" (default, or absent)"microvm"
ExecutorGoroutine inside control-plane (executeRunInlineSync)Scheduler → manager → Firecracker / Kata / K8s Job
IsolationSame OS process — trust first-party codeSeparate kernel (gVisor) or hypervisor (Kata)
Latency to first token~50–200 ms~150 ms warm / ~1.5 s cold-boot
EgressUnrestricted (trusted code)Harness allowlist; deny-default; iptables REDIRECT required in prod
Crash resume30 s recovery sweep + CompletedStep journal replayVM lifecycle; recovery sweep re-schedules (≤ 3 attempts)
Secret deliveryResolved inline at step time, never loggedShort-TTL JWT over vsock; args stripped from audit
Use caseLoop agents, bridge replies, dashboard runs, trusted workflowsUser-supplied code, exec tools, untrusted packages
Downgrade safetyN/ANever falls back to shared — failure is explicit (microvm_unavailable)

How routing works

When POST /v1/runs arrives, the control-plane reads manifest.isolation from the resolved agent version and dispatches to either executeRunInline (shared) or scheduleAgentSpec (microVM). The caller supplies only the input; the tier comes from the manifest.

Unknown values in manifest.isolation are rejected at agent-version publish time with HTTP 400 — a typo fails at deploy, not at run time.

No silent downgrade. If the microVM tier is unavailable (scheduler unreachable, quota exceeded, manager down), the run fails with code microvm_unavailable. A VM that exits unexpectedly produces microvm_exit; exhausting the 3-attempt resume limit produces microvm_resume_exhausted. None of these ever fall back to the shared tier — the isolation declaration is a security boundary.

Shared tier

The shared tier is a goroutine inside the control-plane. Every live Lantern run today executes here: loop agents, bridge replies, dashboard runs, cron-triggered runs, and sessions. It drives either the plain-LLM tool-use loop (for agents with no workflow JSONB) or the workflow interpreter (for agents with a graph saved in the visual editor). Crash-resume is handled by the recovery sweep — see Durable execution.

Entry points: POST /v1/runs, POST /v1/sessions/{id}/messages, cron scheduler, loop agent tick, bridge-triggered run.

MicroVM tier

The microVM tier is required for agents that run user-supplied code, exec arbitrary tools, or load untrusted packages. Declare it in the manifest:

manifest:
  isolation: microvm          # routes this agent version to the W12 stack
  image_digest: …@sha256:…
  limits: { vcpu: "250m", memory: "128Mi", timeout: "60s" }
  egress_rules: [{ host: "api.openai.com" }]
  idempotent: true

The in-guest tool runner (shipped 2026-07-23) gives the harness a typed tool registry — shell_exec and http_fetch — so workflow-graph agents can route to the microVM tier as a real step executor.

Service-health sweep

A background loop TCP-probes peer services every 60 s (LANTERN_HEALTH_SWEEP_INTERVAL). After 3 consecutive failures it declares the peer DOWN and texts the owner's self-chat — once on transition, no storms. Read the current snapshot:

GET /v1/system/health    # JWT-authed
{
  "services": [
    {
      "name": "runtime-manager",
      "addr": "localhost:50054",
      "up": false,
      "consecutiveFailures": 5,
      "lastChecked": "2026-07-23T10:00:00Z"
    }
  ]
}

Operating it in production

In plain termsFour things an operator cares about, and where each lives: work is never lost (checkpointing), crashed work restarts itself (recovery), a runaway tenant can't take the platform down (throttling), and you can see all of it (monitoring). Everything below is shipped and test-gated — the knobs are environment variables, the views are REST, CLI, and the dashboard.
CapabilityMechanismKnobs / views
CheckpointingEvery step journaled to journal_events before it runs (both tiers) · Firecracker VM snapshots persisted to S3 (ADR 0007)Durable execution
Auto-restartRecovery sweep steals expired run leases and re-drives from the last completed step · microVM runs re-scheduled with LANTERN_RESUME=1, ≤3 attempts · scheduler is HA via leader electionLANTERN_RECOVERY_INTERVAL (default 30 s)
ThrottlingBudget gate blocks over-budget runs with 402 · per-tenant concurrent-VM hard cap (gRPC ResourceExhausted) · spawn-storm guard returns 429 · per-step timeoutLANTERN_SPAWN_RATE_PER_MIN (default 120) · LANTERN_SPAWN_BURST · Budgets
Job monitoringVM list / detail / audit trail / live SSE logs / per-VM metrics / cluster + quota viewsGET /v1/runtime/vms · /metrics · /cluster · /audit · lantern vm list|get|logs|stop|exec · dashboard /runtime
TelemetryFive lantern.run.* OTel metrics · scheduler Prometheus scrape · alert rules + Grafana dashboards + 8 runbooks in infra/monitoring/scheduler :8085/metrics · Observability
TraceabilityW3C trace context propagated control-plane → scheduler → manager → in-VM harness · one correlated identity chain (tenant · run · step · instance) · Ed25519 receiptsReceipts · Observability

System architecture

The technical view — how the control-plane, scheduler, runtime-manager, and in-VM harness collaborate across both tiers on the same journal_events substrate.

Agent Runtime two tiers, one journal · every run routed by its manifest Go Rust 1 · A RUN IS CREATED Entry points SDK · CLI · dashboard · channels · schedules · A2A control-plane GOPOST /v1/runs · :8080 auth → budget gate (402) → reads the agent manifest ISOLATION GATE manifest.isolation = "shared" | "microvm" "shared" · default · trusted first-party "microvm" · untrusted / exec workloads 2a · SHARED TIER — trusted code, fast in-process goroutine · sub-ms dispatch Workflow interpreter nine node types — from ai-step to human approval per-step retry · confidence gate can divert a risky step to a human step_retrying + confidence_evaluated journaled — visible in the run waterfall LLM tool-use loop capability-addressed models · web search built in idempotency keys · token deltas stream live Guarantees crash-resume · exactly-once side effects · budget-metered 2b · MICROVM TIER — untrusted code, walled in hardware isolation per workload · session-scoped VM reuse · invariant #5 runtime-scheduler GO :50055 · HA leader election placement: warm-pool · region · fair-share · cost · health runtime-manager RUST :50054 · per node Firecracker · Kata · K8s Job · Wasm fail-closed isolation gate harness RUST · PID 1 inside the VM in-guest tool runner: shell_exec · http_fetch — via the egress allowlist kernel-attested secret vends · deny-default egress · audit stream step events + vm_exit reported up → journal · crash re-drive (≤3) Honest failure tier unavailable → run fails microvm_unavailable — NEVER silently downgraded to shared step events report path · vm_exit 3 · ONE JOURNAL UNDER BOTH — the flight recorder journal_events — every step written down before it runs crash-resume: 30 s recovery sweep re-drives · replay skips finished steps · side-effect dedup powers the run waterfall, live streams, signed receipts, and rehearsals Observability — invariant #9 one attribute contract on every span · five run metrics · peer-health alerts Isolation is a property of the agent version, not the caller.
The isolation gate reads manifest.isolation from the resolved agent version. Both tiers checkpoint to the same journal_events substrate.

What makes it different

In this section

  • Headless agent quickstart — write your first agent.yaml and run it end-to-end in ~15 minutes
  • Isolation classes — the decision tree from trusted to hostile, and the fail-closed gate
  • Durable execution — exactly-once under crash: journal, replay, per-step retry, idempotency keys
  • Token streamingmessage_delta / message_completed / message_error contract and the SDK async iterator
  • Sessions & memory — interactive multi-turn sessions on the shared tier, and LLM-distilled long-term memory
  • Observability — OTel span attributes, the five lantern.run.* metrics, service-health sweep
  • Identity & secrets — per-instance Ed25519 keys and short-TTL secret vending (microVM tier)
  • Verifiable receipts — signed, offline-verifiable proof of what ran
Interactive agents use sessions, not runs. For multi-turn conversations see Agents; for the full REST surface see the API reference. Sessions execute on the shared tier.