LanternDocs

Agent runtime

Every Lantern run executes in one of two tiers — shared (inline executor inside the control-plane) or microVM (W12 Kubernetes/Firecracker/Kata stack) — declared in the agent version's manifest.isolation at publish time, not overridable by the caller.

In plain termsThe runtime is the engine room — the part of Lantern that actually executes your agents. The one decision you make is trust: agents running code you wrote go in the fast shared lane, and agents running code you don't fully trust (user uploads, arbitrary shell commands, packages from the internet) go in the microVM lane, where each run gets its own tiny disposable virtual machine that can't touch anything else. Everything downstream — crash recovery, live streaming, receipts — works the same in both lanes.

The runtime, end to end

A run arrives from any entry point, passes the gates, executes in its tier, checkpoints to the journal, streams live, and ends with proof. Every cell is clickable.

Entry points · every one becomes the same kind of run
Fail-closed by designwhen a guarantee can't be met, the run fails loudly — never downgrades silently
  • Never a bare pod
  • Deny-default egress
  • Secrets vended, never baked in
  • VM reports bound to their own run
  • microvm_unavailable, not a quiet fallback
The life of your run Read top to bottom. Gates: the request, authentication with tenant fencing, the budget check that returns 402 when over budget, and the manifest isolation setting of shared or microvm. Execution runs in the shared tier for your own code or inside a microVM for untrusted code, and a tier failure fails loudly. Status moves queued, running, succeeded, with failed, cancelled and waiting also possible. Evidence is the journal, an Ed25519 receipt and OTel traces. The life of your run Read top to bottom: gated, executed, tracked, proven. 1 The gates, before anything runs Your request POST /v1/runs or channel, cron, A2A Who are you? Auth: JWT or API key everything tenant-fenced Can you afford it? Over budget: HTTP 402 no spend, no run How sandboxed? Manifest isolation decides: shared or microvm (ADR 0022) 2 Execution: one of two tiers shared microvm Shared tier Shared tier: your own code Inline executor; tokens stream live Workflow graph or plain LLM loop, every step journaled Risky steps can wait for a human, parked at zero compute Crash-safe: resumes at the last completed step MicroVM tier microVM tier: untrusted code Its own virtual machine per workload Separate kernel · egress only by allowlist Tools run inside: shell_exec, http_fetch Tier down? The run fails loudly, never downgraded 3 Status: where your run is right now queued running succeeded Also: waiting (human approval), failed, cancelled Durable execution: crash-safe resume Crash? The recovery sweep (default every 30 s) re-drives the run; finished steps are never re-billed. 4 Your evidence, what you hold afterward journal_events every step on the record, streamed live Verifiable receipts Ed25519 receipt signed proof, verifiable with no account OTel traces tenant, run and step on every span Control plane Untrusted guest microVM wall
The same story in full detail: POST /v1/runs through auth, budget, and isolation gates to the tier that executes, the status machine, and a signed receipt — every box is real code.

Two tiers, one journal

The journal is the runtime's flight recorder: every step of every run is written down before it executes, in one place, regardless of which tier ran it. That one habit is what makes crash recovery, the dashboard waterfall, and signed receipts possible.

Concretely: both tiers write to the same journal_events table. The run waterfall, Ed25519 receipts, and crash-replay are tier-agnostic — there is no second event store.

Shared tierMicroVM tier
Declared as"shared" (default, or absent)"microvm"
ExecutorGoroutine inside control-plane (executeRunInlineSync)Scheduler → manager → Firecracker / Kata / K8s Job
IsolationSame OS process. First-party code onlyFirecracker or Kata microVM, or a gVisor / K8s Job pod (see Isolation classes)
EgressNot sandboxed (first-party code)Per-VM allowlist, deny by default. Default mode: in-guest proxy + iptables REDIRECT (required in prod). Opt-in broker mode (Firecracker only): no NIC, host-side allowlist over vsock
Crash resumeRecovery sweep (default 30 s) + completed-step journal replayRecovery sweep re-schedules the VM; gives up after 3 attempts
Secret deliveryResolved inline at step time, never loggedDefault: short-TTL values vended to the workload over a peer-authenticated socket. Broker mode: credentials injected on the host, never enter the VM
Use caseLoop agents, bridge replies, dashboard runs, trusted workflowsUser-supplied code, exec tools, untrusted packages
Downgrade safetyN/ANever falls back to shared — failure is explicit (microvm_unavailable)

How routing works

When POST /v1/runs arrives, the control-plane reads manifest.isolation from the resolved agent version and dispatches to either executeRunInline (shared) or scheduleAgentSpec (microVM). The caller supplies only the input; the tier comes from the manifest.

Unknown values in manifest.isolation are rejected at agent-version publish time with HTTP 400 — a typo fails at deploy, not at run time.

No silent downgrade. If the microVM tier is unavailable (scheduler unreachable, quota exceeded, manager down), the run fails with code microvm_unavailable. A VM that exits unexpectedly produces microvm_exit; exhausting the 3-attempt resume limit produces microvm_resume_exhausted. None of these ever fall back to the shared tier — the isolation declaration is a security boundary.

Shared tier

The shared tier is a goroutine inside the control-plane. Most runs execute here: loop agents, bridge replies, dashboard runs, cron-triggered runs, and sessions. It drives either the plain-LLM tool-use loop (for agents with no workflow JSONB) or the workflow interpreter (for agents with a graph saved in the visual editor). Crash-resume is handled by the recovery sweep — see Durable execution.

Entry points: POST /v1/runs, POST /v1/sessions/{id}/messages, cron scheduler, loop agent tick, bridge-triggered run.

MicroVM tier

The microVM tier is required for agents that run user-supplied code, exec arbitrary tools, or load untrusted packages. Declare it in the manifest:

manifest:
  isolation: microvm          # routes this agent version to the W12 stack
  image_digest: …@sha256:…
  limits: { vcpu: "250m", memory: "128Mi", timeout: "60s" }
  egress_rules: [{ host: "api.openai.com" }]
  idempotent: true

The in-guest tool runner gives the harness a typed tool registry:shell_exec, http_fetch, read_file, write_file and list_dir. File tools are confined to /workspace. Workflow-graph agents reach it through the engine's tool_call step.

A tool call into a microVMThe workflow engine never runs tools itself: it asks the runtime-manager, which hands a JSON envelope to the harness inside the guest.

1. ExecTool

tool_name, args (Struct), run_id, step_id, vm_id, timeout, idempotency_key run_id:step_id:attempt, tenant_id.

engine/step_executor.go:405engine/step_executor.go:417v1/runtime.proto:469

Service-health sweep

A background loop TCP-probes peer services every 60 s (LANTERN_HEALTH_SWEEP_INTERVAL). After 3 consecutive failures it declares the peer DOWN and texts the owner's self-chat — once on transition, no storms. Read the current snapshot:

GET /v1/system/health    # JWT-authed
{
  "services": [
    {
      "name": "runtime-manager",
      "addr": "localhost:50054",
      "up": false,
      "consecutiveFailures": 5,
      "lastChecked": "2026-07-23T10:00:00Z"
    }
  ]
}

Operating it in production

In plain termsFour things an operator cares about, and where each lives: work is never lost (checkpointing), crashed work restarts itself (recovery), a runaway tenant can't take the platform down (throttling), and you can see all of it (monitoring). The knobs are environment variables, the views are REST, CLI, and the dashboard.
CapabilityMechanismKnobs / views
CheckpointingEvery step journaled to journal_events before it runs (both tiers) · Firecracker VM snapshots persisted to S3 (ADR 0007)Durable execution
Auto-restartRecovery sweep steals expired run leases and re-drives from the last completed step · microVM runs re-scheduled with LANTERN_RESUME=1, ≤3 attempts · scheduler is HA via leader electionLANTERN_RECOVERY_INTERVAL (default 30 s)
ThrottlingBudget gate blocks over-budget runs with 402 · per-tenant concurrent-VM hard cap (gRPC ResourceExhausted) · spawn-storm guard returns 429 · per-step timeoutLANTERN_SPAWN_RATE_PER_MIN (default 120) · LANTERN_SPAWN_BURST · Budgets
Job monitoringVM list / detail / audit trail / live SSE logs / per-VM metrics / cluster + quota viewsGET /v1/runtime/vms · /v1/runtime/metrics · /v1/runtime/cluster · /v1/runtime/audit · lantern vm list|get|logs|stop|exec · dashboard /runtime
TelemetryFive lantern.run.* OTel metrics · scheduler Prometheus scrape · Prometheus alert rules and Grafana dashboards in infra/monitoring/, runbooks in docs/runbooks/scheduler :8085/metrics · Observability
TraceabilityW3C trace context propagated control-plane → scheduler → manager → in-VM harness · one correlated identity chain (tenant · run · step · instance) · Ed25519 receiptsReceipts · Observability

System architecture

The technical view — how the control-plane, scheduler, runtime-manager, and in-VM harness collaborate across both tiers on the same journal_events substrate.

The manifest, not the caller, picks the tier, and one journal sits under both Routing of a run across two tiers. A run is created from the SDK, CLI, dashboard, schedules or A2A, passes the control-plane auth and budget gate, and is routed by the agent manifest field isolation, set on the agent version and not by the caller. The shared tier, the default, runs in the control plane: a workflow interpreter with nine node types, per-step retry and an optional confidence gate that can divert a step to a human, and an LLM tool-use loop with capability-addressed models, built-in web search and idempotency keys. The microVM tier, for untrusted code, goes through the runtime-scheduler for placement and the runtime-manager for the isolation gate; inside the sandbox wall the harness runs as PID 1 beside an in-guest tool runner offering shell_exec, http_fetch, read_file, write_file and list_dir. If the microVM tier is unavailable the run fails with microvm_unavailable and is never downgraded to the shared tier. Both tiers write to one journal_events log, which powers recovery, the run waterfall, live streams and signed receipts. The manifest, not the caller, picks the tier, and one journal sits under both Isolation is a property of the agent version: shared for trusted first-party code, microvm for untrusted or exec workloads. Run created SDK, CLI, dashboard, schedules, A2A control-plane POST /v1/runs · :8080 auth, then budget gate (402) Route by manifest isolation: "shared" | "microvm" set on the agent version shared (default) microvm Shared tier: trusted code, runs in the control plane Workflow interpreter nine node types per-step retry gate can divert to a human LLM tool-use loop capability-addressed web_search built in idempotency keys Guarantees resumes at the last completed step after a crash side effects dedup on idempotency keys spend is budget-metered MicroVM tier: untrusted code, walled in runtime-scheduler placement on a node runtime-manager isolation gate Sandbox Harness (PID 1) egress allowlist secret vends, audit In-guest tool runner shell_exec, http_fetch, read_file, write_file, list_dir Tier unavailable: the run fails code: microvm_unavailable, never downgraded journal_events: one event log under both tiers every step is written before it runs; microVM steps arrive through the harness, manager and control-plane report path 30 s recovery sweep re-drives crashed runs; powers the run waterfall, live streams, signed receipts Control plane Trusted Lantern infrastructure Untrusted guest Sandbox wall Refused
The isolation gate reads manifest.isolation from the resolved agent version. Both tiers checkpoint to the same journal_events substrate.

What makes it different

In this section

  • Headless agent quickstart — write your first agent.yaml and run it end-to-end in ~15 minutes
  • Isolation classes — the decision tree from trusted to hostile, and the fail-closed gate
  • Durable execution — crash-resume: journal, replay, per-step retry, idempotency keys
  • Token streaming — message_delta / message_completed / message_error contract and the SDK async iterator
  • Sessions & memory — interactive multi-turn sessions on the shared tier, and LLM-distilled long-term memory
  • Sandbox lifecycle — pause / resume / fork a microVM, streaming exec, with TS / Python / CLI examples
  • Observability — OTel span attributes, the five lantern.run.* metrics, service-health sweep
  • Identity & secrets — per-instance Ed25519 keys and short-TTL secret vending (microVM tier)
  • Verifiable receipts — signed, offline-verifiable proof of what ran
Interactive agents use sessions, not runs. For multi-turn conversations see Agents; for the full REST surface see the API reference. Sessions execute on the shared tier.