LanternDocs

Durable execution

A crash mid-run does not lose work, re-spend tokens, or fire a side effect twice. The shared tier (inline executor) recovers through an event-sourced journal_events table and a CompletedStep replay gate, with external deliveries de-duplicated by idempotency key. MicroVM runs are re-scheduled by the same recovery sweep (see below).

In plain termsImagine an agent that's three steps into a five-step task — it's already sent an email and spent money on model calls — and the server restarts. Without durability you'd either lose the run or start over and send that email twice. Lantern writes each step to the database the moment it finishes, like a checkpoint in a video game. After a crash, the run reloads at the last checkpoint: finished steps are skipped (their saved results are reused), and only the unfinished work runs again.
Durable execution A run checkpoints every step to the journal. If the process crashes mid-step, the recovery sweep (default every 30 seconds) finds runs whose run_locks lease is absent or expired, wins the lease with an UPSERT that only steals expired leases, and re-drives the run: the shared tier re-walks the graph, the microVM tier reschedules the same run id with LANTERN_RESUME=1 up to 3 attempts. Replay skips completed steps through the journal cache and claimSideEffect dedup, re-executes the in-flight step with the same Idempotency-Key, then continues with new work and issues a receipt. A crash costs one step, never the run Completed work is replayed from the journal, and every side effect is deduplicated. 1 The run every step checkpointed to the journal run created POST /v1/runs run_locks lease step 1 done step_completed output in journal row step 2 done connector call dedup key per attempt step 3 in flight LLM + Idempotency-Key step_started, no end process crashes deploy, OOM, node loss the goroutine is gone 2 Recovery nothing is asked of the caller lease expires; the sweep notices on its next tick Recovery sweep, every 30 s Finds running or queued runs whose run_locks lease is absent or expired recovery.go · RunRecoveryLoop Lease steal UPSERT that only wins an expired lease Two replicas can race; exactly one wins acquireRecoveryLock Re-drive Shared tier: interpreter re-walks the graph microVM tier: same run_id, LANTERN_RESUME=1, up to 3 attempts redriveRun · resumeMicroVMRun 3 Replay completed work is never re-executed step 1 skipped CompletedStep cache hit returns the journal row step 2 skipped no second side effect claimSideEffect dedup step 3 re-executes same Idempotency-Key provider dedups step 4 done first new work since the crash run succeeded Ed25519 receipt issued crash in the waterfall The guarantee No double side effects on replay: journal cache, claimSideEffect key, provider Idempotency-Key. Control plane Failure
Crash-resume: the recovery sweep wins a run_locks UPSERT, re-drives the run, and the CompletedStep hook skips nodes that already have a step_completed row.
A run, end to endFrom POST /v1/runs to a journaled result, including the budget gate, the optional model router, and what happens when the process dies mid-step.

1. POST /v1/runs

The SDK sends { agentName, input }. The route requires the runs:execute scope.

server/main.go:430handlers/rest.go:579

Event-sourced journal

Every step transition is appended to journal_events before the next step begins. The journal — not in-memory state — is authoritative. On the shared tier the inline executor writes these events directly. On the microVM tier the control plane accepts step_event reports on /v1/runtime/report and writes them to the same table; the harness does not emit step events yet.

run        step           journal events
────────────────────────────────────────────────
run_1      step_a         step_started → step_completed
run_1      step_b         step_started → step_completed
run_1      step_c         step_started   ◀── crash here
                          (no step_completed written)

Other event kinds you will see in the waterfall: step_retrying (between retry attempts), step_waiting (run parked at an approval node), anomaly_detected (runaway_loop, excessive_steps, retry_storm, cost_spike or token_spike), confidence_evaluated (confidence gate decision, when enabled), and confidence_gate_bypassed (gating is on but no handler is wired — step auto-approves).

CompletedStep replay

The workflow interpreter receives a CompletedStep hook wired to journalCompletedStep. Before invoking any side-effecting node (ai-step, tool, connector, subagent, approval) the interpreter queries journal_events for an existing step_completed row. If one is found, the cached output is returned and the underlying side-effect is never re-invoked.

The plain-LLM path has the equivalent checkCachedLLMStep cache: before calling the model router, the executor checks the journal for a previous successful LLM step for this run. If found, it replays the cached response — the provider is never contacted again.

No re-spent tokens on crash-replay. Completed LLM steps are read back from the journal. The provider sees only the original call, not a retry.

Resume from the last completed step

On re-drive, the interpreter walks the graph from the trigger node again. Nodes that already have a step_completed row are skipped via the CompletedStep hook. Execution continues from the first incomplete node. This is idempotent: a re-drive that re-walks an already-finished graph is a no-op.

Side-effect dedup via idempotency keys

Every external side effect carries a key derived from a one-way hash of the three identifiers, pipe-delimited:

idempotency_key = sha256("runID|stepID|attempt")

LLM provider calls receive an Idempotency-Key HTTP header on every request to OpenAI and Anthropic (derived in internal/handlers/llm_idempotency.govia a one-way hash — the key never carries secret material). A crash then replay to the same provider dedups at the provider instead of double-billing.

External deliveries (email, connector auth pages, browser and cross-app actions) claim a side_effect_receipts row before dispatching. A re-drive that reaches the same delivery finds the receipt and short-circuits. Workflow connector and tool nodes rely on the CompletedStep skip instead.

Steps must be idempotent by authorship. The idempotency key protects the delivery; writing the step to tolerate re-invocation is the other half of the contract.

Shared-tier recovery sweep

The shared tier uses run_locks to detect orphaned runs. Each in-flight run holds a lock row (run_id, worker_id, expires_at). A background loop — RunRecoveryLoop — runs every 30 s (override: LANTERN_RECOVERY_INTERVAL, a Go duration; invalid or ≤0 values fall back to 30 s, so the sweep cannot be disabled). Each pass:

1
Find orphans
SELECT runs WHERE status IN ('running','queued') AND (no lock row OR lock expired). Capped at 20 runs per pass.
2
Lease steal (distributed guard)
UPSERT run_locks with a fresh 10-minute TTL. Only the replica that wins the UPSERT proceeds; others skip silently — no thundering-herd double-execution.
3
Re-drive
Winner calls redriveRun → runWorkflowIfPresent (graph) or executeRunInlineSync (plain LLM). The CompletedStep hook skips already-finished nodes.
4
Resume
Interpreter continues at the first node without a step_completed row. No node that already completed is ever re-invoked.
MicroVM runs are handled separately by the recovery sweep. When the sweep finds an orphaned microVM run it calls resumeMicroVMRun — see MicroVM-tier crash-resume below — rather than re-driving it via the shared-tier inline executor. The two paths are distinct to avoid racing with the scheduler's own state machine.

Scheduler HA (microVM tier)

Leader election keeps one active scheduler, and placement state is durable. VMs lost with a node are reconciled to terminated; microVM runs are retried by the control-plane recovery sweep (resumeMicroVMRun, max 3 attempts).

Per-step retry policy

Workflow nodes can declare a retry policy in node.Data["retry"]:

{
  "retry": {
    "maxAttempts": 3,
    "backoffMs": 500,
    "retryableClasses": ["llm", "connector"]
  }
}

maxAttempts is the total number of attempts including the first (default 1 = no retry). Between attempts the interpreter journals a step_retrying event with {attempt, of, error} — visible in the run waterfall. Retryable classes: any, timeout, llm, connector, tool.

Retry wraps executeNode outside the CompletedStep check. A crash-replay skips a node only when it has a step_completed row — it does not re-retry a node that was mid-retry when the crash happened.

MicroVM-tier crash-resume

The recovery sweep handles microVM runs separately from shared-tier runs. When the sweep finds an orphaned microvm run (no live VM, lock expired), it calls resumeMicroVMRun:

  1. If a live (non-terminal) VM exists for this run — skip. The VM is self-managing.
  2. Count all runtime_vms rows for this run (total VMs ever spawned = resume attempts).
  3. If attempts ≥ 3 — mark the run failed with code microvm_resume_exhausted. No further retries.
  4. Otherwise — re-schedule a new VM via the dispatcher, injecting LANTERN_RESUME=1 in the environment.
# environment injected into the re-scheduled VM
LANTERN_RESUME=1

LANTERN_RESUME=1 signals a re-drive; skipping completed work is the workload's responsibility (no SDK helper reads it yet). The 3-attempt cap is a hard constant (maxMicroVMResumeAttempts = 3). On exhaustion the run is marked failed and its error field carries {code: microvm_resume_exhausted, attempts}.

Why it matters

  • No double-spend. LLM steps are replayed from the journal, never re-billed at the provider.
  • No double-send. Idempotency keys and side_effect_receipts de-dup LLM calls and external deliveries across retries.
  • No babysitting. The 30 s recovery sweep re-drives crashed shared-tier runs and re-schedules microVM runs (up to 3 attempts) without intervention.
  • Retry visibility. step_retrying events appear in the waterfall so you see exactly how many attempts a step took.

Each resume is still one trace per span — see Observability — and the completed run's journal is what the verifiable receipt is signed over.