LanternDOCS

Durable Execution

A crash mid-run does not lose work, re-spend tokens, or fire a side effect twice. Both the shared tier (inline executor) and the microVM tier guarantee exactly-once completion through the same mechanism: an event-sourced journal_events table and a CompletedStep replay gate.

In plain termsImagine an agent that's three steps into a five-step task — it's already sent an email and spent money on model calls — and the server restarts. Without durability you'd either lose the run or start over and send that email twice. Lantern writes each step to the database the moment it finishes, like a checkpoint in a video game. After a crash, the run reloads at the last checkpoint: finished steps are skipped (their saved results are reused), and only the unfinished work runs again.
Durable Execution a crash costs one step, never the run — and never a double-spent side effect 1 · THE RUN — every step checkpointed to the journal run created POST /v1/runs run_locks lease step 1 ✓ step_completed → journal output cached in the event row step 2 ✓ connector call · dedup key (run_id | step_id | attempt) step 3 · in flight… LLM call · Idempotency-Key step_started journaled, no completion yet ⚡ process crashes deploy · OOM · node loss — the goroutine is simply gone 2 · RECOVERY — nothing is asked of the caller recovery sweep every 30s finds runs in running/queued whose run_locks lease is absent or expired internal/handlers/recovery.go · RunRecoveryLoop lease steal run_locks UPSERT that only wins an EXPIRED lease — two replicas can race; exactly one wins acquireRecoveryLock re-drive shared tier: interpreter re-walks the graph from the trigger microVM tier: re-schedule same run_id · LANTERN_RESUME=1 · ≤3 attempts redriveRun → runWorkflowIfPresent | resumeMicroVMRun lease expires — the sweep notices on its next tick 3 · THE REPLAY — completed work is never re-executed step 1 — skipped CompletedStep cache hit journal row returns the output step 2 — skipped side effect NOT re-fired claimSideEffect dedup step 3 — re-executes SAME Idempotency-Key sent provider dedups — no double-billed tokens step 4 ✓ first genuinely new work since the crash run succeeded · receipt issued Ed25519 over the journal hash the crash is visible in the waterfall, not the output The guarantee exactly-once side effects — dedup at every boundary
Crash-resume: the recovery sweep wins a run_locks UPSERT, re-drives the run, and the CompletedStep hook skips nodes that already have a step_completed row.

Event-sourced journal

Every step transition is appended to journal_events before the next step begins. The journal — not in-memory state — is authoritative. On the shared tier the inline executor writes these events directly; on the microVM tier the harness streams them to the manager via the RuntimeHarness.Report RPC, which writes to the same table.

run        step           journal events
────────────────────────────────────────────────
run_1      step_a         step_started → step_completed
run_1      step_b         step_started → step_completed
run_1      step_c         step_started   ◀── crash here
                          (no step_completed written)

Other event kinds you will see in the waterfall: step_retrying (between retry attempts), step_waiting (run parked at an approval node), anomaly_detected (token budget breach), confidence_evaluated (confidence gate decision, when enabled), and confidence_gate_bypassed (gating is on but no handler is wired — step auto-approves).

CompletedStep replay

The workflow interpreter receives a CompletedStep hook wired to journalCompletedStep. Before invoking any side-effecting node (ai-step, tool, connector, subagent, approval) the interpreter queries journal_events for an existing step_completed row. If one is found, the cached output is returned and the underlying side-effect is never re-invoked.

The plain-LLM path has the equivalent checkCachedLLMStep cache: before calling the model router, the executor checks the journal for a previous successful LLM step for this run. If found, it replays the cached response — the provider is never contacted again.

No re-spent tokens on crash-replay. Completed LLM steps are read back from the journal. The provider sees only the original call, not a retry.

Resume from the last completed step

On re-drive, the interpreter walks the graph from the trigger node again. Nodes that already have a step_completed row are skipped via the CompletedStep hook. Execution continues from the first incomplete node. This is idempotent: a re-drive that re-walks an already-finished graph is a no-op.

Side-effect dedup via idempotency keys

Every external side effect carries a key derived from a one-way hash of the three identifiers, pipe-delimited:

idempotency_key = sha256("runID|stepID|attempt")

LLM provider calls receive an Idempotency-Key HTTP header on every request to OpenAI and Anthropic (derived in internal/handlers/llm_idempotency.govia a one-way hash — the key never carries secret material). A crash then replay to the same provider dedups at the provider instead of double-billing.

Connector and tool calls claim a side_effect_receipts row before dispatching. A re-drive that reaches the same step finds the receipt and short-circuits without re-invoking the connector.

Steps must be idempotent by authorship. The idempotency key protects the delivery; writing the step to tolerate re-invocation is the other half of the contract.

Shared-tier recovery sweep

The shared tier uses run_locks to detect orphaned runs. Each in-flight run holds a lock row (run_id, worker_id, expires_at). A background loop — RunRecoveryLoop — runs every 30 s (override: LANTERN_RECOVERY_INTERVAL; set "0" or "off" to disable). Each pass:

1
Find orphans
SELECT runs WHERE status IN ('running','queued') AND (no lock row OR lock expired). Capped at 20 runs per pass.
2
Lease steal (distributed guard)
UPSERT run_locks with a fresh 10-minute TTL. Only the replica that wins the UPSERT proceeds; others skip silently — no thundering-herd double-execution.
3
Re-drive
Winner calls redriveRun → runWorkflowIfPresent (graph) or executeRunInlineSync (plain LLM). The CompletedStep hook skips already-finished nodes.
4
Resume
Interpreter continues at the first node without a step_completed row. No node that already completed is ever re-invoked.
MicroVM runs are handled separately by the recovery sweep. When the sweep finds an orphaned microVM run it calls resumeMicroVMRun — see MicroVM-tier crash-resume below — rather than re-driving it via the shared-tier inline executor. The two paths are distinct to avoid racing with the scheduler's own state machine.

Scheduler HA (microVM tier)

On the microVM tier, the scheduler itself is not a single point of failure. Placement state is durable, and a replacement scheduler picks up pending and in-flight work. For VMs with idempotent: true and a recent snapshot, the scheduler issues a fresh Spawn on another node. Non-idempotent VMs are marked failed and the caller is notified via the event stream.

Per-step retry policy

Workflow nodes can declare a retry policy in node.Data["retry"]:

{
  "retry": {
    "maxAttempts": 3,
    "backoffMs": 500,
    "retryableClasses": ["llm", "connector"]
  }
}

maxAttempts is the total number of attempts including the first (default 1 = no retry). Between attempts the interpreter journals a step_retrying event with {attempt, of, error} — visible in the run waterfall. Retryable classes: any, timeout, llm, connector, tool.

Retry wraps executeNode outside the CompletedStep check. A crash-replay skips a node only when it has a step_completed row — it does not re-retry a node that was mid-retry when the crash happened.

MicroVM-tier crash-resume

The recovery sweep handles microVM runs separately from shared-tier runs. When the sweep finds an orphaned microvm run (no live VM, lock expired), it calls resumeMicroVMRun:

  1. If a live (non-terminal) VM exists for this run — skip. The VM is self-managing.
  2. Count all runtime_vms rows for this run (total VMs ever spawned = resume attempts).
  3. If attempts ≥ 3 — mark the run failed with code microvm_resume_exhausted. No further retries.
  4. Otherwise — re-schedule a new VM via the dispatcher, injecting LANTERN_RESUME=1 in the environment.
# environment injected into the re-scheduled VM
LANTERN_RESUME=1

The in-VM agent reads LANTERN_RESUME and consults the journal CompletedStep cache to skip already-completed nodes, then continues from the first incomplete node. The 3-attempt cap is a hard constant (maxMicroVMResumeAttempts = 3) — on exhaustion the run is permanently failed and the dashboard shows a step_failed event with code: microvm_resume_exhausted.

Why it matters

  • No double-spend. LLM steps are replayed from the journal, never re-billed at the provider.
  • No double-send. Idempotency keys and side_effect_receipts de-dup external side effects across retries.
  • No babysitting. The 30s recovery sweep and the microVM scheduler HA recover crashed runs without intervention.
  • Retry visibility. step_retrying events appear in the waterfall so you see exactly how many attempts a step took.

Each resume is still one trace per span — see Observability — and the completed run's journal is what the verifiable receipt is signed over.