Durable execution

A crash mid-run does not lose work, re-spend tokens, or fire a side effect twice. Every step is appended to the journal (journal_events) as it happens. After a restart, a recovery sweep re-drives the run, reuses the saved result of every step that already finished, and runs only what is left. External deliveries are de-duplicated by idempotency key.

Checkpoint and crash-resumeEvery step is journaled before and after. A crash between the two is reclaimed by the recovery sweep and resumed, not restarted.

⏮◀▶ Play▶

Scrub through steps1 / 12

1. acquireRunLease(run_id)

A lease with an expiry: one worker owns a run at a time.

handlers/durable_replay.go

The journal

Every step transition is appended to journal_events before the next step begins. The journal, not in-memory state, is authoritative. The shared tier writes events directly. On the microVM tier the control plane accepts step_event reports on /v1/runtime/report and writes them to the same table; the harness does not emit step events yet.

run_1   step_a  step_started -> step_completed
run_1   step_b  step_started -> step_completed
run_1   step_c  step_started    <- crash here (no step_completed written)

Other event kinds in the run waterfall: step_retrying, step_waiting (parked at an approval node), anomaly_detected (runaway_loop, excessive_steps, retry_storm, cost_spike, token_spike), confidence_evaluated and confidence_gate_bypassed (the confidence gate, when enabled).

Replay skips finished steps

On re-drive, the workflow interpreter walks the graph from the trigger again. Before any side-effecting node (ai-step, tool, connector, subagent, approval) it asks journal_events for a step_completed row. If one exists, the cached output is returned and the side effect is never re-invoked. Execution continues at the first incomplete node. A re-drive of an already-finished graph is a no-op.

The plain-LLM path has the same idea (checkCachedLLMStep): completed LLM steps are read back from the journal, so the provider sees only the original call, not a retry.

Idempotency keys

Every external side effect carries a key derived from a one-way hash of three identifiers:

idempotency_key = sha256("runID|stepID|attempt")

• Model calls send an Idempotency-Key header to OpenAI and Anthropic, so a crash-then-replay dedups at the provider instead of double-billing (llm_idempotency.go; the key never carries secret material).
• External deliveries (email, connector auth pages, browser and cross-app actions) claim a side_effect_receipts row before dispatching. A re-drive that reaches the same delivery finds the receipt and stops. Workflow connector and tool nodes rely on the completed-step skip instead.

Steps must tolerate re-invocation

The key protects the delivery. Writing the step so a second invocation is safe is the other half of the contract.

Recovery sweep (shared tier)

Each in-flight run holds a lock row (run_id, worker_id, expires_at) in run_locks. A background loop, RunRecoveryLoop, runs every 30 s. Override with PAL_RECOVERY_INTERVAL (a Go duration); invalid or non-positive values fall back to 30 s, so the sweep cannot be disabled.

1. Find orphans. Runs with status running or queued and no lock row, or an expired one. At most 20 per pass.
2. Steal the lease. An UPSERT on run_locks with a fresh 10-minute TTL. Only the replica that wins proceeds; the others skip, so two replicas never execute the same run.
3. Re-drive. The winner calls redriveRun, which runs the workflow interpreter (graph agents) or executeRunInlineSync (plain LLM).
4. Resume. Execution continues at the first node without a step_completed row.

Orphaned microVM runs take a separate path, resumeMicroVMRun (below), so the sweep does not race the scheduler’s own state machine.

Per-step retry

A workflow node can declare a retry policy in node.Data["retry"]:

{ "retry": { "maxAttempts": 3, "backoffMs": 500, "retryableClasses": ["llm", "connector"] } }

maxAttempts counts the first try (the default 1 means no retry). Between attempts the interpreter journals step_retrying with {attempt, of, error}. Retryable classes: any, timeout, llm, connector, tool. Retry wraps the node outside the completed-step check, so a node that was mid-retry at the crash is not re-retried; it is skipped only if it has a step_completed row.

MicroVM-tier crash-resume

For an orphaned microvm run (no live VM, lock expired) the sweep calls resumeMicroVMRun:

1. If a live (non-terminal) VM exists for the run, skip. The VM manages itself.
2. Count the runtime_vms rows for the run. Total VMs ever spawned equals resume attempts.
3. At 3 or more attempts (maxMicroVMResumeAttempts, a constant), mark the run failed with {code: microvm_resume_exhausted, attempts}.
4. Otherwise re-schedule a new VM with PAL_RESUME=1 in its environment.

PAL_RESUME=1 signals a re-drive. Skipping completed work is the workload’s responsibility; no SDK helper reads it yet. The scheduler is highly available through leader election, and VMs lost with a node are reconciled to terminated.

Inspecting and recovering a run

The run waterfall at /runs/<run_id> in the dashboard renders the journal. The same events are on REST and, in dev, in Postgres:

# SSE: journal events as they land
curl -N $API/v1/runs/<run_id>/events -H "Authorization: Bearer $TOKEN"

# Straight from the journal
psql $DATABASE_URL -c "SELECT seq, kind, step_id FROM journal_events WHERE run_id = '<run_id>' ORDER BY seq"

A run is stuck when it is running but its lease in run_locks has expired. There is no manual resume endpoint: the sweep re-drives it once the lease lapses (30 s poll; a lease won by an earlier recovery lasts up to 10 minutes). If a run stays stuck longer, look in the control-plane logs for a panic or a database error, since those block the sweep.

What exactly-once covers

The journal gives exactly-once completion: the terminal event (run_succeeded, run_failed or run_cancelled) and the status change happen once. External side effects are at-most-once only if the receiver honors the idempotency key. A third-party API that does not, hit by a crash between the call and the receipt write, can fire twice. Guard such calls with a step that checks a precondition first.

loop and subagent nodes are durable too: a crash mid-loop replays from the last completed iteration, and a re-driven parent finds the child’s completed step instead of spawning it again. An approval wait survives a restart. A child that stops on an approval fails the parent step with ErrChildRunPaused; nested approvals are not supported yet.

Each resume is one trace (Observability), and the completed run’s journal is what the receipt is signed over. Session turns are not journaled; see Sessions and memory.