LanternDocs

Token streaming

Session replies are streamed token-by-token from the model provider through the control-plane to the client. The streaming contract is a sequence of three named SSE events on GET /v1/sessions/{id}/events.

In plain termsStreaming is why chat UIs feel alive: instead of waiting ten seconds for a finished answer, words appear as the model writes them. Lantern passes each fragment ("token") straight through to your app the moment it arrives — nothing along the path holds the response back to deliver it in one lump. Your app listens on a single long-lived HTTP connection ("SSE" — server-sent events, the same technique every chat product uses) and receives three kinds of messages: a fragment arrived, the reply finished, or something went wrong.
Token streaming, end to end Sequence between the LLM provider, the control plane and your client. The client posts a message and opens the SSE events stream. The control plane streams from the provider and re-emits every token as a message_delta event, flushed immediately, never accumulated, then message_completed with usage, or message_error on failure. Failover to another provider only happens if zero deltas were emitted. Tool-using completions buffer tool rounds internally and emit the final text as deltas. Token streaming, end to end The model's first word reaches your UI while it is still thinking about the second. LLM provider control-plane your client POST /v1/sessions/{id}/messages SSE transport: GET /v1/sessions/{id}/events GET /v1/sessions/{id}/events (SSE) stream request (Idempotency-Key) token deltas (provider SSE) The event contract: message_delta, message_completed, message_error message_delta · seq 1..N · flushed per token message_completed · text, usage, costUsd message_error · on failure, never a splice Consuming the stream: TS SDK streamMessage TS SDK: for await (const chunk of client.sessions.streamMessage(id, { content })). Dashboard chat and the webchat widget render the same events. Also: POST /v1/completions with stream: true. Two rules that keep it honest Failover rule: never a mid-sentence provider swap No mid-sentence provider swap Failover to the next provider happens only if zero deltas were emitted. After the first token, an error is a clean message_error to discard the partial stream. Tool-turn buffering: the honest limitation Tool-using turns buffer, by design Agent completions with tools run the tool rounds non-streaming internally, emit tool_call_* events, then the final text as message_delta. Tool-free turns stream.
Provider deltas flow through the control-plane and Redis pub/sub to connected SSE clients. Tool-using turns buffer; tool-free turns stream every token.
Sessions only. Session turns run in the control plane's session handler (not the run executor). The streaming contract described here covers interactive sessions. Headless microVM runs stream step_started / step_completed / step_failed events, not token deltas.

The three events

A single assistant turn produces events in this order:

message_delta

Emitted for each token chunk during streaming. Not emitted for tool-bearing turns (see limitation below).

{
  "sessionId": "sess_abc123",
  "turnId":    "tur_xyz789",    // stable across all events in this turn
  "seq":       1,               // monotone per-turn counter; starts at 1
  "delta":     "Hello"          // text fragment; one or more tokens
}

message_completed

Emitted exactly once per turn, after all deltas (or immediately for tool-bearing turns that do not stream). Contains the full assembled text and usage metadata.

{
  "sessionId": "sess_abc123",
  "turnId":    "tur_xyz789",
  "text":      "Hello, how can I help you today?",
  "usage": {
    "tokensIn":  42,
    "tokensOut": 12,
    "costUsd":   0.000018
  }
}

message_error

Emitted when the LLM call fails mid-stream after at least one delta has been sent. Clients should discard any partial response accumulated so far for this turnId.

{
  "sessionId": "sess_abc123",
  "turnId":    "tur_xyz789",
  "error":     "provider error — partial response discarded"
}

The turnId is stable across all three event types for the same assistant turn. Clients can use it to correlate deltas to their final message_completed, and to discard in-progress buffers when a message_error arrives.

Tool-bearing turns do not stream deltas. When the session has tools attached, the LLM call runs through a non-streaming tool-use loop (up to 5 rounds). For those turns the client receives a single message_completed without any preceding message_delta events.

SDK: sessions.streamMessage()

The TypeScript SDK exposes token streaming as an async iterator. It handles sequence reordering, unknown event filtering, and legacy server compatibility automatically.

import { LanternClient } from "@lantern/sdk";

const client = new LanternClient({ apiKey: "..." });

for await (const event of client.sessions.streamMessage(sessionId, {
  content: "Summarise the last quarter in 3 bullets.",
})) {
  if (event.type === "delta") {
    process.stdout.write(event.delta);        // stream tokens as they arrive
  } else if (event.type === "completed") {
    console.log("\nTotal cost:", event.usage.costUsd);
  }
}

The iterator yields a discriminated-union type SessionStreamEvent:

type SessionStreamEvent =
  | { type: "delta";     delta: string }
  | { type: "completed"; text: string; usage: { tokensIn: number; tokensOut: number; costUsd: number } };

Out-of-order seq values are buffered and yielded in order. A message_error from the server throws a typed MessageStreamError. Unknown event kinds are silently ignored so old and new server versions are both handled.

SSE transport

Named SSE events are delivered on the existing session event stream:

GET /v1/sessions/{id}/events
Authorization: Bearer <token>

# Server-sent events:
event: message_delta
data: {"sessionId":"...","turnId":"...","seq":1,"delta":"Hello"}

event: message_delta
data: {"sessionId":"...","turnId":"...","seq":2,"delta":", world"}

event: message_completed
data: {"sessionId":"...","turnId":"...","text":"Hello, world","usage":{...}}

Legacy clients that subscribe to agent.message events continue to receive them — the new named events are additive. The _event key in the payload tells the session event bus which SSE event name to emit.

Provider streaming

The control-plane calls the model provider in streaming mode via streamWithFailover. Each chunk from the provider triggers an onDelta callback that publishes a message_delta to the Redis session channel, which the SSE handler fans out to connected clients. The Idempotency-Key header is set on the provider request (invariant #8) so a crash-retry dedups at the provider.

Provider failover: if the primary provider fails before any deltas have been sent, the failover chain retries on the next configured provider. If the primary fails after deltas have been sent, a message_error is emitted and the partial response is discarded.

Dashboard and webchat

The dashboard session chat and the embedded webchat widget both consume these events. The webchat widget — served at /widget.js — renders token-by-token as deltas arrive and flips to the final text on message_completed.

Related: Sessions & memory covers how sessions are created and managed. Observability covers the OTel span attributes set on LLM calls.