LanternDOCS

Token Streaming

Session replies are streamed token-by-token from the model provider through the control-plane to the client. The streaming contract is a sequence of three named SSE events on GET /v1/sessions/{id}/events.

In plain termsStreaming is why chat UIs feel alive: instead of waiting ten seconds for a finished answer, words appear as the model writes them. Lantern passes each fragment ("token") straight through to your app the moment it arrives — nothing along the path holds the response back to deliver it in one lump. Your app listens on a single long-lived HTTP connection ("SSE" — server-sent events, the same technique every chat product uses) and receives three kinds of messages: a fragment arrived, the reply finished, or something went wrong.
Token streaming, end to end the model's first word reaches your UI while it is still thinking about the second LLM provider control-plane your client provider SSE — OpenAI delta.content · Anthropic content_block_delta re-emitted per token — flushed immediately, never accumulated (http.Flusher per delta) event: message_delta data: {"sessionId","turnId","seq":1..N,"delta"} event: message_completed data: {"turnId","text","usage":{tokensIn,tokensOut,costUsd}} event: message_error data: {"turnId","error"} — legacy agent.message still emitted alongside GET /v1/sessions/{id}/events (SSE) · or POST /v1/completions {"stream":true} for await (const chunk of client.sessions.streamMessage(id, {content})) { ... } TS SDK: ordered by seq (out-of-order buffered) · dashboard chat + webchat widget render the same events TWO RULES THAT KEEP IT HONEST No mid-sentence provider swap Failover to the next provider happens only if ZERO deltas have been emitted. After the first token, an error is a clean message_error — never a splice. Tool-using turns buffer — by design, and we say so A turn that calls tools (up to 5 tool rounds) can't stream partial reasoning; it emits one message_completed. Tool-free turns always stream. SSE transport — GET /v1/sessions/{id}/events The event contract — message_delta / message_completed / message_error Consuming the stream — TS SDK streamMessage Failover rule — never a mid-sentence provider swap Tool-turn buffering — the honest limitation
Provider deltas flow through the control-plane and Redis pub/sub to connected SSE clients. Tool-using turns buffer; tool-free turns stream every token.
Shared tier only. Sessions run on the shared tier (inline executor). The streaming contract described here covers interactive sessions. Headless microVM runs stream step_started / step_completed / step_failed events, not token deltas.

The three events

A single assistant turn produces events in this order:

message_delta

Emitted for each token chunk during streaming. Not emitted for tool-bearing turns (see limitation below).

{
  "sessionId": "sess_abc123",
  "turnId":    "tur_xyz789",    // stable across all events in this turn
  "seq":       1,               // monotone per-turn counter; starts at 1
  "delta":     "Hello"          // text fragment; one or more tokens
}

message_completed

Emitted exactly once per turn, after all deltas (or immediately for tool-bearing turns that do not stream). Contains the full assembled text and usage metadata.

{
  "sessionId": "sess_abc123",
  "turnId":    "tur_xyz789",
  "text":      "Hello, how can I help you today?",
  "usage": {
    "tokensIn":  42,
    "tokensOut": 12,
    "costUsd":   0.000018
  }
}

message_error

Emitted when the LLM call fails mid-stream after at least one delta has been sent. Clients should discard any partial response accumulated so far for this turnId.

{
  "sessionId": "sess_abc123",
  "turnId":    "tur_xyz789",
  "error":     "provider error — partial response discarded"
}

The turnId is stable across all three event types for the same assistant turn. Clients can use it to correlate deltas to their final message_completed, and to discard in-progress buffers when a message_error arrives.

Tool-bearing turns do not stream deltas. When the session has tools attached, the LLM call runs through a non-streaming tool-use loop (up to 5 rounds). For those turns the client receives a single message_completed without any preceding message_delta events.

SDK: sessions.streamMessage()

The TypeScript SDK exposes token streaming as an async iterator. It handles sequence reordering, unknown event filtering, and legacy server compatibility automatically.

import { LanternClient } from "@lantern/sdk";

const client = new LanternClient({ apiKey: "..." });

for await (const event of client.sessions.streamMessage(sessionId, {
  content: "Summarise the last quarter in 3 bullets.",
})) {
  if (event.type === "delta") {
    process.stdout.write(event.delta);        // stream tokens as they arrive
  } else if (event.type === "completed") {
    console.log("\nTotal cost:", event.usage.costUsd);
  }
}

The iterator yields a discriminated-union type SessionStreamEvent:

type SessionStreamEvent =
  | { type: "delta";     delta: string }
  | { type: "completed"; text: string; usage: { tokensIn: number; tokensOut: number; costUsd: number } };

Out-of-order seq values are buffered and yielded in order. A message_error from the server throws a typed MessageStreamError. Unknown event kinds are silently ignored so old and new server versions are both handled.

SSE transport

Named SSE events are delivered on the existing session event stream:

GET /v1/sessions/{id}/events
Authorization: Bearer <token>

# Server-sent events:
event: message_delta
data: {"sessionId":"...","turnId":"...","seq":1,"delta":"Hello"}

event: message_delta
data: {"sessionId":"...","turnId":"...","seq":2,"delta":", world"}

event: message_completed
data: {"sessionId":"...","turnId":"...","text":"Hello, world","usage":{...}}

Legacy clients that subscribe to agent.message events continue to receive them — the new named events are additive. The _event key in the payload tells the session event bus which SSE event name to emit.

Provider streaming

The control-plane calls the model provider in streaming mode via streamWithFailover. Each chunk from the provider triggers an onDelta callback that publishes a message_delta to the Redis session channel, which the SSE handler fans out to connected clients. The Idempotency-Key header is set on the provider request (invariant #8) so a crash-retry dedups at the provider.

Provider failover: if the primary provider fails before any deltas have been sent, the failover chain retries on the next configured provider. If the primary fails after deltas have been sent, a message_error is emitted and the partial response is discarded.

Dashboard and webchat

The dashboard session chat and the embedded webchat widget both consume these events. The webchat widget — served at /widget.js — renders token-by-token as deltas arrive and flips to the final text on message_completed.

Related: Sessions & memory covers how sessions are created and managed. Observability covers the OTel span attributes set on LLM calls.