Docs / Routing & escalation

Routing & escalation

The ladder decides where a request can go; the gate decides whether it stays. This page covers the ladder, the learned start-rung bandit, speculative escalation, cross-provider failover, budget caps, and the modes that tune all of it.

Firstpass always opens on the cheapest rung and escalates exactly one rung per gate failure, bounded by per-request, per-session, and per-day budgets. The gate decides what ships; the ladder, bandit, and speculation knobs change where the cascade starts and how fast it runs — never the correctness of the served output.

The ladder#

The ladder is an ordered list of provider/model rungs, cheapest first. A request opens at the start rung and climbs exactly one rung per gate failure. No gate failure = no escalation. A budget or rung-cap hit = serve the best attempt on hand.

The escalation rule

A ladder is an ordered list of rungs, cheapest first. Each rung is provider/model. A request opens on the start rung and climbs one rung per gate failure until something passes or a cap is hit.

Configuring the ladder

toml

[ladder]
rungs = [
  "anthropic/claude-haiku-4-5",   # rung 0
  "openai/gpt-mini",              # rung 1
  "anthropic/claude-sonnet-5",    # rung 2
]
max_rungs = 3                      # hard cap on the climb

The gate on rung 0 fails, so the request climbs exactly one rung; rung 1 passes and is served. A mis-started rung only changes where the climb begins — the gate always decides what ships.

Key idea The gate is the arbiter of what ships; the ladder is just the order in which Firstpass tries rungs. An expensive rung fires only on proof of need — a gate failure on the cheaper rung. No gate failure = cheap rung served, regardless of how many rungs are configured above it.

Common mistake Setting max_rungs too high "just in case." Every escalation costs real tokens on the higher rung. If the quality bar is set by the gate (it should be), a longer ladder only adds spend on requests that would have passed earlier — it does not improve quality above the gate's threshold.

The ladder is cheapest-first; the gate decides every climb — no gate failure, no escalation, no extra cost.

Learned start-rung#

The bandit predicts the best start rung for a traffic class — but only changes the entry point. The gate still verifies every output that ships. A wrong prediction wastes one rung's compute; it never changes what is served.

Always starting at rung 0 is wasteful when a class of traffic reliably escalates. An optional bandit learns a better start rung — where to begin the cascade. The gate still decides what to serve, so a mis-predicted start only changes the entry point, never correctness. This is the "predict-to-start, verify-to-serve" split.

The two algorithms

Two algorithms are available (ADR 0007). The default is UCB1, and the bandit is default-off.

Algorithm	Character	When
`ucb1` (default)	Deterministic, fully auditable — the same history yields the same choice.	You need every routing decision to be replayable from the receipt.
`thompson`	Beta posteriors with discounting for drift; Monte-Carlo propensities for clean off-policy evaluation.	Traffic shifts over time and you want the sampler to track it, with propensities OPE can trust.

toml

[bandit]
enabled = false            # default-off
algo    = "ucb1"           # or "thompson"

Predict-to-start, verify-to-serve: the bandit only sets where the cascade begins. A mis-drawn start never changes correctness — the gate still proves every served output.

Key idea Predict-to-start, verify-to-serve: the bandit is a cost optimizer, not a correctness mechanism. A mis-drawn start rung only means the cascade begins one rung higher than optimal — the gate still catches every bad output before it ships.

Under the hood UCB1 picks the arm with the highest upper-confidence bound — fully deterministic and replayable from the receipt. Thompson samples from a Beta posterior with drift discounting, producing per-arm propensities that off-policy evaluation (OPE) can consume directly. Both write the chosen rung to the receipt, so every routing decision is auditable.

The bandit is default-off; when on, it predicts the start rung — the gate still proves what ships. A wrong prediction is an inefficiency, not an error.

Speculative escalation#

When escalation is likely, speculation fires the next rung in parallel — so a probable gate failure pays no extra serial latency. The first rung to clear the gate is served; the rest are aborted. Output is byte-identical to the serial path. Latency changes; correctness never does.

When an escalation is likely, waiting for the gate to fail before starting the next rung pays serial latency. Speculation prefetches k rungs ahead in parallel, serves the first that passes, and discards the rest. The served bytes are byte-identical to the serial path — speculation changes latency, never output.

The speculation band

To avoid speculating on requests that will obviously pass or obviously fail, a speculation_band fires the prefetch only in the marginal gate-pass zone:

toml

[speculation]
enabled = true
lookahead = 1              # prefetch 1 rung ahead
speculation_band = [0.35, 0.65]   # only speculate when the pass-signal is marginal

In the marginal band, the next rung is prefetched in parallel so a likely escalation pays no serial latency. The first rung to clear the gate is served; the speculative rung is aborted. Latency changes; output never does.

Under the hood SSE streaming is buffered to the gate boundary before the serve decision is made, then re-emitted as SSE once the winning rung is chosen. The served-failure guarantee holds under speculation: if every speculative rung fails or is aborted, Firstpass serves the best attempt on hand — never a router error in place of a completion.

Common mistake Setting speculation_band = [0, 1] to always speculate. This fires parallel rungs on every request — including obvious cheap passes — burning higher-rung compute as wasted cost. The band is a guard: only speculate when the pass signal is genuinely marginal.

Speculation trades higher-rung compute for lower latency on likely escalations — output is byte-identical, only the timing changes.

Failover & budgets#

A transport error or provider 5xx triggers automatic cross-provider failover — the next rung fires exactly as it would on a gate failure. Budget caps bound spend at three levels (per request, per session, per day). Cap exhaustion serves the best attempt on hand, never a router error.

Escalation doubles as resilience. A transport error or a 5xx from a provider triggers cross-provider failover — the next rung may be a different vendor, so one provider's outage degrades gracefully instead of failing the request.

Key idea Failover and escalation use the same mechanism at different triggers. Gate failure → climb one rung. Transport error or provider 5xx → climb one rung. The served-failure guarantee holds in both cases: Firstpass never returns a router error in place of a completion.

Budget caps

Spend is bounded at three levels, plus the rung cap:

per_request

Cap on total cost for a single request across all rungs and gates.

per_session

Cap across a session / conversation.

per_day

A daily ceiling so an unattended loop can't produce a surprise bill.

max_rungs

The maximum number of rungs a single request may climb.

toml

[budget]
per_request_usd = 0.05
per_session_usd = 1.00
per_day_usd     = 250.00

Under the hood Budget accounting runs in the receipt layer. Each attempt's cost — including gate cost — is summed against the applicable caps before the next rung fires. Gate cost counts against per_request_usd even when the rung's output is not served: verifying a bad answer still costs tokens.

Escalation and failover are the same climb, triggered by different failures — gate or transport. Budget caps bound total spend at request, session, and day granularity.

Modes#

A mode is a named preset over the escalation knobs — one word that shifts the cost/quality/latency trade-off without editing config. The gate still validates every response regardless of the mode in force.

A mode is a named preset over the escalation knobs — a single dial that shifts the cost/quality/latency balance without editing config. Set it per task, per turn, or per request.

What each mode changes

Mode	What it does
`observe`	Shadow only — route and gate, record everything, serve the original path. Zero behavior change.
`cost`	Bias hard to the cheap tier; escalate reluctantly.
`balanced` (default)	The no-op baseline — escalate on proof of need.
`quality`	Add one rung of headroom (+1 rung) for harder work.
`latency`	Turn speculation on — prefetch ahead, serve the first pass.
`max`	Start at the top rung when nothing less will do.

Inspect the resolved presets with firstpass modes. The mode in force is recorded on every receipt.

Under the hood latency enables speculation (lookahead = 1 in the marginal band). quality widens the cascade headroom by one rung. max starts at the top rung and skips cheaper options entirely. observe runs the gate and records the receipt but serves the original provider response byte-for-byte — a zero-risk shadow. The resolved knobs for any mode are visible in firstpass modes.

Common mistake Using max mode as a blanket default. max starts at the strongest rung, paying the 15–75× tier premium on every request — including the easy ones a cheaper model would ace. balanced with a well-tuned gate handles the vast majority of traffic at a fraction of the cost; reserve max for the rare task that genuinely requires it.

Modes are one-word presets over the escalation knobs — the gate still runs on every response; the mode only changes the starting point and headroom.

Mode precedence#

The most-specific source wins: a per-request x-firstpass-mode header beats everything. The built-in fallback is balanced. The mode in force is written to every receipt.

When more than one source names a mode, the most specific wins:

x-firstpass-mode header > per-route routing_mode > FIRSTPASS_MODE_PROFILE env > balanced.

bash

# override the mode for a single request
curl localhost:8080/v1/messages \
  -H "x-firstpass-mode: quality" \
  -d @request.json

Mode resolution is pure precedence: header → per-route → env default → balanced. The receipt always records which one won.