Docs / How it works

How it works

Firstpass is a proxy that sits in front of your provider calls. A request enters, moves through route → prove → escalate → serve, and its outcome feeds back to the router. Here is the full journey.

Firstpass intercepts your model calls and runs them through four moves: Route to the cheapest rung, Prove the real output against a gate, Escalate exactly one rung only on gate failure, and Learn from deferred outcomes. The gate is what makes cheapest-first safe — it reads the actual completion, not a pre-execution prediction, and only escalates on evidence. On a measured run of 974 real coding tasks, 82% of requests were served from the first rung at a 7.7% served-failure rate. The remaining 18% escalated once; zero paid the strong tier as a default.

The request flow#

The cascade is five boxes in sequence: Route → Gate → (Escalate → re-Gate) → Serve → Receipt + Feedback. The green path — gate passes on the first rung — covers the majority of traffic. The red loop fires only on gate failure, climbs one rung, and re-gates. On exhaustion the cascade always terminates with a real model answer, never a router error.

The client keeps its existing endpoint — POST /v1/messages or POST /v1/chat/completions. Firstpass intercepts, runs the cascade, and returns a normal response. The identical served bytes go back to the client; everything else is bookkeeping.

One request, end to end. The green path (gate passes on the first rung) covers the majority of traffic; the red loop is the escalation that only fires on proof of need.

Key idea The gate-first design is what makes cheapest-first safe. The cheap model is not selected because a classifier predicted it would pass — it is selected first, then its output is proved. No prediction commits before the output exists. Escalation fires on evidence of failure, not on suspicion.

The cascade always terminates with a real served answer. The green (pass) path handles the majority of traffic; the red (escalate) path fires only on proved gate failure.

1 · Route#

Every request opens on the cheapest rung — the first entry in rungs. There is no per-prompt classifier deciding which model should answer before any output exists. The cheap model takes every first pass. An optional learned start-rung bandit can shift where the cascade begins for traffic classes it has seen escalate repeatedly, but it never changes what gets served — only the gate decides that.

The request opens on the start rung — by default the cheapest model in the ladder. Rungs are ordered cheap-to-capable and named provider/model:

The ladder

toml

[ladder]
rungs = [
  "anthropic/claude-haiku-4-5",   # rung 0 — start here
  "openai/gpt-mini",              # rung 1
  "anthropic/claude-sonnet-5",    # rung 2
]

No classifier — proof instead

There is no per-prompt classifier deciding which model. The cheap model simply takes the first pass. An optional learned start-rung bandit can move the entry point for classes of traffic it has seen escalate — but that only sets where the cascade begins, never what it serves.

Under the hood Firstpass forwards the entire request body verbatim, swapping only the model field. For routing decisions it reads: model name, tool count, image blocks, and request headers. It does not inspect, summarize, or rewrite the message history — and tool_use/tool_result blocks are forwarded byte-for-byte, not re-verified.

The cheap rung opens every request. The gate — not a classifier — decides whether it stays there.

2 · Prove#

The gate reads the real candidate output — not a prompt-time prediction — and returns one of three verdicts: pass, fail, or abstain. A pass serves immediately and stops the cascade. Firstpass ships five gate kinds: inline checks, JSON-Schema, a bring-your-own subprocess, a native LLM judge, and self-consistency. On 974 real coding tasks scored by unit tests, 82% cleared the gate at rung 0 — see Benchmarks.

What a gate sees that a classifier can't

The candidate output is handed to the gate — the heart of the system. A gate reads the actual response and returns a verdict: pass, fail, or abstain. Firstpass ships five kinds (full detail): inline checks, JSON-Schema, a bring-your-own subprocess, a native LLM judge, and self-consistency.

Why the real output matters A predictive router commits before any output exists. A gate commits after — it can catch a malformed JSON, a failing test, or a hallucinated field that no prompt-time classifier could have seen.

If the gate passes, the output is served immediately and the cascade stops. On 974 MBPP tasks, that first-rung pass happened for 82% of requests.

Key idea Every gate kind operates on the already-returned completion. A json-schema gate can catch a hallucinated field. An inline check catches a missing key in microseconds. A subprocess gate can run cargo test or pytest on the actual output. These are things a prompt-time classifier decides blind — the gate has the answer in hand before it rules.

Common mistake Assuming the gate itself costs as much as a model call. The gate runs on the completion that already returned — it does not need a separate model call unless you configure an LLM-judge gate. Inline, schema, and subprocess gates add fractions of a millisecond, not token spend. Only the llm-judge gate incurs a second model call, and its cost is reported separately in the receipt's gate_cost_usd field so you can see it clearly.

The gate commits after the output exists, catches failures no classifier could have predicted, and stops the cascade the moment it passes.

3 · Escalate#

On gate failure, Firstpass climbs exactly one rung, re-runs the call, and re-gates the new output. Escalation is bounded by max_rungs, per-request/session/day budgets, and a hard termination guarantee: on exhaustion, Firstpass serves the best attempt — a real model answer, never a router error in place of a completion.

On gate failure, Firstpass climbs exactly one rung and re-runs the candidate + gate. Escalation is bounded:

The four rules

One rung at a time — never a jump straight to the top (unless max mode asks for it).
Budget-capped — per-request, per-session, and per-day caps stop runaway spend; max_rungs caps the climb.
Cross-provider failover — a transport error or 5xx from a provider fails over to the next rung, which can be a different provider entirely.
Speculative (optional) — in the marginal gate-pass zone, prefetch the next rung in parallel so a likely escalation doesn't pay serial latency. Served bytes are byte-identical to the serial path.

Failover vs escalation

See Routing & escalation for the ladder, the bandit, speculation, and budgets in full. Failover (triggered by a 5xx or transport error) and escalation (triggered by gate failure) both advance one rung, but they are logged separately in the receipt so you can distinguish "provider down" from "output failed the gate."

Key idea Escalation is bounded by design — there is no path to an infinite loop. max_rungs caps the climb, budget caps stop runaway spend, and on exhaustion the served-failure guarantee kicks in: a real model answer goes back to the client, never an error message in place of a completion.

Common mistake Treating escalation as the normal path and sizing the ladder accordingly (many strong-model rungs "just in case"). The cheap rung passes 82% of real coding traffic — escalation is the exception. A ladder heavy with strong-model rungs means paying the strong-tier price on traffic the cheap rung would have handled. Let the gate data from observe mode tell you how many rungs you actually need.

One rung at a time, bounded by caps, with cross-provider failover — and always a real answer at the end, even if every rung is exhausted.

4 · Learn#

Deferred outcomes — a CI test passing hours later, a user accepting an answer, a reviewer flagging a problem — are posted to POST /v1/feedback. These drive adaptive conformal calibration (Gibbs–Candès), which nudges the serve threshold as your traffic distribution drifts over time. Two Prometheus gauges expose the live state of the loop.

Deferred outcomes

Serving is not the end. When the true outcome of a request becomes known later — a test passed in CI, a user accepted the answer, a reviewer flagged it — you report it to POST /v1/feedback. Those deferred outcomes drive adaptive conformal calibration (Gibbs–Candès), which nudges the serve threshold as your traffic drifts. Two Prometheus gauges expose the loop live: firstpass_serve_threshold and firstpass_realized_served_failure. See The guarantee.

The calibration loop

Under the hood Conformal calibration is a statistical guarantee, not a heuristic. The Gibbs–Candès method nudges the serve threshold so the realized served-failure rate tracks your configured target (default 10%) as traffic distribution shifts over time. There is no model to retrain — the threshold is a single scalar that adjusts in response to feedback. The two Prometheus gauges let you see the current threshold and the trailing realized rate in any monitoring dashboard.

POST real-world outcomes to /v1/feedback; conformal calibration keeps the serve threshold honest as traffic drifts.

Observe vs enforce#

Set FIRSTPASS_MODE=observe to shadow the cascade without changing what's served. Every receipt records the full decision — rungs tried, gate verdicts, what would have been served — so you can validate before going live. Flip to enforce once the numbers convince you.

Firstpass runs in one of two modes, set by FIRSTPASS_MODE:

observe

Shadow. Route and gate every request, record the full decision and what would have happened — but serve your original path unchanged. Zero behavior change; pure measurement. The safe way to start.

enforce

Live. The cascade actually decides what is served. Structured enforce is the default-on production path.

Recommended rollout

The recommended rollout is observe first — collect a few days of receipts, run firstpass savings and firstpass evals to see what the cascade would have done — then flip to enforce once the numbers convince you.

Key idea Observe mode is byte-identical to your existing provider path from the client's point of view — every response is your original provider's answer. The cascade runs in parallel and records receipts, but changes nothing. It is the no-risk way to see exactly what Firstpass would have done on real traffic before you commit to enforce.

Start in observe, collect receipts, run firstpass savings and firstpass evals, then flip to enforce with data behind you.

Anatomy of a decision#

Every request produces one hash-chained receipt — a single JSON object capturing the rungs tried, each gate verdict, the mode, the cost, and a chain link that makes tampering detectable. The receipt is the ground truth for every routing decision Firstpass makes.

What's in a receipt

Every request produces one receipt capturing the whole decision — the rungs tried, each gate verdict, the mode in force, the cost, and the chain link. A trimmed example:

json

{
  "seq": 4021,
  "request_id": "req_9f2c…",
  "mode": "balanced",
  "attempts": [
    { "rung": 0, "model": "anthropic/claude-haiku-4-5",
      "gate": [{"name": "json-valid", "verdict": "pass"},
               {"name": "schema", "verdict": "fail", "reason": "missing 'total'"}],
      "cost_usd": 0.0003, "escalated": true },
    { "rung": 1, "model": "openai/gpt-mini",
      "gate": [{"name": "json-valid", "verdict": "pass"},
               {"name": "schema", "verdict": "pass", "score": 0.83}],
      "cost_usd": 0.0018, "served": true }
  ],
  "gate_cost_usd": 0.0000,
  "total_cost_usd": 0.0021,
  "prev_hash": "sha256:a91f…",
  "hash": "sha256:5c02…"
}

Inspecting after the fact

Inspect any decision after the fact with firstpass trace <request_id> or firstpass explain. The receipt is the ground truth — see Receipts & audit.

Under the hood The prev_hash field chains each receipt to the previous one using SHA-256. This makes the log tamper-evident: any deletion or modification of a past receipt breaks the chain. The receipt store is append-only by design, and firstpass audit walks the chain and reports the first broken link.

The receipt is the ground truth for every decision — rungs tried, gate verdicts, cost, and hash chain link. firstpass trace <request_id> retrieves any decision by ID.