Cheap until proven otherwise.
The cheapest model takes the first pass. It clears your gate — proven on the real output, not guessed from the prompt — or it escalates. Wrong answers capped at ≤10% at 95% confidence, and the router tunes itself as your traffic drifts.
No API keys, nothing installed — uvx firstpass demo prints a signed receipt.
WHAT IT COST
73% cheaperat proven quality — not predicted
the mechanism
Four moves. No black box.
Predictive routers guess a model from the prompt and hope. Firstpass runs the cheap model, checks what actually came back, and only spends more when the check fails — then learns from the outcome.
Open the cheapest rung
Every request starts on the least expensive model in your ladder. No per-prompt prediction — the cheap model simply takes the first pass.
Gate the real output
A gate checks the actual response: JSON validity, a schema, your test suite, an LLM judge, or self-consistency. Proof, not a guess.
One rung, only on fail
Fail the gate and it climbs exactly one rung — budget-capped, with cross-provider failover on transport or 5xx errors.
Tune the threshold
Deferred outcomes feed back through /v1/feedback. Adaptive conformal moves the serve threshold as your traffic drifts.
why it's different
Prediction guesses. Firstpass proves.
Guess the model from the prompt
- → A classifier picks a model before any output exists — right or wrong, you can't tell until it's too late.
- → No artifact of why a request went where it did. "Trust the model."
- → Drift silently degrades the classifier; nothing recalibrates.
- → No bound on how often a wrong answer reaches your user.
Verify the output, then serve it
- ✓ The gate reads the real response. It passes on proof or it escalates.
- ✓ Every decision is a tamper-evident, hash-chained receipt an auditor can re-verify.
- ✓ Outcomes recalibrate the serve threshold live under drift.
- ✓ A distribution-free served-failure bound — earned on real tasks, not asserted.
the guarantee
A number you can put in a contract.
On 974 real MBPP coding tasks with real test gates, Firstpass held served-failure under a 10% distribution-free target at 95% confidence — while serving 82% of requests from the cheap tier. The bound is earned from the run, not assumed.
docs/benchmarks/mbpp-live-base.txt.the gate is the point
Bring your own definition of "good."
A gate is any check on the real output. Firstpass ships several and lets you plug in your own — your tests, your linter, your judge.
Built-in inline
non-empty, json-valid, and a JSON-Schema subset — zero config, running the moment you route.
Subprocess — bring your own
Any executable. The candidate arrives as JSON on stdin (never argv, injection-resistant); print a verdict. Your tests, your linter, your policy.
LLM judge
A native judge with anti-gaming built in: a model never grades its own output, and the candidate is fenced as untrusted data under a pinned prompt.
Self-consistency
Resample k times and score by agreement — semantic-entropy style, for tasks without a crisp oracle.
Fail-open or fail-closed
Per-gate on_abstain decides what an inconclusive check does. Flaky gates auto-disable on a rolling error budget.
Priced on the receipt
Every gate's cost is written onto the receipt, so verification overhead is visible, not hidden.
system architecture
One hop between your agent and every provider.
A drop-in proxy that speaks both wire dialects inbound — Anthropic /v1/messages and OpenAI /v1/chat/completions, both live-verified. Your key passes straight through: BYOK, zero markup, redacted from every log.
tamper-evident by construction
Every decision leaves a receipt.
Each decision is a SHA-256 hash-chained, append-only record. Export the sealed log and an external auditor re-derives the chain from genesis — no proxy, no database. Tamper or reorder and it breaks at that index and exits non-zero.
{
"trace_id": "0192f3a1-7c4e-7abc", "prev_hash": "9f2c…a1b7", // chains to the prior decision — tamper-evident
"attempts": [
{ "rung": 0, "model": "…/haiku-4-5", "gate": "fail", "cost": 0.0007 }, // cheap first — the gate caught it
{ "rung": 1, "model": "…/sonnet-5", "gate": "pass", "cost": 0.0121 } // escalated, proven, served
],
"final": { "served_rung": 1, "savings_usd": 0.0502 } // vs always-top-tier
}firstpass export + verify
Export the sealed log as JSONL; anyone re-derives the hash chain from genesis on their own machine. A single altered or reordered receipt breaks the chain at its index and exits non-zero — the EU-AI-Act-style logging story, built in.
Durable, never-drop
FIRSTPASS_RECEIPTS=durable spills receipts to disk under backpressure and drains them on boot with the chain verified valid — no served decision is ever silently lost.
the cost / quality dial
One dial: how hard should this one try?
Cost, balanced, quality, latency, or max — a mode is a posture the router honors, chosen per task, per turn, even per request with one x-firstpass-mode header. Same guarantee underneath; the mode only moves where on the cost/quality frontier you sit.
Open on the cheapest rung and stay there while the gate clears. Escalate only on hard proof of need — most requests never leave the cheap tier.
Let the learned start-rung open a step higher when your own receipts say the cheap model usually fails this kind of task.
Raise the serve threshold and prefer stronger rungs. Fewer marginal passes served; the guarantee target tightens.
Fire speculative escalation in the marginal zone, so the next rung is already in flight if the gate fails — parallel tokens buy tail latency.
Open at the top of the ladder, for the rare request where the answer has to be right the first time and cost is no object.
latency is the exception: same frontier point, speculation trades parallel tokens for tail latency.(α, latency, $) budget and it solves for the knobs, lands behind a validation gate: worth exposing only once it measurably moves the frontier on real traffic. How modes compose →the research
Verify proportional to doubt.
Today Firstpass proves before it serves. The next question is how much proof each answer needs — an easy request three signals already agree on shouldn't pay for a full test run; a genuinely uncertain one should. That's elastic verification: spend verification budget as a function of measured doubt, with a conformal guarantee over the verify-or-skip decision itself, so cutting a check is never a silent risk.
Value-of-information gating
Estimate what a verification step is worth before running it — skip the checks that can't change the decision, spend on the ones that can.
Per-query difficulty
Predict how hard a request will be for the cheap rung, from the prompt and cheap probe signals — to choose where to start and how much to check.
Session-level routing
Route over a whole trajectory, not one turn — carrying difficulty and outcome context across an agent's session.
docs/benchmarks/probe-study-mbpp.txt · elastic-validation.txt.agent-first
Onboards in one step. Built for agents to run themselves.
One env var in, one env var out. No SDK, no rewrite — the proxy speaks the wire format your agent already uses, and exposes itself for machines to discover.
One command onboards you
firstpass onboard --apply detects your shell and agents, starts the proxy in observe mode (zero config, zero risk), routes your agent, and verifies itself. Idempotent, dry-run by default; firstpass offboard undoes all of it.
Discoverable at runtime
GET /v1/capabilities is a machine-readable descriptor; llms.txt and AGENTS.md ship machine-readable setup — an agent can wire and unwire itself with no human in the loop.
MCP server built in
firstpass mcp exposes traces, savings, evals, policy rehearsal, route explanation, and receipt verification as tools — so the agent reads its own decisions and rehearses changes before enforcing them.
get started
See the whole loop in ~10 seconds.
No API keys — the demo stands up a mock upstream and drives one real decision, escalation and all. It builds from source, so it needs a Rust toolchain; installing the proxy below doesn't.
uvx firstpass demo # no keys, no clone, no toolchain — watch the receipt printOr install the proxy
Read https://dshakes.github.io/firstpass/llms.txt, then install Firstpass and
route this project through it. Run `firstpass onboard` first and show me the
plan before you apply anything. Keep it in observe mode; don't touch my keys.uvx firstpass # run without installing
pip install firstpass # or install itbrew install dshakes/tap/firstpass-proxycurl --proto '=https' --tlsv1.2 -LsSf \
https://github.com/dshakes/firstpass/releases/latest/download/firstpass-proxy-installer.sh | shdocker run -p 8080:8080 -e FIRSTPASS_BIND=0.0.0.0:8080 \
ghcr.io/dshakes/firstpass:latestcargo install --git https://github.com/dshakes/firstpass firstpass-proxyThe first tab is the one you hand to your agent — every other tab is a live install channel that republishes on each release: pip · uvx · Homebrew · Docker · curl | sh · prebuilt binaries. (Cargo builds from source and needs a Rust toolchain.) Then point your agent at it: run firstpass-proxy, set ANTHROPIC_BASE_URL=http://127.0.0.1:8080, and unset it to offboard — one env var, same wire format your agent already speaks.
status
Shipped in the open — honestly labeled.
GA is a checklist we publish, not an adjective. Here's what runs today and what's still ahead.
- Both wire dialects. Anthropic /v1/messages + OpenAI /v1/chat/completions, both live-verified end-to-end.
- Enforce through tools & streaming. Structured (tool/image) enforce is default-on with verbatim per-rung carry; SSE opens instantly and streams the gated result.
- The earned guarantee. ≤10%@95% distribution-free bound on 974 real MBPP tasks (committed artifact), Learn-then-Test calibration, live adaptive threshold.
- Learned start-rung. UCB1 or discounted Thompson with logged propensities and a speculative-deferral band — default-off, honest policy stamp on every receipt.
- Selectable modes. cost / balanced / quality / latency / max presets, per route or per request via x-firstpass-mode.
- Five gate kinds. inline · schema · subprocess · LLM-judge (maker ≠ checker) · k-sample self-consistency, each with an error budget and gate cost priced onto the receipt.
- Receipts. hash-chained; export + verify re-derive with no proxy or DB; durable never-drop mode.
- Policy rehearsal. firstpass ope replays your logs with IPS / SNIPS / DR and bootstrap CIs before you enforce.
- Packaged everywhere. pip · uvx · Homebrew · Docker · curl | sh · binaries, auto-published each release.
- Constraint-form modes. explicit cost / latency / quality constraints behind a validation gate — the presets above ship today.
- Elastic verification. verify proportional to doubt, with a conformal guarantee over the verify/skip decision — signal validated, the serving path go/no-go gated.
- Per-rung P(gate-pass | features). The predictor ships today — default-off, shadow-only, recorded on every receipt; acting on it (and joint rung / sample-count routing on the easy slice) is gated on an offline AUC / Brier eval.
- Session / trajectory routing across a whole agent run, not one turn.
- Live wire verification for the OpenAI-compatible, Gemini, Bedrock, and Vertex adapters (smoke jobs activate when repo secrets are set).
- Benchmark vs the unified routing + cascading baseline, including a drift scenario the bound must survive.
- GA hardening — 30-day soak, external security audit, hosted multi-tenant plane.
Full detail: the phased roadmap and the GA handoff — every gate stated, every remaining item enumerated.
Stop guessing which model. Start proving it.
Open source, pre-GA, and honest about what's shipped. Read the deep docs or read the code.