Docs / Choose your model by proof
Choose your model by proof
A public benchmark cannot tell you which cheaper model is good enough for your traffic. Firstpass measures it on your own task outcomes, with an error bound that holds however often you look, and leaves the switch to you.
[route.challenger] to a route. A randomized share of sessions is served by the challenger ladder through the normal enforce path. You post the task outcome of each session to /v1/feedback. Firstpass reports one of three states: certified, halted, or undecided. It never switches your model on its own.What and why#
On 500 real SWE-bench agent runs per model, no gate available on a finished agent run was precise enough for per-request escalation to beat always calling the top model. Choosing the right model for the workload is what won: the cheapest model on par with opus-4.5 resolved 374 vs 384 tasks (difference not significant) at $0.219 vs $0.754 per task, 71% cheaper. Numbers and method: agentic-real-runs. That selection was made on a public benchmark. This feature makes the same choice on your traffic, using the only honest evidence for an agent task: the task outcome itself. Design record: ADR 0014.
alpha at whatever moment you read it, so checking the panel every morning does not inflate the error rate.1. Configure a challenger#
Put the challenger on an enforce route. Every rung of both ladders must resolve to a provider and a price. Restart the proxy after editing.
[[route]]
mode = "enforce"
ladder = ["anthropic/claude-opus-4-5"] # what you run today (the control)
gates = ["non-empty"]
[route.challenger]
ladder = ["openrouter/minimax-m2.5"] # what you would like to switch to
percent = 10 # % of sessions served by it, randomized per session
# margin = 0.05 # default: may be at most 5 points worse
# alpha = 0.05 # default: error rate of each conclusion| Key | Default | Meaning |
|---|---|---|
ladder | required | The ladder under test, cheapest first, as provider/model. |
percent | required | Percent of sessions served by the challenger, strictly between 0 and 100. The rest are the control. |
margin | 0.05 | How much worse (task success rate) the challenger may be and still be certified. |
alpha | 0.05 | Error rate of each conclusion (certify and halt). |
2. Post task outcomes#
Arms are assigned per session. Send the same x-firstpass-session header for every request of one agent run; without it, on an enforce route, Firstpass derives a session from the request itself (see Coding agents), and single-turn requests are each their own session. Then report the outcome with the standard feedback call. The body is trace_id, gate_id, verdict (pass, fail or abstain), reporter, and an optional score in [0, 1].
# 1. Group an agent run into one session so it stays on one arm
curl http://localhost:8080/v1/messages \
-H "x-firstpass-session: run-4821" \
-H "content-type: application/json" -d @request.json
# 2. Find the receipt for that session (trace_id and session_id are both listed)
curl -s localhost:8080/v1/receipts?limit=5
# 3. Post the task outcome: the CI result, test run or rating you already have
curl http://localhost:8080/v1/feedback \
-H "content-type: application/json" \
-d '{ "trace_id": "<trace_id from the receipt>",
"gate_id": "tests",
"verdict": "pass",
"reporter": "ci" }' pass or fail of each session counts (whatever its gate_id) and abstain is ignored, so mixing several signals makes "first one wins" arbitrary. A trace owned by another tenant returns 404.3. Read the result#
Two places show the same evidence. firstpass experiment replays your receipts and feedback through the test (add --json or --tenant ID); with nothing to report it tells you the next step instead of printing an empty table. The panel at /panel has an Experiments tab fed by GET /v1/experiments: a card per route with the state, each arm's success rate and $/request, and a log-scale bar for each e-value (the evidence) racing toward 1/alpha.
firstpass experiment
open http://localhost:8080/panel#experimentsThe three states#
| State | Meaning (error at most alpha) | What happens |
|---|---|---|
| certified | The challenger is no worse than your ladder by more than margin. | If it is also cheaper per request, the report prints the ladder to adopt. You make that edit. |
| halted | The challenger is worse by more than margin. | New sessions stop going to it automatically, with an error log and a metric. Traffic only moves back to your ladder. |
| undecided | Not enough evidence yet. | Keep running and keep posting outcomes. |
The panel also shows unavailable when the receipt history could not be read at startup. Firstpass then cannot know whether the challenger was already halted, so it fails closed and serves only your ladder.
To adopt a certified challenger, replace the route's ladder with the challenger's and remove [route.challenger]. The report and the panel print the exact line.
Paired mode: single-turn routes, no feedback call#
On a route where every request stands alone and your gate is a real check, set mode = "paired". After each sampled request is answered by your ladder, the challenger runs on the same request off the hot path and both answers are graded by your gates. No caller is ever served by the challenger and there is nothing to post to /v1/feedback. Because both models see the same task, the test decides 3 to 4 times faster than live mode in the MBPP replay.
[route.challenger]
ladder = ["anthropic/claude-haiku-4-5"]
mode = "paired"
sample_rate = 0.25 # share of requests re-run on the challenger
max_usd_per_day = 5.0 # required: the challenger makes real callsThe panel card says paired, shows the sample rate and today's challenger spend against the cap, and counts requests compared instead of sessions. Today's spend is rebuilt from the store at startup, so a restart does not reset the cap. What gets certified is gate-pass non-inferiority, so it is only as good as your gate: a route gated only by non-empty is rejected.
Defaults and how long it takes#
These figures come from replaying the design on the 500 real runs per model (incumbent opus-4.5, 50/50 split, 400 replays, capped at 20,000 sessions), not from a live run. At the default margin = 0.05:
| Challenger vs incumbent | Outcome | Median sessions |
|---|---|---|
| Genuinely close (1 point worse, e.g. minimax-2.5, gemini-3-flash) | Certified in 99% of replays, halted in 1% | about 3,600 |
| 10 points worse (haiku-4.5) | Halted in 99% | about 2,200 |
| 20 points worse (gpt-5-mini) | Halted in 100% | about 220 |
A tighter margin = 0.03 certified only 63% of the good challengers within 20,000 sessions, which is why the default is 0.05. The cost is information, not the method: two arms at about 76% success need thousands of sessions to separate a 4-point gap. A lower percent slows the experiment but limits how much traffic the challenger sees.
Live, paired mode (real sonnet-5 vs haiku-4-5, tests run in a sandbox, full write-up): on all 974 MBPP tasks haiku passed 84.6% against sonnet's 95.4% and was halted at pair 137, robust to the order of the pairs. On 290 APPS contest problems the gap was 4.5 points (65.2% vs 60.7%), and the experiment correctly stayed undecided: inside the margin, but not yet provably so.
Limits#
- Outcomes, on the traffic it saw. A workload shift after certification is not covered. Run a fresh experiment.
- Cost is reported, not tested. Adoption is advised only when the challenger's $/request is lower.
- You need outcome feedback. Without it there is nothing to certify, and gate verdicts alone would repeat the failure the benchmark found.
- Changing a ladder, the margin or alpha starts a fresh experiment. Receipts from the old configuration are ignored and counted.
- The halt state is per process and rebuilt from receipts at startup. With several replicas each halts on the outcomes it receives, so read the report against the store all replicas write to. Without a shared log halt state is not shared across replicas; with one, each replica syncs cluster halt state on the first request per (tenant, route) per process lifetime.
- With a shared Redis log the daily spend cap (
max_usd_per_day) is cluster-wide. Each replica seeds its local ledger from incomingLogRecord::Pairentries, so the combined spend across all replicas is tracked. Without a shared log the cap is per-process and N replicas can spend up to N× the cap. The cluster can still overshoot by at most roughly one sync interval (~5 s) of spend, becausemay_spendis checked before the call and the ledger is updated asynchronously. - Log errors are fail-closed. If the shared log is unreachable when a challenger evaluation starts, the challenger is skipped for that request (live mode serves the incumbent; paired mode skips the evaluation). Spend debited to the local process ledger and health tallied before the log call are preserved — they are not lost. Pair records whose append failed are stored in SQLite with
in_log=false; on the next restart with a shared log configured these rows are replayed locally (evidence and cost count toward the cap). They are not re-pushed to Redis, so other replicas do not see them — only the process that restarted recovers them. - Errors and cut-off answers are not random. They cluster on long, hard requests. An incumbent error drops its pair; a challenger error counts as a failure, so errors can never flatter the challenger toward certification. Answers that stopped at
max_tokensare graded as they are. Each experiment reports both per arm underhealth, and the panel warns when either is non-zero: read them before trusting the rates. The live run hit this before the enforce timeout was raised to 600 s.
firstpass experiment or the panel. Certified and cheaper means you adopt it; halted means Firstpass already stopped it.