Firstpass
DocsGuaranteeBenchmarksCLIGitHub

Docs / Choose your model by proof

Choose your model by proof

A public benchmark cannot tell you which cheaper model is good enough for your traffic. Firstpass measures it on your own task outcomes, with an error bound that holds however often you look, and leaves the switch to you.

Add a [route.challenger] to a route. A randomized share of sessions is served by the challenger ladder through the normal enforce path. You post the task outcome of each session to /v1/feedback. Firstpass reports one of three states: certified, halted, or undecided. It never switches your model on its own.

What and why#

On 500 real SWE-bench agent runs per model, no gate available on a finished agent run was precise enough for per-request escalation to beat always calling the top model. Choosing the right model for the workload is what won: the cheapest model on par with opus-4.5 resolved 374 vs 384 tasks (difference not significant) at $0.219 vs $0.754 per task, 71% cheaper. Numbers and method: agentic-real-runs. That selection was made on a public benchmark. This feature makes the same choice on your traffic, using the only honest evidence for an agent task: the task outcome itself. Design record: ADR 0014.

Key idea Routers that predict which model will do well do not prove it on your outcomes. Here the test is on outcomes, and it is anytime-valid: each conclusion is wrong with probability at most alpha at whatever moment you read it, so checking the panel every morning does not inflate the error rate.

1. Configure a challenger#

Put the challenger on an enforce route. Every rung of both ladders must resolve to a provider and a price. Restart the proxy after editing.

toml
[[route]]
mode   = "enforce"
ladder = ["anthropic/claude-opus-4-5"]     # what you run today (the control)
gates  = ["non-empty"]

[route.challenger]
ladder  = ["openrouter/minimax-m2.5"]      # what you would like to switch to
percent = 10                               # % of sessions served by it, randomized per session
# margin = 0.05                            # default: may be at most 5 points worse
# alpha  = 0.05                            # default: error rate of each conclusion
KeyDefaultMeaning
ladderrequiredThe ladder under test, cheapest first, as provider/model.
percentrequiredPercent of sessions served by the challenger, strictly between 0 and 100. The rest are the control.
margin0.05How much worse (task success rate) the challenger may be and still be certified.
alpha0.05Error rate of each conclusion (certify and halt).

2. Post task outcomes#

Arms are assigned per session. Send the same x-firstpass-session header for every request of one agent run; without it, on an enforce route, Firstpass derives a session from the request itself (see Coding agents), and single-turn requests are each their own session. Then report the outcome with the standard feedback call. The body is trace_id, gate_id, verdict (pass, fail or abstain), reporter, and an optional score in [0, 1].

bash
# 1. Group an agent run into one session so it stays on one arm
curl http://localhost:8080/v1/messages \
  -H "x-firstpass-session: run-4821" \
  -H "content-type: application/json" -d @request.json

# 2. Find the receipt for that session (trace_id and session_id are both listed)
curl -s localhost:8080/v1/receipts?limit=5

# 3. Post the task outcome: the CI result, test run or rating you already have
curl http://localhost:8080/v1/feedback \
  -H "content-type: application/json" \
  -d '{ "trace_id": "<trace_id from the receipt>",
        "gate_id":  "tests",
        "verdict":  "pass",
        "reporter": "ci" }' 
Common mistake Post one outcome per session, and the same kind every time. Only the first pass or fail of each session counts (whatever its gate_id) and abstain is ignored, so mixing several signals makes "first one wins" arbitrary. A trace owned by another tenant returns 404.

3. Read the result#

Two places show the same evidence. firstpass experiment replays your receipts and feedback through the test (add --json or --tenant ID); with nothing to report it tells you the next step instead of printing an empty table. The panel at /panel has an Experiments tab fed by GET /v1/experiments: a card per route with the state, each arm's success rate and $/request, and a log-scale bar for each e-value (the evidence) racing toward 1/alpha.

bash
firstpass experiment
open http://localhost:8080/panel#experiments

The three states#

StateMeaning (error at most alpha)What happens
certifiedThe challenger is no worse than your ladder by more than margin.If it is also cheaper per request, the report prints the ladder to adopt. You make that edit.
haltedThe challenger is worse by more than margin.New sessions stop going to it automatically, with an error log and a metric. Traffic only moves back to your ladder.
undecidedNot enough evidence yet.Keep running and keep posting outcomes.

The panel also shows unavailable when the receipt history could not be read at startup. Firstpass then cannot know whether the challenger was already halted, so it fails closed and serves only your ladder.

To adopt a certified challenger, replace the route's ladder with the challenger's and remove [route.challenger]. The report and the panel print the exact line.

Paired mode: single-turn routes, no feedback call#

On a route where every request stands alone and your gate is a real check, set mode = "paired". After each sampled request is answered by your ladder, the challenger runs on the same request off the hot path and both answers are graded by your gates. No caller is ever served by the challenger and there is nothing to post to /v1/feedback. Because both models see the same task, the test decides 3 to 4 times faster than live mode in the MBPP replay.

toml
[route.challenger]
ladder          = ["anthropic/claude-haiku-4-5"]
mode            = "paired"
sample_rate     = 0.25   # share of requests re-run on the challenger
max_usd_per_day = 5.0    # required: the challenger makes real calls

The panel card says paired, shows the sample rate and today's challenger spend against the cap, and counts requests compared instead of sessions. Today's spend is rebuilt from the store at startup, so a restart does not reset the cap. What gets certified is gate-pass non-inferiority, so it is only as good as your gate: a route gated only by non-empty is rejected.

Defaults and how long it takes#

These figures come from replaying the design on the 500 real runs per model (incumbent opus-4.5, 50/50 split, 400 replays, capped at 20,000 sessions), not from a live run. At the default margin = 0.05:

Challenger vs incumbentOutcomeMedian sessions
Genuinely close (1 point worse, e.g. minimax-2.5, gemini-3-flash)Certified in 99% of replays, halted in 1%about 3,600
10 points worse (haiku-4.5)Halted in 99%about 2,200
20 points worse (gpt-5-mini)Halted in 100%about 220

A tighter margin = 0.03 certified only 63% of the good challengers within 20,000 sessions, which is why the default is 0.05. The cost is information, not the method: two arms at about 76% success need thousands of sessions to separate a 4-point gap. A lower percent slows the experiment but limits how much traffic the challenger sees.

Live, paired mode (real sonnet-5 vs haiku-4-5, tests run in a sandbox, full write-up): on all 974 MBPP tasks haiku passed 84.6% against sonnet's 95.4% and was halted at pair 137, robust to the order of the pairs. On 290 APPS contest problems the gap was 4.5 points (65.2% vs 60.7%), and the experiment correctly stayed undecided: inside the margin, but not yet provably so.

Limits#

Add four lines of config, post one outcome per session, read firstpass experiment or the panel. Certified and cheaper means you adopt it; halted means Firstpass already stopped it.