Firstpass puts a distribution-free bound on how often a wrong answer is served. It is computed from a real run — not assumed — and tracked live as your traffic drifts. This is the headline claim, and this page shows exactly how it is earned.
Firstpass makes two distinct guarantees. The delivery guarantee: even under provider exhaustion or failure, Firstpass always serves a real model answer — never a router error in place of a completion. The quality bound: a distribution-free statistical upper bound on the served-failure rate (wrong answers served), configurable target defaulting to ≤10% at 95% confidence. Measured on 974 real coding tasks, Firstpass hit 7.7% served-failure with 82% served from the cheap tier — both under the bound, on a live run with real unit-test gates.
Firstpass will always serve a real model answer — never a router error in place of a completion. Even if every rung exhausts or the provider fails, the best available model attempt is served. This is unconditional and always active; it is not config-gated.
What this means, precisely
Every request that enters the serving path exits with a model-generated completion. Firstpass does not substitute a router error — a 502, a fallback stub, a "provider unavailable" message — for an actual model answer. If a clean route succeeds, that completion is served. If every rung fails (provider timeouts, rate limits, escalation ceiling hit), the best real model attempt buffered during the request is served. The client always receives a model-generated token stream.
Two guarantees, not one
Holding these separately avoids a common source of confusion:
I
Delivery guarantee
A model answer always arrives. The router never replaces a completion with a router-generated error. Always active — not config-gated.
II
Quality bound
A statistical upper bound on how often the served answer is wrong. Earned from calibration data, held live by adaptive conformal. Covered in the sections below.
Key idea "Served-failure" in Firstpass means a wrong model answer was served — not that delivery failed. A served failure is a request where the model completed, the gate passed it, and the oracle later found it incorrect. The delivery guarantee and the quality bound are orthogonal: one says something always arrives, the other says how often that something is correct.
Common mistake Reading the served-failure rate as a delivery failure rate. If the served-failure rate is 7.7%, that means 7.7% of served answers were wrong — not that 7.7% of requests failed to deliver. Every one of those requests still got a model completion; the model just happened to be incorrect on that task.
Under the hood Exhaustion case: if every rung has been attempted and all failed, Firstpass serves the best real model attempt buffered during the request rather than propagating the provider error upstream. This is implemented in the serving path and is always active — not config-gated.
Delivery is unconditional: Firstpass always returns a real model answer even under provider exhaustion — the served-failure rate measures answer quality, not delivery.
The quality bound is a distribution-free upper bound on the served-failure rate — the fraction of served outputs that are actually wrong. You pick a target α and a confidence 1−δ; Firstpass certifies whether the serve threshold holds the bound. Two methods are available: split-conformal (Hoeffding bound on served-failure rate, default) and Learn-then-Test (exact-binomial bound on the gate's false-accept rate). Both require no distributional assumptions.
What is being bounded
The guarantee is a bound on the served-failure rate: the fraction of served outputs that are actually wrong. Firstpass offers two ways to earn it, both distribution-free (no assumption about the shape of your data):
Split-conformal — a Hoeffding upper confidence bound on the served-failure rate.
Learn-then-Test / RCPS — fixed-sequence exact-binomial testing that reports the gate's false-accept rate at the chosen threshold (firstpass calibrate --method ltt).
Either way, you pick a target failure rate α and a confidence 1−δ, and Firstpass certifies whether the chosen serve threshold holds the bound.
Key idea The bound is distribution-free: it does not assume your inputs follow a Gaussian, Pareto, or any other shape. It is derived from the empirical outcomes of a real calibration run — which is why reproducing the benchmark with your own data is meaningful, not cosmetic.
Firstpass certifies a distribution-free upper bound on served-failure rate — you supply the target and confidence, calibration supplies the threshold.
On 974 real MBPP coding tasks, gated by real unit tests, with pre-registered α=0.10, δ=0.05: realized served-failure was 7.7% (under the 10% target at 95% confidence), calibrated risk was 5.5%, and 82% of requests were served from the cheap tier with no escalation.
Reading the numbers
The committed result — from docs/benchmarks/mbpp-live-base.txt — is 974 real MBPP coding tasks, gated by real unit tests, live against an Anthropic solver, with pre-registered α=0.10, δ=0.05:
≤10%
distribution-free target, 95% conf.
5.5%
calibrated risk (conformal)
7.7%
realized served-failure at threshold
82%
served from the cheap tier
Both the calibrated risk (5.5%) and the realized served-failure (7.7%) sit under the 10% target, at 95% confidence — while 82% of requests never left the cheapest rung.
Read that as: on this run, serving every output that clears the threshold produced a wrong answer 7.7% of the time, and the machinery certifies the true rate is at or below 10% with 95% confidence — while the expensive tier was needed for fewer than one request in five.
What the raw artifact also shows The gate's false-accept rate — P(full visible pass | actually incorrect) — was 14.7%, and its false-reject rate was 0.0%. The conformal step turns that imperfect gate into a certified serve threshold rather than trusting the gate blindly.
Under the hood The calibrated risk (5.5%) and the realized served-failure (7.7%) can differ because calibration uses a held-out slice, not the full run. Calibrated risk is what conformal guarantees the rate stays under; realized served-failure is what actually happened on the live run. Both sit under the 10% line, which is what the guarantee requires. The gap between them is slack the held-out calibration set purchased — a larger calibration set typically narrows it.
On 974 real coding tasks: 7.7% served-failure (below the 10% bound at 95% confidence), 82% served cheap — full artifact at Benchmarks.
Conformal calibration holds out a labeled slice, derives a gate score threshold such that serving above it keeps served-failure under α, and expresses uncertainty as a Hoeffding upper confidence bound. No distributional assumptions. Outputs: threshold, served fraction, calibrated risk, feasibility flag.
How it works
Split-conformal calibration holds out a labeled slice of outcomes and derives a threshold on the gate's score such that serving above it keeps the failure rate under α. The bound is a Hoeffding upper confidence bound, so it is valid for any data distribution — no Gaussian assumptions, no model of the errors. The reported figures are the threshold (0.500 here), the served fraction (82%), the calibrated risk (5.5%), and a feasibility flag.
Key idea Conformal speaks in served-failure rate terms: the output is a certified bound on how often a served answer is wrong. The feasibility flag tells you whether the calibration data was large enough to certify the target α at the chosen confidence — if infeasible, collect more labeled outcomes before deploying the threshold.
Common mistake Running calibration on the same data used to tune the gate. Conformal requires a held-out calibration set — using the training data conflates fitting with bounding, and the certified threshold will undercount the true failure rate. Hold out a fresh, labeled sample and run calibration on that alone.
Conformal gives a Hoeffding-bounded served-failure guarantee from a held-out calibration slice — no distributional assumptions, just labeled outcomes and a feasibility check.
LTT frames threshold selection as multiple hypothesis testing: it walks candidate thresholds in a fixed sequence, applies an exact-binomial test at each, and stops at the first that provably controls risk. Where conformal bounds the served-failure rate, LTT bounds the gate's false-accept rate — the same property viewed through the gate rather than the served output.
What LTT measures
Learn-then-Test frames threshold selection as multiple hypothesis testing. It walks candidate thresholds in a fixed sequence, applying an exact-binomial test at each, and stops at the first that provably controls risk. Where conformal reports a served-failure bound, LTT reports the gate's false-accept rate at the chosen threshold — the same guarantee viewed through the gate.
Which to use Conformal is the default and speaks directly in served-failure terms. LTT is the choice when you want the exact-binomial guarantee expressed as the gate's false-accept rate, or when the fixed-sequence testing better matches your risk framing.
Key idea The two methods certify the same underlying property from two angles. Conformal says: "the served-failure rate is ≤ α." LTT says: "the gate's false-accept rate is ≤ α." On the benchmark run, the gate's false-accept rate was 14.7% — the conformal step converts that imperfect gate into a certified serve threshold by being conservative about what it serves.
LTT certifies the gate's false-accept rate via exact-binomial testing — the same guarantee as conformal expressed through the gate rather than the served output.
A static threshold decays as your traffic drifts. Adaptive conformal (Gibbs–Candès) tracks the threshold live: each deferred outcome reported to /v1/feedback nudges the threshold to keep the realized failure rate near target. No recalibration run, no downtime.
Why drift matters
A static threshold derived from calibration data drifts as your traffic changes — model updates, input distribution shifts, new task types. Adaptive conformal (Gibbs–Candès) tracks the threshold live: each deferred outcome reported to /v1/feedback nudges the threshold up or down to keep the realized failure rate near the target. No recalibration run, no downtime — the bound follows the distribution.
Each reported outcome moves the serve threshold to keep realized served-failure under target as traffic drifts — no recalibration run, no downtime.
Closing the feedback loop
Adaptive conformal requires you to close the loop: after a served answer's true outcome is known (a test run passes, a user confirms, an oracle grades it), post it to POST /v1/feedback with the trace ID. Firstpass matches the outcome to the gate score from that request and updates the threshold. The tighter and faster the feedback loop, the more responsive the threshold.
Under the hood The Gibbs–Candès update is an online algorithm: it adjusts the threshold by a small step each time an outcome arrives, proportional to whether that outcome was a failure or a success. This keeps the time-averaged failure rate near α without reprocessing the full calibration holdout on each update.
Common mistake Reporting outcomes selectively — only posting feedback when an answer was wrong, or only when it was right. Adaptive conformal requires representative feedback: cherry-picking which outcomes you report shifts the threshold in the direction of your bias. Report all outcomes, or at minimum a random sample.
Adaptive conformal keeps the quality bound live as traffic drifts, consuming each /v1/feedback post to nudge the threshold — no batch recalibration needed.
The guarantee is observable in Prometheus in real time: firstpass_serve_threshold tracks the live adaptive threshold, and firstpass_realized_served_failure is the measured failure rate from reported outcomes. Alert on the second approaching your target α.
What to watch
The loop is observable in Prometheus, so you can watch the guarantee hold in real time:
firstpass_serve_threshold
The current threshold the router serves above. Moves as adaptive conformal responds to feedback.
firstpass_realized_served_failure
The measured served-failure rate from reported outcomes. This is what you alert on if it approaches your target.
Key idea The threshold and the realized failure rate move together: if the threshold rises (more conservative), fewer requests serve cheap but fewer failures pass through; if it falls, more serve cheap but the failure rate edges up. A widening gap between the two metrics is a signal that feedback reporting has gone stale.
Common mistake Alerting only when firstpass_realized_served_failurecrosses your target rather than approaching it. By the time the realized rate crosses α, the threshold has been serving at risk for some window. Alert at 80–90% of your target to give the feedback loop time to respond.
Two Prometheus gauges give real-time visibility: the adaptive threshold and the realized failure rate — watch both, alert before the rate hits target.
Every published number regenerates from a committed command. Run the same harness against the same 974-row mbpp.jsonl to recompute the conformal bound from your own live gate/oracle outcomes with the same pre-registered α/δ.
What the harness does
Every published number regenerates from a committed command. The bound machinery is the reproducible part — the harness recomputes the conformal bound from your run's gate/oracle outcomes with the same pre-registered α/δ.
bash
# fetch the same public mbpp.jsonl (974 rows), then:
FIRSTPASS_CODING_DATASET=./mbpp.jsonl \
cargo run --release -p firstpass-bench -- --coding-live
Needs your provider key and Docker (for the sandboxed test gate); the run costs a few dollars. See Benchmarks for the full artifact index and provenance rules.
Under the hood The harness runs each of the 974 MBPP tasks through Firstpass (cheap tier first, gate checked, escalate on fail), records gate scores and oracle outcomes, then feeds them to the conformal calibration to derive the threshold and certified risk. The pre-registered α=0.10, δ=0.05 are committed in the artifact — you can change them in your own run, but the published numbers will not match if you do.
The published benchmark is fully reproducible: committed harness + committed data + same pre-registered parameters = same certified bound.