Firstpass v0.2.1
GitHub
DocsGuaranteeBenchmarksCLIGitHub

Docs / Benchmarks

Benchmarks

Every number Firstpass publishes regenerates from a committed command. This page is the index of those artifacts, the provenance rules that keep them honest, and the exact commands to reproduce them on your own account.

On 974 real coding tasks with real unit-test gates, Firstpass served 82% of requests from the cheap rung at a 7.7% served-failure rate — below the pre-registered 10% target at 95% confidence. Every published number here traces to a committed, reproducible artifact; if the artifact isn't in docs/benchmarks/, the number is stale.

The provenance rule#

Every public number Firstpass cites must have a committed artifact — the harness's raw stdout, unedited. No artifact means the number is stale; the missing artifact is the bug to file, not the number to debate.
The rule If a number appears on this site and its artifact isn't committed under docs/benchmarks/, treat the number as stale and the missing artifact as the bug to file. Artifacts are the harness's stdout, unedited; the harness self-labels simulation vs live.

This is deliberate discipline. A benchmark you can't rerun is marketing; a benchmark with a committed command and raw output is evidence.

No committed artifact means the number is stale — open a bug, not a debate.

MBPP live bound#

The headline result: 82% of outputs cleared the gate on the cheap rung at a 7.7% served-failure rate across 974 real MBPP coding tasks — under the pre-registered 10% target at 95% confidence. The full artifact is docs/benchmarks/mbpp-live-base.txt.

The headline guarantee artifact: docs/benchmarks/mbpp-live-base.txt. 974 real MBPP coding tasks, gated by real unit tests in a Docker sandbox, live against an Anthropic solver, with pre-registered α=0.10, δ=0.05.

974
tasks
77%
candidate oracle pass-rate
5.5%
calibrated risk
7.7%
served-failure at threshold
82%
served (cheap tier)
On 974 MBPP tasks: the realized served-failure rate is 7.7%, below the 10% distribution-free target line at 95% confidence, with calibrated risk 5.5%. Separately, 82% of requests are served from the cheap tier. Served-failure vs the ≤10% target 0%2%4%6%8%10% 10% target (95% conf.) 5.5% calibrated risk 7.7% realized served-fail Where requests are served 82% cheap tier 18% 82% of requests clear the gate on the cheapest rung — no escalation, no expensive call. The remaining 18% escalate on proof of need. tasks 974 real test gates live

Reading the numbers

The raw artifact also records the gate's real error: a 14.7% false-accept rate (P(full visible pass | incorrect)) and 0.0% false-reject. The conformal step is what turns that imperfect gate into a certified serve threshold. See The guarantee for the math.

Under the hood Served-failure is not a system error — it means the gate cleared the cheap-rung output, Firstpass served it, and then the oracle (the unit test) found it incorrect. The gate is imperfect by design: with a 14.7% false-accept rate, some incorrect outputs slip through. The conformal step converts that imperfect gate into a certified bound — given the gate's calibrated error on held-out data, it guarantees the realized served-failure rate stays under 10% at 95% confidence. The 7.7% result demonstrates the bound holds in practice.
Key idea The 82% figure is the efficiency number: four out of five requests never needed a strong-model call. The 7.7% served-failure rate is the safety number: of the cheap-tier serves, how often does the unit test disagree? Both hold simultaneously — that is the point of the conformal bound.
82% cheap-tier, 7.7% served-failure, 974 live tasks, real unit-test gates — one committed artifact, one command to reproduce.

Probe study#

The go/no-go for elastic verification: the pass-count regime signal works — 77% of traffic is decidable at ~99% safety with k=5 cheap probes. The entropy AUC signal does not (0.431, near chance). Verdict: GO on elastic verification, using pass-count only.

The elastic-verification go/no-go: docs/benchmarks/probe-study-mbpp.txt. k=5, n=150 MBPP tasks, cheap-model probe. Verdict: GO with a corrected signal — the entropy AUC failed (0.431), the pass-count regime signal succeeded (77% of traffic decidable at ~99% safety).

Probe signal breakdown

Elastic verification probe study (k=5, n=150 MBPP): 0 of 5 visible passes covers 12% of traffic at 0% oracle-correct, so escalate now; 5 of 5 covers 65% at 99% oracle-correct, so serve without the expensive gate; 1 to 4 of 5 covers 23% and is mixed, so run the gate. 77% of traffic is decidable by the cheap probe alone at about 99% safety. 0%20%40%60%80%100% 12% of traffic 0 / 5 pass oracle-correct 0% → escalate now 23% of traffic 1–4 / 5 pass mixed → run the gate 65% of traffic 5 / 5 pass oracle-correct 99% → serve, skip gate 77% of traffic decidable by the cheap probe alone, at ~99% safety (k=5, n=150)
Under the hood Two signals were tested for elastic verification: entropy (how confident is the cheap model?) and pass-count (how many of k=5 cheap probes produce a passing output?). Entropy AUC was 0.431 — near chance, effectively useless as a routing signal. Pass-count succeeded: 5/5 passes correlate with 99% oracle-correct; 0/5 correlates with 0% oracle-correct. The mixed band (1–4 of 5) routes to the full gate. This is the signal the elastic verifier uses.

Full analysis on the Elastic verification page.

Pass-count wins, entropy loses — 77% of traffic is decidable by the cheap probe alone at ~99% safety.

Artifact index#

Four artifacts back the published numbers; two are committed today. If a row has no committed badge, it hasn't been run on a live key — treat it as a runnable template, not an evidence record.
ArtifactBacksCommandCost
mbpp-live-base.txt committedMBPP served-failure bound, base (test) gate--coding-livekey + Docker, ~$5
probe-study-mbpp.txt committedElastic-verification go/no-go (ADR 0008)--probe-studykey + Docker
mbpp-live-judged.txtMBPP bound with an LLM judge on the gate--coding-live + judgekey + Docker, ~$10
live-200.txt200-task live cost/success table--livekey, ~a few $
Two artifacts committed, two still unrun — the badge is the evidence.

Reproduce#

One command per artifact. The bound machinery reproduces exactly — give it your own run's gate/oracle outcomes and it recomputes the same certified threshold. The raw pass/fail counts will differ (model updates, sampling); cross-run dollar comparisons are not meaningful.

The MBPP dataset is not vendored (license + size); the command fetches the same public mbpp.jsonl the published runs used — 974 rows, with loader-rejected rows counted in the artifact.

bash
# the live served-failure bound
FIRSTPASS_CODING_DATASET=./mbpp.jsonl \
  cargo run --release -p firstpass-bench -- --coding-live

# the probe / elastic-verification study
FIRSTPASS_CODING_DATASET=./mbpp.jsonl FIRSTPASS_PROBE_K=5 \
  firstpass-bench --probe-study

What reproduces and what doesn't

What is reproducible A re-run on your account will differ (model updates, sampling). The bound machinery is the reproducible part: the harness recomputes the conformal bound from your own run's gate/oracle outcomes with the same pre-registered α/δ. Cross-artifact dollar comparisons are not meaningful — prices drift — but each artifact is internally consistent.
Under the hood The receipt's counterfactual_baseline_usd field is how Firstpass computes savings: what would this traffic have cost if every call went to the strong tier? Known modeling gap: cache-read tokens in the counterfactual are priced at full input rate, not the cheaper cache-read rate that providers charge when prompt caching is in play. This means reported savings may be slightly overstated on heavily cached traffic. The served-failure bound is unaffected — this only touches cost accounting.
Common mistake Comparing dollar figures across artifact runs. Prices drift — a counterfactual_baseline_usd from one run and a total_cost_usd from another run months later aren't directly comparable. Compare rates (served-failure rate, escalation rate, cheap-tier fraction) across runs, not absolute dollar figures.
The conformal bound is what reproduces — the raw numbers are yours; the certified threshold is universal.