Verify proportional to doubt: run a cheap probe, and only pay for expensive verification where the probe is uncertain — with a conformal guarantee over the verify/skip decision itself. The serving path is implemented and off by default; you turn it on with a calibrated threshold.
Uniform verification charges the full gate on every output. Elastic verification asks a cheap question first — how uncertain are the samples? — and collapses 77% of traffic to a probe-only decision at ~99% safety, letting the expensive gate concentrate on the 23% that is genuinely mixed. The serving path is implemented, config-gated, and off by default; supply a calibrated λ under escalation.elastic to enable it. The pre-registered entropy signal failed; the page explains what validated instead and why.
Status: implemented, default-off The serving path is in the code, config-gated, and off by default — supply a calibrated λ under escalation.elastic to enable it; until you do, Firstpass serves exactly as it does today. The go/no-go study below came back GO with a corrected signal. Uniform verification remains the default — elastic verification earns default status only after it replicates on your distribution.
Uniform verification pays full gate cost on every output, easy or hard. The elastic alternative inserts a cheap probe step: it runs k cheap samples and counts visible passes. Outputs that are obviously correct (all k pass) skip the gate; outputs that are obviously wrong (none pass) escalate immediately; only the genuinely uncertain middle runs the full gate. The skip/verify decision is placed under a conformal guarantee so that skipping stays safe by construction — not by optimism.
Uniform vs elastic — the key difference
Today every served output pays for the gate. But most outputs are obviously fine or obviously broken — uniform verification spends the same effort on both. Elastic verification asks a cheaper question first: how much doubt is there? A cheap probe sorts traffic into "clearly serve," "clearly escalate," and "genuinely uncertain," and only the uncertain slice pays for the full gate. The verify/skip decision itself is placed under a conformal guarantee, so skipping stays safe by construction.
Key idea The probe doesn't decide what is correct — only where doubt lives. A unanimous probe (all k pass, or none pass) tells you something reliable about the output's quality with high probability. A split probe tells you the output is genuinely uncertain, and that is exactly the slice that should pay for the full gate. The insight is that confidence is checkable cheaply even when correctness is not.
k=5 cheap probes fan out; the visible-pass count routes each request to escalate now, run the full gate, or serve directly — verification budget collapses onto the uncertain middle.
Under the hood The conformal guarantee on the skip decision works by bounding the probability of a wrong skip. It isn't a soft heuristic — skipping when all k samples pass is justified by a statistical coverage argument: at k=5 with 99% oracle-correct rate in the study, the probability of a bad skip is bounded by construction. When you calibrate λ on your own distribution in shadow mode you are measuring whether your data meets that same threshold before trusting the skip path.
Elastic verification sorts traffic by doubt, not by content — only the uncertain 23% pays full gate cost, and the skip decision is covered by a conformal guarantee, not guesswork.
ADR 0008 pre-registered a go/no-go: build the skip logic only if a probe signal is meaningfully better than chance and skip-safety is high. k=5 samples per task over n=150 MBPP tasks with a cheap model. The pre-registered signal — entropy AUC — failed at 0.431, below the no-signal line. The winning signal is visible-pass count, which was not pre-registered but is reported as such. GO verdict, with the corrected signal. Numbers hold for this distribution; replicate on yours before enabling.
What was measured, and what wasn't
ADR 0008 pre-registered a go/no-go: build the skip logic only if a probe signal is meaningfully better than chance and skip-safety is high. The study (docs/benchmarks/probe-study-mbpp.txt) ran k=5 samples per task over n=150 MBPP tasks with a cheap model.
k=5
samples per task
150
MBPP tasks
99%
skip-safety when confident
77%
traffic decidable by probe alone
Under the hood MBPP (Mostly Basic Programming Problems) is a benchmark of ~370 Python programming tasks with test-suite oracles — a good fit for visible-pass counting because "passes all tests" is a clean boolean gate. The study deliberately chose a distribution where the probe signal could be measured with high confidence. This is also the limitation: MBPP is code-correct, short-output, single-model. The regimes need measurement on your actual distribution — open-ended generation, multi-step reasoning, and longer outputs may produce very different traffic splits.
Common mistake Reading the 77%/99% numbers as if they apply to your workload. They apply to this study's distribution: n=150 MBPP tasks, one cheap model, code-correct gate. For a different task kind — creative writing, multi-hop reasoning, tool-use chains — the regime splits may look entirely different. The point of shadow mode (see Turning it on safely) is to measure your own splits before the skip path touches anything you serve.
The GO verdict is justified by visible-pass count on n=150 MBPP with a cheap model — the pre-registered entropy signal failed; the corrected signal validated. Replicate before generalizing.
The winning signal is the visible-pass count across k cheap samples. It cleanly splits traffic: 0/5 → escalate now (12% of traffic, 0% oracle-correct); 5/5 → serve, skip gate (65%, 99% oracle-correct); 1–4/5 → run the gate (23%, mixed). Verification budget collapses onto the 23% that actually needs it — the other 77% is decided by the cheap probe alone, at roughly 99% safety.
The winning signal is the visible-pass count across the k cheap samples. It cleanly splits traffic into three regimes:
k=5, n=150 MBPP. 0/5 → escalate now (12% of traffic, 0% oracle-correct); 5/5 → serve without the expensive gate (65%, 99% oracle-correct); 1–4/5 → run the gate (23%). 77% is decided by the cheap probe alone at ~99% safety.
Regime
Traffic
Oracle-correct
Action
0 / 5 visible passes
12%
0%
Escalate now — skip the cheap attempt.
1–4 / 5 visible passes
23%
mixed
Run the gate — verification concentrates here.
5 / 5 visible passes
65%
99%
Serve — skip the expensive gate.
Why the boundaries work
The verification budget collapses onto the 23% that actually needs it. The other 77% is decided by the cheap probe alone, at roughly 99% safety.
The structural reason the extremes are reliable: if none of k independent cheap-model attempts passes the gate, the task is almost certainly beyond what the cheap model can handle — escalation is appropriate and no budget is wasted on a known failure. If all k pass, the consensus of independent samples is a strong positive signal; in the study that unanimity corresponded to 99% oracle-correctness, making a conformal skip safe. The 1–4 range is genuinely ambiguous — cheap samples disagree — and that is exactly where paying for the full gate is worth it.
Key idea The 0/5 boundary is a hard structural signal, not just low confidence: every independent sample failed, which means the cheap model consistently cannot solve the task. The 5/5 boundary is a strong consensus: every independent attempt found an answer that passes the gate. The middle — where samples disagree — is the only range that carries irreducible uncertainty, and that is where the full gate earns its cost.
Verification collapses onto the 23% where cheap samples genuinely disagree — the unanimous extremes (0/5 and 5/5) are decided by the probe alone at high reliability.
The pre-registered signal was entropy AUC of self-consistency against oracle-failure. It came back at 0.431, below the 0.5 no-signal line. Entropy measures spread, not direction — five samples that all fail look as "confident" as five that all pass. Visible-pass count carries the directional signal entropy discards, and that is the signal that validated. The failed metric is reported as failed.
The direction problem
The study is honest about what did not work. The pre-registered continuous metric — the AUC of self-consistency entropy against oracle-failure — came back at 0.431, below the 0.5 no-signal line. Entropy discards direction: it measures how spread the samples are, not whether they agree on the right answer. Five samples that all fail look as "confident" as five that all pass.
Under the hood AUC (area under the ROC curve) measures how well a continuous score separates two classes — here, oracle-pass vs oracle-fail. An AUC of 0.5 means the score is no better than random at separating them; 0.431 is slightly worse than random, which can happen when the score carries an inverted signal or pure noise. Entropy carries inverted signal for the extreme regimes: 0/5 and 5/5 are both low entropy (high agreement), but they have opposite oracle-correctness. A metric that treats those two identically will never separate them, and that is exactly what 0.431 reflects.
The correction The verdict is GO — but justified by the pass-count regime signal, not the entropy AUC. Firstpass ships the signal that validated, and says so plainly. A metric that failed its pre-registration is reported as failed.
Key idea If you design a new probe signal: measure direction, not just spread. Self-consistency in the sense of "all samples agree" is only useful if you also know what they agree on. Pass-count captures that; entropy does not. This distinction is the lesson that makes visible-pass count the right signal for gate-compatible tasks.
Entropy AUC = 0.431; it failed because it discards direction. Visible-pass count carries direction and validated — the failed metric is named, not buried.
The serving path is merged, config-gated, and off by default. Supply escalation.elastic with a calibrated λ to enable; leave it absent for byte-identical behavior. The recommended first step is shadow mode: compute the probe signal on live traffic and record it on the receipt without acting on it, so you verify the three regimes hold on your distribution before any skip logic touches what you serve. Measure, then enable.
Shadow mode — the recommended first step
The serving path is merged and config-gated, default-off: set escalation.elastic with a calibrated λ and the expensive gates to skip, and elastic verification takes effect; leave it unset and behavior is byte-identical to today. The recommended first move is shadow measurement — compute the probe signal on live traffic and record it on the receipt without acting on it — so you confirm the three regimes hold on your own distribution before any skip logic touches what you serve. Measure, then enable.
toml
# firstpass.toml# Step 1 — shadow: record the probe signal on receipts without acting on it.# Leave escalation.elastic absent. Run traffic. Inspect your regime splits.## Step 2 — enable: once your own 5/5 regime shows ≥99% oracle-correct,# supply a calibrated λ to activate the skip path.
[escalation.elastic]
lambda = 0.9 # example — calibrate this on YOUR distribution's shadow data
Under the hood "Config-gated, default-off" means exactly that: the serving path code is compiled in and present; when escalation.elastic is absent from the config, the router skips the elastic branch entirely and falls through to uniform verification as if the branch does not exist. Byte-for-byte identical behavior. Setting escalation.elastic activates the probe on every request for that escalation tier, runs k cheap samples in parallel, and routes by visible-pass count. The probe result is also written to the receipt so you can audit which requests were skipped, which were escalated, and which ran the full gate.
Common mistake Enabling elastic verification without first running shadow measurement on your own traffic. The regime splits in this page come from MBPP with a cheap model. Your distribution may have a smaller 5/5 block, a different mix in the uncertain middle, or task types where the cheap probe cannot form a meaningful signal at all. Enabling without validation risks skipping the gate on a 5/5 cohort that is not actually 99% oracle-correct on your data. Shadow mode costs only the probe samples — it does not change what you serve — so there is no reason to skip it.
Supply escalation.elastic with a calibrated λ to enable; leave it absent for byte-identical uniform verification. Always validate the regime splits on your distribution in shadow mode first.
One model, one distribution, n=150. The regimes need replication on other task kinds before elastic verification earns default status. Probe cost is real — k=5 cheap samples cost something, and that cost must be covered by the gate savings on the 77% skip path. Not a default today; uniform verification is the shipped behavior.
One model, one distribution. The study is n=150, a single cheap model, MBPP only. The regimes must replicate on other task kinds before default-on.
Probe cost is real.k=5 cheap samples cost something; the win is that they're far cheaper than the expensive gate they replace on 77% of traffic.
Not a default. Uniform verification remains the shipped behavior. Elastic verification earns default status only after replication.
Key idea The savings case holds when two things are true simultaneously: the 5/5 block on your distribution is large enough and reliable enough (close to 99% oracle-correct), and the gate cost being skipped is substantially more expensive than k cheap probe samples. If your traffic lands mostly in the uncertain middle, or if your task type defeats cheap-probe signal, elastic verification may add probe cost without meaningful gate savings. Shadow mode tells you which case you are in before you commit.
Elastic verification is implemented and off by default — it earns its place on your stack through shadow validation on your distribution, not by inheriting the study's numbers.