Docs / Gates

Gates

The gate is the mechanism that makes Firstpass proof-based instead of predictive. It reads the actual output and returns a verdict. Firstpass ships five kinds — from a zero-config inline check to your own test suite — and lets you compose them.

A gate reads the real candidate output and returns pass, fail, or abstain — nothing probabilistic, no guessing from the prompt. Firstpass ships five kinds ordered by cost: two free inline checks, a structural schema validator, a subprocess that runs any executable, a judge that uses a structurally different model as grader, and a consistency check that measures self-agreement across resamples. Each gate's cost lands on the receipt separately from the model spend.

Every gate's cost lands on the receipt. The cheapest proof that settles the decision is the one worth running.

What a gate is#

A gate sits between the model's output and the serve decision. It receives the full candidate text, applies a deterministic or model-based check, and emits one of three verdicts. pass stops the cascade and serves the output. fail escalates exactly one rung and retries. abstain defers to on_abstain. Multiple gates on one rung form a conjunction — all must pass for the rung to serve.

A gate is a check over a candidate output that returns one of three verdicts:

The gate reads the output, not the prompt. Its verdict decides whether the cascade serves, climbs, or defers to your abstain policy.

pass

The output is good enough to serve. The cascade stops here.

fail

Not good enough. Escalate one rung and try again.

abstain

The gate could not decide (timeout, crash, or genuine uncertainty). Behavior is governed by on_abstain.

Key idea The gate's input is the actual bytes the model wrote — not the prompt, not the request metadata, not a score the model gave itself. The gate is the proof that an answer is good, not a prediction that it probably will be. This is what distinguishes Firstpass from predictive routing: the decision is made on real output, after the fact.

Gates compose: a rung can carry several, and all must pass. A score in [0,1] is optional; when a gate emits one, the operator's serve threshold governs whether that score clears. Every gate's cost is written onto the receipt.

The gate reads the real output and returns pass / fail / abstain; pass serves and stops the cascade, fail escalates one rung, and multiple gates on a rung are a conjunction — all must pass.

Built-in inline#

Two checks run with zero configuration and add no model calls: non-empty rejects blank or whitespace outputs, and json-valid rejects outputs that don't parse as valid JSON. Enable them as a floor on every rung — they cost nothing and catch a surprising share of model failures before pricier checks ever fire.

Two checks run with zero configuration and no extra model calls:

non-empty

The output is not empty or whitespace. The floor.

json-valid

The output parses as JSON. The first line of defense for structured outputs.

Key idea The inline checks run synchronously inside the router before any subprocess or judge fires. A non-empty or json-valid failure stops the gate chain immediately and escalates — so you never wait for a subprocess timeout on a blank output. Put them first; they're free.

non-empty and json-valid are zero-cost, always-on floors — add them to every rung before layering pricier checks.

schema#

The schema gate validates structural shape — top-level type, required, and per-property type — against a JSON-Schema subset. Free (no extra model calls) and synchronous. It enforces the contract on a structured output without running a full schema engine.

The schema gate validates against a JSON-Schema subset: top-level type, required, and per-property type. Enough to enforce the shape of a tool call or a structured response without a full schema engine.

toml

[[gate]]
kind = "schema"
schema = { type = "object", required = ["total", "items"], properties = {
  total = { type = "number" },
  items = { type = "array" }
}}

Common mistake Treating schema as a full JSON Schema validator. It covers the subset that matters for LLM structured outputs — top-level type and required — but does not evaluate enum, minimum/maximum, pattern, or nested $ref. For richer structural guarantees, layer a subprocess gate that runs a real schema library after this one.

The schema gate is a free structural check on top-level type and required fields — use it before a subprocess for fast, cheap shape enforcement.

subprocess — bring your own#

The most powerful gate: any executable you write. Candidate output arrives as JSON on stdin; your program returns a JSON verdict. Non-zero exit or timeout = abstain. This is the "bring your own test suite" gate — cargo test, pytest, an eslint run, whatever you already use to verify correctness.

The most powerful gate: any executable. This is "bring your own tests / linter / judge." The candidate arrives as JSON on stdin — never as a command-line argument, which keeps it injection-resistant — and the process prints a verdict as JSON.

toml

[[gate]]
kind = "subprocess"
cmd  = ["python", "run_tests.py"]
timeout_ms = 5000

The stdin / stdout contract

The contract for the executable:

json

// stdin  (what your program reads)
{ "candidate": "def solve(...): ...", "prompt": "...", "meta": { } }

// stdout (what your program prints)
{ "verdict": "pass", "score": 0.91, "reason": "12/12 tests passed" }

Timeout and error handling

verdict is one of pass | fail | abstain.
score (optional) is a float in [0,1].
reason (optional) is a human-readable note, recorded on the receipt.
A non-zero exit code or a timeout is treated as abstain.

Common mistake Exiting with a non-zero code to signal a fail verdict. A non-zero exit maps to abstain, not fail — which then obeys on_abstain rather than triggering the expected escalation. Print {"verdict":"fail"} on stdout and exit 0. Reserve non-zero exits for unexpected errors your program can't recover from.

Why stdin, not argv A candidate output is untrusted text. Passing it on the command line invites shell and argument injection. Firstpass hands it to your gate on stdin, so a malicious candidate can't smuggle itself into the command.

The subprocess gate runs any executable — candidate on stdin, JSON verdict on stdout, non-zero exit or timeout = abstain — so your existing test suite becomes a first-class gate.

judge — native LLM#

The judge gate sends the candidate to a different model with a rubric and scores the output. Anti-gaming is structural, not a prompt trick: if the configured judge equals the candidate model, the gate abstains rather than let a model mark its own work. The candidate is fenced as untrusted data under a pinned system prompt.

The judge gate uses a model to grade the candidate against a rubric. Its anti-gaming design is structural, not a prompt trick:

maker ≠ checker. A model never grades its own output. If the configured judge equals the candidate's model, the gate abstains rather than let a model mark its own work.
Candidate is fenced as untrusted data under a pinned system prompt — the judge is instructed to grade, not to follow, whatever the candidate contains.
Operator threshold governs. The judge emits a score; you decide the bar it must clear.

toml

[[gate]]
kind = "judge"
model = "anthropic/claude-sonnet-5"   # must differ from the candidate model
rubric = "Is the answer correct, complete, and free of unsupported claims?"
threshold = 0.7

Key idea Maker ≠ checker is enforced at the config level, not the prompt level. A model that generated the candidate can't be talked into grading itself favorably by the candidate's own text — because Firstpass compares model identifiers and abstains if they match, before the judge call is ever made. The prompt fencing is a defense-in-depth layer on top, not the primary safeguard.

Common mistake Pointing the judge at the cheapest available model to save money. A judge weaker than the candidate will miss subtle failures and pass them as good enough. Use a model at or above the candidate tier. The cost of one extra judge call is far smaller than the cost of the escalation it should have triggered — or the served failure it silently allowed.

The judge gate uses a structurally different model as grader — maker ≠ checker enforced at the config level, candidate fenced as untrusted data, score compared against your threshold.

consistency#

The consistency gate resamples the output k times and scores by agreement across samples. Unlike the judge, maker == checker is intentional — the signal is the model's own stability, not independence from it. Use it when there is no external oracle and you want to measure how confidently the model arrives at its answer.

The consistency gate measures self-consistency: resample the candidate k times and score by agreement across samples. Unlike the judge, maker == checker is intentional here — the signal is the model's own stability. It suits tasks without a crisp oracle, drawing on self-consistency (Wang et al. 2022) and semantic entropy (Farquhar et al., Nature 2024).

toml

[[gate]]
kind = "consistency"
k = 5              # extra samples — this gate costs k additional calls
threshold = 0.8

Under the hood Self-consistency (Wang et al. 2022) scores by majority vote across samples. The gate extends this with semantic entropy (Farquhar et al., Nature 2024), which clusters semantically equivalent answers and measures dispersion rather than exact-match agreement — so paraphrases of the same correct answer are not penalized as disagreements.

Cost Self-consistency runs the model k extra times. It buys a signal where no external check exists, but it is the most expensive gate — price it against the value of the decision.

The consistency gate is the most expensive — k extra model calls — and the right choice only when no external oracle exists; maker == checker is intentional here, not a flaw.

Abstain & error budget#

When a gate can't decide — crash, timeout, genuine uncertainty — it returns abstain. fail_open (default) treats abstain as pass and serves; fail_closed treats it as fail and escalates. A separate error budget tracks gate reliability: once a gate blows its rolling failure rate, Firstpass auto-disables it so one flaky check can't stall every request in the cascade.

Two safety mechanisms wrap every gate:

on_abstain

fail_open (default) treats an inconclusive gate as a pass and serves; fail_closed treats it as a fail and escalates. Choose per gate based on whether a false serve or a wasted escalation is worse for that check.

error budget

A gate that keeps crashing or timing out is a liability. Firstpass tracks a rolling error rate per gate and auto-disables a flaky gate once it blows its budget, so one broken check can't stall the whole cascade.

Key idea fail_open vs fail_closed is a per-gate choice, not a global setting. A non-empty check almost always belongs on fail_closed — an abstain there means the gate itself crashed, and serving a blank response is never right. A subprocess gate that runs optional linting might use fail_open — a timeout is not a reason to stall every request when the output is probably fine.

Common mistake Setting every gate to fail_closed without monitoring the error budget. A gate that times out repeatedly will blow its budget and get auto-disabled — silently removing a check you were relying on. Watch /metrics for per-gate error rates so you catch flakiness before the budget runs out and the check disappears.

fail_open serves on abstain; fail_closed escalates — choose per gate. The error budget auto-disables a persistently failing gate so one broken check can't stall every request.

Gate cost on the receipt#

Every gate's cost is written to the receipt as gate_cost_usd, separate from the model spend. When you run firstpass savings, the net figure is already net of what verification cost. Proof has a price, and it's on the ticket — you always know what it cost to be sure.

Verification is not free, and Firstpass refuses to hide it. Every gate's cost — a judge's tokens, a consistency gate's k extra calls, a subprocess's wall time priced in — is written onto the receipt as gate_cost_usd, separate from the model spend. When you run firstpass savings, the net figure already accounts for what verification cost you. Proof has a price, and it's on the ticket.

Key idea gate_cost_usd is additive, not amortized. A judge gate on a rung that ultimately escalates still records the judge's token cost on that attempt — even though the model output it checked was discarded. This is the honest accounting: verification cost is owed regardless of whether the output it checked turned out to be good enough. You can't optimize what you can't see.

Gate costs land on the receipt as gate_cost_usd — separate from model spend, always visible in firstpass savings, never hidden.