Firstpass
DocsGuaranteeBenchmarksCLIGitHub

Docs / Coding agents

Coding agents

Claude Code, Codex, Gemini CLI and aider are multi-turn, tool-calling clients. This page covers how Firstpass groups their turns into sessions, how it gates tool calls, what the routing-granularity eval found, and what has and has not been tested.

Point the agent at the proxy. In observe mode nothing changes and every model call gets a receipt (count_tokens metadata calls do not). In enforce mode, tool-call turns are judged by non-empty, json-valid and tool-args-valid, turns are grouped into sessions with no header, and the evidence says to keep a session on one model.

1. Point the agent at the proxy#

AgentHowEndpoint
Claude Codeexport ANTHROPIC_BASE_URL="http://127.0.0.1:8080" or firstpass launch claudePOST /v1/messages
Codexfirstpass launch codex (sets OPENAI_BASE_URL)POST /v1/responses
Gemini CLIfirstpass launch gemini (sets GOOGLE_GEMINI_BASE_URL)POST /v1beta/models/{m}:generateContent
aideraider --openai-api-base=http://127.0.0.1:8080/v1 …POST /v1/chat/completions

Leaving is unsetting the variable. firstpass launch refuses to start the agent if no proxy is listening.

Observe forwards byte-identical and writes receipts. It also relays the upstream's retry-after, request-id, x-request-id, anthropic-ratelimit-* and x-ratelimit-* response headers, so an agent backs off correctly on a 429. POST /v1/messages/count_tokens and POST /v1beta/models/{m}:countTokens are passed through and write no receipt: they carry no routing decision, so no ladder runs.

Gemini credentials — the x-goog-api-key header is relayed to the Gemini upstream only. It is never sent to Anthropic or OpenAI rungs. An escalation ladder mixing Gemini and OpenAI rungs uses the two keys independently.

2. Sessions without a header#

Session promotion (a session that had to escalate starts there next turn) and per-session experiments need to know which requests belong together. Firstpass picks the first of these that is present:

#SourceStored as
1x-firstpass-session headerverbatim, capped at 256 bytes
2Agent session header: x-claude-code-session-id, session_id, conversation_idagent:<hash>
3Claude Code metadata.user_id, when it embeds session_meta:<hash>
4prompt_cache_key, on agent-shaped requestspck:<hash>
5Salted hash of the conversation prefix (system or instructions, declared tool names, first user message)conv:<hash>

Derived ids are salted hashes, so a stored session id never carries raw prompt text or a raw caller id. Header lookups (1 and 2) work on every path. The body-derived ones (3 to 5) are computed only on enforce routes, where the body is already being parsed; an observe-only deployment parses no bodies.

Known ceiling Two conversations with the same system prompt, first user message and tool names share a session. The cost is an early promotion inside the promotion window, never a wrong answer. Send x-firstpass-session to separate them. Plain single-turn requests get no derived session. Source: body_session in crates/firstpass-proxy/src/proxy.rs.

3. Gating tool-call turns#

An agent step often returns a tool call and no text. Read naively that is an empty answer that fails every gate and climbs the ladder on every step. Enforce mode treats it as a tool turn when the request declared tools:

GateOn a tool turn
non-emptyPasses. An undeclared tool call with empty text fails.
json-validPasses. Text beside the call is preamble and is not parsed; the JSON contract applies to the final answer turn.
tool-args-validValidates every call's arguments against input_schema (Anthropic) or parameters (OpenAI), recursively. Fails on a tool the request did not declare, on calls when the request declared no tools, and on any schema violation. Passes when there are no tool calls.

schema and any test, judge or consistency gate still need a [[gate]] block. These built-ins check structure, not correctness: to know the code is right, gate on your tests.

Also in enforce mode: thinking and redacted_thinking blocks and tool_use are streamed back intact, stop_reason is carried across providers, and credentials are scoped. A rung's configured api_key_env is the only key used for that rung (it never falls back to the client's), and a Claude Code OAuth Bearer is accepted on the Anthropic path only.

4. A route for an agent#

toml
[[route]]
match  = {}                                    # every request; agent = "claude-code"/"codex"/"gemini-cli" is detected from User-Agent, others send x-firstpass-agent
mode   = "enforce"
ladder = ["anthropic/claude-haiku-4-5", "anthropic/claude-sonnet-5"]
gates  = ["non-empty", "json-valid", "tool-args-valid"]

[escalation]
max_rungs_per_request = 2
session_promotion = { after_failures = 2, window = "30m" }
bash
FIRSTPASS_CONFIG=./firstpass.toml firstpass doctor   # a wrong key is a hard parse error
FIRSTPASS_MODE=enforce FIRSTPASS_CONFIG=./firstpass.toml firstpass-proxy

firstpass doctor catches wrong config keys and undefined gate ids (a misspelled tool-args-valid fails parse instead of being silently skipped). Anthropic rungs are priced built in; any other provider needs a [[price]] for every rung.

5. Where to switch models: the eval#

On 500 real SWE-bench agent runs, switching models mid-session did not pay on either ladder tested. Choosing a model per workload was far cheaper, but quality non-inferiority at 1 point was not shown.

Replay of public runs, $0, pre-registered. Haiku to opus ladder, n=500, resolved counts against always-top (384/500, $0.754 per instance). Source: agent-granularity.md.

PolicyResolved$ per instanceΔ $ per instance vs always-top
always-cheap333/500$0.331[-0.460, -0.387]
always-top384/500$0.754—
escalate-and-pin (pre-registered cost model)384/500$1.066[+0.309, +0.316]
escalate-and-pin (restart, secondary)384/500$0.838[+0.081, +0.086]
per-session / repo (cross-fit)378/500$0.720[-0.048, -0.021]
run-level cascade (proxy for per-call)382/500$1.071[+0.295, +0.340]
Practice this implies Route per workload or route with certified selection, keep a session sticky (that is what session promotion does), and do not switch models mid-session.

Limits stated by the source: one scaffold (mini-SWE-agent, bash only), one benchmark (SWE-bench Verified, 500 instances), one run per model per instance, and a context model calibrated on 20 cheap-model trajectories, so the overhead figures are estimates. Certified selection also works across replicas: with a shared Redis log, paired mode's spend cap is cluster-wide and log errors fail closed (details).

6. What has been tested#

Generated by scripts/agent-compat/run_compat.sh against firstpass 0.9.2 in observe mode, with a fake local upstream and no real API keys. Source: agent-compat.md.

CLIResultNotes
Claude Code 2.1.xpassTurn completed through /v1/messages.
aiderpassTurn completed through /v1/chat/completions.
Codexpass, fixture onlyA /v1/responses request returned 200. The run had a shell alias, not the Codex binary.
Gemini CLI 0.26.0pass, fixture onlyPOST /v1beta/models/{m}:generateContent returned 200. Tested via firstpass launch gemini against a fake local upstream. The Gemini binary was not available in CI.
clineskipNeeds interactive auth setup. Its /v1/chat/completions protocol is served.
cursor-agentskipBound to Cursor cloud OAuth.

Limits#

Point the agent at the proxy, gate tool turns with tool-args-valid, let sessions group themselves, and route per workload rather than per step. Claude Code, aider, and the Gemini CLI inbound protocol are supported; Codex and Gemini CLI have not been run as binaries through the harness. On Gemini 2.5+/3 with tool use, turns carrying thoughtSignature are observe passthrough — text-only turns and pre-2.5 models are gated normally.