Docs / Coding agents
Coding agents
Claude Code, Codex, Gemini CLI and aider are multi-turn, tool-calling clients. This page covers how Firstpass groups their turns into sessions, how it gates tool calls, what the routing-granularity eval found, and what has and has not been tested.
count_tokens metadata calls do not). In enforce mode, tool-call turns are judged by non-empty, json-valid and tool-args-valid, turns are grouped into sessions with no header, and the evidence says to keep a session on one model.1. Point the agent at the proxy#
| Agent | How | Endpoint |
|---|---|---|
| Claude Code | export ANTHROPIC_BASE_URL="http://127.0.0.1:8080" or firstpass launch claude | POST /v1/messages |
| Codex | firstpass launch codex (sets OPENAI_BASE_URL) | POST /v1/responses |
| Gemini CLI | firstpass launch gemini (sets GOOGLE_GEMINI_BASE_URL) | POST /v1beta/models/{m}:generateContent |
| aider | aider --openai-api-base=http://127.0.0.1:8080/v1 … | POST /v1/chat/completions |
Leaving is unsetting the variable. firstpass launch refuses to start the agent if no proxy is listening.
Observe forwards byte-identical and writes receipts. It also relays the upstream's retry-after, request-id, x-request-id, anthropic-ratelimit-* and x-ratelimit-* response headers, so an agent backs off correctly on a 429. POST /v1/messages/count_tokens and POST /v1beta/models/{m}:countTokens are passed through and write no receipt: they carry no routing decision, so no ladder runs.
Gemini credentials — the x-goog-api-key header is relayed to the Gemini upstream only. It is never sent to Anthropic or OpenAI rungs. An escalation ladder mixing Gemini and OpenAI rungs uses the two keys independently.
2. Sessions without a header#
Session promotion (a session that had to escalate starts there next turn) and per-session experiments need to know which requests belong together. Firstpass picks the first of these that is present:
| # | Source | Stored as |
|---|---|---|
| 1 | x-firstpass-session header | verbatim, capped at 256 bytes |
| 2 | Agent session header: x-claude-code-session-id, session_id, conversation_id | agent:<hash> |
| 3 | Claude Code metadata.user_id, when it embeds session_ | meta:<hash> |
| 4 | prompt_cache_key, on agent-shaped requests | pck:<hash> |
| 5 | Salted hash of the conversation prefix (system or instructions, declared tool names, first user message) | conv:<hash> |
Derived ids are salted hashes, so a stored session id never carries raw prompt text or a raw caller id. Header lookups (1 and 2) work on every path. The body-derived ones (3 to 5) are computed only on enforce routes, where the body is already being parsed; an observe-only deployment parses no bodies.
x-firstpass-session to separate them. Plain single-turn requests get no derived session. Source: body_session in crates/firstpass-proxy/src/proxy.rs.3. Gating tool-call turns#
An agent step often returns a tool call and no text. Read naively that is an empty answer that fails every gate and climbs the ladder on every step. Enforce mode treats it as a tool turn when the request declared tools:
| Gate | On a tool turn |
|---|---|
non-empty | Passes. An undeclared tool call with empty text fails. |
json-valid | Passes. Text beside the call is preamble and is not parsed; the JSON contract applies to the final answer turn. |
tool-args-valid | Validates every call's arguments against input_schema (Anthropic) or parameters (OpenAI), recursively. Fails on a tool the request did not declare, on calls when the request declared no tools, and on any schema violation. Passes when there are no tool calls. |
schema and any test, judge or consistency gate still need a [[gate]] block. These built-ins check structure, not correctness: to know the code is right, gate on your tests.
Also in enforce mode: thinking and redacted_thinking blocks and tool_use are streamed back intact, stop_reason is carried across providers, and credentials are scoped. A rung's configured api_key_env is the only key used for that rung (it never falls back to the client's), and a Claude Code OAuth Bearer is accepted on the Anthropic path only.
4. A route for an agent#
[[route]]
match = {} # every request; agent = "claude-code"/"codex"/"gemini-cli" is detected from User-Agent, others send x-firstpass-agent
mode = "enforce"
ladder = ["anthropic/claude-haiku-4-5", "anthropic/claude-sonnet-5"]
gates = ["non-empty", "json-valid", "tool-args-valid"]
[escalation]
max_rungs_per_request = 2
session_promotion = { after_failures = 2, window = "30m" }FIRSTPASS_CONFIG=./firstpass.toml firstpass doctor # a wrong key is a hard parse error
FIRSTPASS_MODE=enforce FIRSTPASS_CONFIG=./firstpass.toml firstpass-proxyfirstpass doctor catches wrong config keys and undefined gate ids (a misspelled tool-args-valid fails parse instead of being silently skipped). Anthropic rungs are priced built in; any other provider needs a [[price]] for every rung.
5. Where to switch models: the eval#
Replay of public runs, $0, pre-registered. Haiku to opus ladder, n=500, resolved counts against always-top (384/500, $0.754 per instance). Source: agent-granularity.md.
| Policy | Resolved | $ per instance | Δ $ per instance vs always-top |
|---|---|---|---|
| always-cheap | 333/500 | $0.331 | [-0.460, -0.387] |
| always-top | 384/500 | $0.754 | — |
| escalate-and-pin (pre-registered cost model) | 384/500 | $1.066 | [+0.309, +0.316] |
| escalate-and-pin (restart, secondary) | 384/500 | $0.838 | [+0.081, +0.086] |
| per-session / repo (cross-fit) | 378/500 | $0.720 | [-0.048, -0.021] |
| run-level cascade (proxy for per-call) | 382/500 | $1.071 | [+0.295, +0.340] |
- Mid-session escalate-and-pin costs more than always-top at equal quality on both ladders (gpt-5-mini to gpt-5.2: +$0.052 to +$0.054 per instance). With prompt caching, the top model's first call re-reads the accumulated context at the full input rate.
- Per-session routing on a cheap/top ladder loses quality against always-top: the lower bound of the resolved-rate difference is -0.024 (haiku to opus) and -0.042 (gpt-5-mini to gpt-5.2), below the pre-registered -0.01.
- The run-level cascade costs more. It is a proxy: per-call cascade is not identifiable from recorded whole runs, because there are no per-step gate verdicts.
- Workload-level pool selection resolved 374/500 at $0.219 per instance against 384/500 at $0.754 (Δ $ per instance [-0.582, -0.490]; Δ resolved [-0.050, +0.008]). Non-inferiority at 1 point is not shown.
- Exploratory, not a result: a per-repo pool selection arm resolved 382/500 at $0.122 per instance. It was added after the pre-registration and is labelled exploratory here. It needs per-repo history and was not tested on unseen repos.
Limits stated by the source: one scaffold (mini-SWE-agent, bash only), one benchmark (SWE-bench Verified, 500 instances), one run per model per instance, and a context model calibrated on 20 cheap-model trajectories, so the overhead figures are estimates. Certified selection also works across replicas: with a shared Redis log, paired mode's spend cap is cluster-wide and log errors fail closed (details).
6. What has been tested#
Generated by scripts/agent-compat/run_compat.sh against firstpass 0.9.2 in observe mode, with a fake local upstream and no real API keys. Source: agent-compat.md.
| CLI | Result | Notes |
|---|---|---|
| Claude Code 2.1.x | pass | Turn completed through /v1/messages. |
| aider | pass | Turn completed through /v1/chat/completions. |
| Codex | pass, fixture only | A /v1/responses request returned 200. The run had a shell alias, not the Codex binary. |
| Gemini CLI 0.26.0 | pass, fixture only | POST /v1beta/models/{m}:generateContent returned 200. Tested via firstpass launch gemini against a fake local upstream. The Gemini binary was not available in CI. |
| cline | skip | Needs interactive auth setup. Its /v1/chat/completions protocol is served. |
| cursor-agent | skip | Bound to Cursor cloud OAuth. |
Limits#
- The compatibility run was observe mode only. Enforce behaviour for tool turns is covered by the proxy's own tests, not by driving the real CLIs through the harness.
- Codex was not run as a binary. Only the Responses API request shape was exercised.
- Gemini CLI was not run as a binary. The
/v1beta/models/{m}:generateContentinbound protocol is served (observe and enforce, streaming and non-streaming), and the e2e test exercises it via a fake local upstream. The Gemini binary itself was not driven through the harness. - The structural gates do not prove correctness.
tool-args-validproves a call matches its declared schema, not that it is the right call. - Gemini inbound untranslatable content falls through to observe passthrough. Binary parts (
inlineData),executableCode,fileData, and any part type not in the allow-list cannot be gated — they are relayed byte-identical without running the ladder. - Gemini 2.5+/3 with tool use: most turns are observe passthrough, not gated. These models return a
thoughtSignaturefield onfunctionCallparts. The signature is an opaque token required for extended-thinking replay; Chat Completions has no equivalent, so the translation would silently drop it. The proxy's allow-list therefore treats any part carrying bothfunctionCallandthoughtSignatureas untranslatable and sends it to observe passthrough. gemini-cli replaysthoughtSignaturein every subsequent turn's history after the first tool call, so for these models most agentic turns with tool use will not pass through the gate. Text-only turns and pre-2.5 models withoutthoughtSignatureare gated normally. Verified against the gemini-cli source (packages/core/src/core/contentGenerator.ts): history is replayed verbatim including function-call parts with all their fields. - Gemini-provider ladders cannot enforce tool turns. A ladder whose rungs resolve to a
GeminiProviderdoes not carry OpenAI-dialect structured content verbatim (enforce_can_handle_routereturnsfalsefor Gemini-provider routes when the request contains tool calls ortool_call/toolmessages). Such routes fall back to observe passthrough on tool-call turns. Text-only requests are enforced normally. If you need tool-call enforcement on a Gemini-provider rung, raise an issue — the translation layer would need to be extended to carry tool schemas and results across the Chat ↔ Gemini wire format boundary.
tool-args-valid, let sessions group themselves, and route per workload rather than per step. Claude Code, aider, and the Gemini CLI inbound protocol are supported; Codex and Gemini CLI have not been run as binaries through the harness. On Gemini 2.5+/3 with tool use, turns carrying thoughtSignature are observe passthrough — text-only turns and pre-2.5 models are gated normally.