Skip to content
docs

Models and routing

Agents ask for a capability such as reasoning-large, not a vendor model. The control plane maps it to a concrete Anthropic or OpenAI model from the providers you have keyed. You swap a model, or ride out a provider outage, with no agent code change, and a cheap agent never fails over into an expensive model.

Model routing and failoverAgents ask for a capability, not a model. Pick a scenario to see the default direct path, the router path, and how errors fall through.
Agent or appmodel: "auto"CheckBudgetagent + tenant capsControl planecallLLMWithFailoverModel router:50053Cacheprompt + semanticProvider Ae.g. AnthropicProvider Be.g. OpenAIRecordUsageagent_usage_daily
Swipe sideways to pan.

1. Ask by capability

Service code never names a vendor model.

Providers and keys

Section titled “Providers and keys”

Two providers today:

  • Anthropic: Opus, Sonnet, Haiku (defaults claude-opus-4-8, claude-sonnet-4-6, claude-haiku-4-5-20251001; override with PAL_OPUS_MODEL, PAL_SONNET_MODEL, PAL_HAIKU_MODEL)
  • OpenAI: gpt-4o and gpt-4o-mini

Add a key under Settings > Model providers or over the API:

POST /v1/settings/llm-providers
{ "provider": "anthropic", "apiKey": "sk-ant-..." }

Keys are AES-256-GCM encrypted at rest when PAL_CREDENTIAL_KEY is set (plaintext otherwise, so set it outside dev). They never appear in logs, traces or run state. The control plane resolves them per tenant at call time. Scored picks (auto, cheap, best) only consider providers you have keyed.

Capability names

Section titled “Capability names”

Use these strings in model fields. Resolution happens in the control plane (llm_proxy.go). Fixed names are pinned to one provider; the rest pick by which providers are keyed.

Name Resolves to
reasoning-frontier Opus (PAL_OPUS_MODEL), always Anthropic
reasoning-large, code-large Sonnet, always Anthropic
reasoning-small Haiku, always Anthropic
chat-large, vision-large gpt-4o, always OpenAI
chat-small, vision-small gpt-4o-mini, always OpenAI
chat-edge, code-small, cheap, fast Haiku if Anthropic is keyed, else gpt-4o-mini
best Opus if Anthropic is keyed, else gpt-4o
auto, balanced, empty string Scored pick among keyed providers (How auto works)
embed-large, embed-small, rerank, transcribe, tts Not servable by text completion; routed to the same scored pick as auto

“Anthropic keyed” means the tenant has an active Anthropic key or the process has ANTHROPIC_API_KEY. Always-pinned names do not check which providers you keyed, so a pinned name whose provider has no key is left to failover.

  • Unknown names route to the balanced default, same as auto. The control plane logs one WARN per distinct name (capped at 256) and never routes an unknown to a hardcoded model.
  • Literal vendor names pass through unchanged: gpt*, o1*, o3* go to OpenAI, claude* to Anthropic. The match is a plain prefix and the name is not validated; a wrong id fails at the provider. Prefer a capability.
  • fast and cheap both map to the cheap tier. Neither optimizes for latency here; use PAL_ROUTE_STRATEGY=fast for that.
  • Which callers use this table: workflow ai-step nodes. Session turns use complexity routing by default (each turn classified trivial, balanced or hard); set PAL_COMPLEXITY_ROUTING=0 for plain auto scoring.

How auto works

Section titled “How auto works”

auto scores each model in a built-in catalog (Opus, Sonnet, Haiku, gpt-4o, gpt-4o-mini) on quality, speed and cost and picks the top scorer among keyed providers. With neither provider keyed it returns gpt-4o and the call fails with a missing-key error.

PAL_ROUTE_STRATEGY changes the weighting for every name that resolves to the scored pick:

Value Picks
balanced (default) Quality first, price as a real penalty: Sonnet, else gpt-4o. Pin reasoning-frontier or best if you want Opus
cheap Haiku or gpt-4o-mini
quality Opus, else gpt-4o
fast Haiku, else gpt-4o-mini

Per-agent model

Section titled “Per-agent model”

Runs, sessions and workflow ai-step nodes with no capability of their own try the agent’s model first.

POST  /v1/agents          { "name": "triage", "model": "fast" }
PATCH /v1/agents/triage   { "model": "reasoning-large" }    # "" or "auto" restores the default chain
GET   /v1/agents/triage   →  { ..., "model": "reasoning-large" }
Allowed value Examples
Default "", auto
Cost or speed target fast, cheap, balanced, best
Capability reasoning-frontier, reasoning-large, reasoning-small, chat-large, chat-small, chat-edge, vision-large, vision-small, code-large, code-small
Vendor model id gpt-*, o1*, o3*, claude-* (lowercase, 100 characters or fewer)

Anything else returns 400, including embedding, rerank and speech capabilities; the error never echoes the value. If the agent’s provider is not keyed, the agent uses the default chain. An ai-step with its own capability keeps it; only auto nodes take the agent’s model. The model that answered is recorded on the run’s llm:main step.

Failover

Section titled “Failover”

The control plane fails over through a ranked chain: each keyed provider’s top model, then that provider’s cheaper tier (a separate quota tier), then the next provider.

  • 429, 402 and 5xx move to the next candidate; a rate limit adds a backoff first.
  • Agent models go first. An agent’s model heads the chain. A cheap-tier agent (Haiku or *-mini) fails over only to the other provider’s cheap tier, never to a Sonnet, gpt-4o or Opus class model. If both cheap tiers fail, the call fails instead of escalating the bill.
  • No mid-sentence swap. In streaming, failover happens only while zero tokens have been emitted. After the first token an error surfaces as message_error, never a splice (Streaming).

OpenAI and Anthropic calls carry an Idempotency-Key header: a one-way hash of run, step, attempt and the request body, with a per-provider suffix on failover targets. A backoff retry or crash replay dedups at the provider instead of double-billing. Ad-hoc /v1/completions calls without a run use a hash of provider, model and messages (llm_idempotency.go).

Model-router service

Section titled “Model-router service”

A separate Rust model-router service exists (gRPC :50053, address PAL_MODEL_ROUTER_ADDR, default model-router:50053). The control plane routes plain completions through it only when PAL_USE_MODEL_ROUTER=1 (also true or on). It is off by default, and any router error falls back to the direct path. Enable it on a non-critical tenant first.

The router has its own mapping. The control plane resolves a model with the table above, converts it back to a router capability, and the router picks the concrete model. A provider supports a capability only if it lists it exactly:

Router capability Anthropic OpenAI
reasoning-frontier claude-opus-4-6 not supported
reasoning-large claude-sonnet-4-6 o3
reasoning-small claude-haiku-4-5 not supported
code-large claude-sonnet-4-6 gpt-4o
chat-large not supported gpt-4o
chat-small not supported gpt-4o-mini
embed-large, embed-small not supported text-embedding-3-large, text-embedding-3-small

auto and unspecified become chat-large. Among providers that support the capability, the optimize target orders them: cheap by price, fast by p50 latency, best by quality score, balanced by a fixed provider preference.