Canary a model upgrade

Axis: traffic. Types: shadow, then canary. No client change; clients keep requesting the alias. The example repo ships sonnet-next-shadow.yaml and opus-5-5-canary.yaml under examples/acme-corp/experiments/.

1. Shadow

Use the shadow config. Remember the limit: this checks first-turn behavior only (ADR-0004).

2. Replay evals

Run the eval suite against the candidate route (evals/suites/model-upgrade.yaml compares sonnet with sonnet-next on the same tasks). This is the multi-step signal shadow cannot give. See writing evals.

halo eval run evals/suites/model-upgrade.yaml --output json > scorecard.json

3. Canary

apiVersion: halos.dev/v1
kind: Experiment
name: opus-5-5-canary
type: canary
axis: traffic
status: running
rings: [ring1-canary]
variants:
  - {name: control, weight: 95, control: true}
  - name: opus-5-5
    weight: 5
    routes:
      opus: {upstream: orchestrator, model: us.anthropic.claude-opus-5-5-v1:0}
metrics:
  primary: {metric: halo.api.error_rate, direction: decrease}
  guardrails:
    - {metric: halo.latency.p50_ms, direction: decrease, maxRegression: 0.10}
    - {metric: halo.latency.p95_ms, direction: decrease, maxRegression: 0.15}
stopping: {method: msprt, alpha: 0.05, minSamples: 500, maxDays: 14, maxSpendUSD: 2000}

Start it with halo exp start opus-5-5-canary --policy-dir policy-repo (edits status: running), commit and merge, then halo gateway compile --policy-dir policy-repo -o policy.json so the gateway picks it up.

Online metrics on the traffic axis come from the gateway, so task success is not one of them: halo validate rejects halo.task.success here. Judge it offline with the replay evals above. Assignment is sticky per user: an agent never changes model mid-task, and the variant comes from the verified identity, not a header. halo gateway routes --user U --session S shows which route a user would hit. Walkthrough: Canary a new model.

4. Conclude or roll back

halo exp analyze opus-5-5-canary --policy-dir policy-repo --clickhouse http://clickhouse:8123
halo exp promote opus-5-5-canary --policy-dir policy-repo --clickhouse http://clickhouse:8123 \
  --ring ring1-canary --release sha256:... --dry-run

halo exp promote opens a PR (human merges); for a traffic change that PR changes the alias route in the Gateway document.

Rollback at any point, without touching a client: halo exp pause opus-5-5-canary --policy-dir policy-repo, commit, and recompile the snapshot. halo-proxy and halo-kong hot-reload the compiled policy (about a second), so users return to the control route. This is only as fast as your policy pipeline: keep an on-call path that can compile and deploy the snapshot without a full review. The faster path needs no pipeline: halo kill opus-5-5-canary --reason "..." trips the signed kill list (walkthrough).

Timeline