Experiments

An Experiment compares variants for a set of rings, under a primary metric and guardrails, with a stopping rule.

Type and axis

	client axis	traffic axis
Variants differ by	profile / release on the machine	model route at the gateway
Change takes effect	on rebuild / next halod pull (channels)	immediately at the gateway (policy hot-reload)
Types	ab, canary	ab, canary, shadow
Example	CLI 2.1.280 vs 2.1.290	opus alias to a newer model

Types: ab splits by weight and compares; canary sends a small share to the candidate and aborts on guardrail breach; shadow mirrors and does not affect responses (shadow traffic).

Traffic first: ADR-0003.

Assignment

• Deterministic: hash(user, salt) % 10000 mapped over normalized variant weights. salt defaults to the experiment name so experiments are independent.
• Sticky per user. Because the variant is a function of the user, an agent session never switches model mid-task. Shadow sampling is keyed on the session id the harness sends (for example Claude Code’s x-claude-code-session-id), so a whole session is either mirrored or not.
• The cohort comes from the identity the gateway verified (OIDC JWT), never from a client header. A request with no verified identity is refused with 401; only with anonymous access enabled (allow_unverified / the proxy’s allowAnonymous) does it get default routing and no experiments.
• At most one A/B or canary applies to a request (the first matching running experiment); any number of running shadow experiments can sample it.

Client-axis delivery

A client-axis variant is a release, not a route, so it has to reach the machine through the same signed path as any release. halo release publish --ring R builds the ring release plus one release per variant of the running client-axis experiment that enrolls R, and serves each on its own channel.

• Channel name: <ring>.x-<experiment>.<variant>. If that exceeds 100 characters, the experiment/variant part becomes the first 20 hex characters of sha256(experiment NUL variant) (<ring>.x-<hash>). A channel is published like a ring: a signed pointer at ring-<channel>.pointer.
• The ring manifest lists the experiment. Its experiments section (name, salt, per-variant weight and channel) is part of the release the signature covers. halod recomputes each channel name and refuses a manifest that names a different one.
• Same function as the gateway. halod computes the variant with policy.Experiment.ResolveVariant, the function halo-proxy uses, from the signed salt and weights. The device and the gateway agree on the variant without talking to each other.
• One running client-axis experiment per ring. A device applies one release, so two overlapping experiments cannot both deliver. Validation rejects the overlap.
• An older halod ignores the experiments section and applies the ring release, which is control.

Subject

halod needs the user id the gateway hashes. It takes the first that is available:

1. subject in the ring endpoint response (GET /api/v1/fleet/ring returns {"ring", "subject"}, the enrolled user).
2. subject: in halod.yaml (for MDM: template the user id into the file). 1 to 256 characters, no control characters.
3. The last subject halod saved.

With no subject, halod applies the ring release, reports errorCode: no_subject_for_experiment and exits non-zero from halod once. It never guesses a variant.

Failure behaviour

• A channel that fails verification keeps last-good. Bad signature, wrong org, replayed seq, expired pointer, or a release for a different experiment/variant fails the whole cycle: keeping last-good <digest>. halod does not fall back to the ring release, because that would silently move treatment devices to control.
• Each channel has its own anti-rollback mark in the halod state file, like a ring.
• Pause and republish to revert. When the experiment is no longer running, the next halo release publish writes a ring release with no experiments and no channels. Devices converge to it on their next pull.

Variant guardrails

A variant profile is held to the ring’s profile more strictly than a child is to its parent. These are errors (halo validate exit 2), not warnings:

Rule	Error when
Exact version pins	a harness version is not exact semver (no ranges, no latest), or no harness is enabled
Telemetry	telemetry.enabled is false
permissions.disableBypass	not true
No widening	the variant adds permissions.allow, permissions.ask or egress.allowedDomains entries the ring profile lacks, or drops a ring permissions.deny or mcp.denied entry
No MCP changes	mcp.servers adds or changes a server the ring does not have
No weakening	sandboxRequired, mcp.managedOnly, hooks.managedOnly or models.enforce is true on the ring and not on the variant; permissions.sandbox is relaxed (read-only to workspace-write, or any sandbox to none/off)
Mode ordering	permissions.mode is more permissive than the ring’s, ranked plan < default < acceptEdits < auto (empty means default; an unknown mode ranks above auto)
Hooks are a subset	the variant has a hook (command and event) the ring profile lacks
Telemetry unchanged	telemetry.otlpEndpoint or telemetry.logPrompts differs from the ring’s, so a variant cannot redirect or start exporting evidence or prompts
Models are a subset	when the ring restricts models.allowed: the variant’s list is empty, adds a model, or its models.default is outside the ring’s list
Instructions and env equal	instructions or env differs from the ring profile’s
Unique channels	a channel name collides with a ring or another channel

Attribution

The variant release carries the experiment and variant in its harness config. Claude Code gets halo.experiment and halo.variant in OTEL_RESOURCE_ATTRIBUTES (next to halo.ring and halo.release). Codex, Gemini CLI and Copilot CLI have no resource-attribute config key, and Halos renders no telemetry for Cursor CLI or OpenCode. On Linux, with telemetry enabled, Halos writes an /etc/profile.d shell function that runs the CLI with OTEL_RESOURCE_ATTRIBUTES including halo.experiment and halo.variant; it reaches login shells only, so an exec that bypasses the shell (timeout, env, xargs) runs the CLI unlabeled. On macOS and Windows Halos cannot set it, and halo render and halo release publish warn that CLI telemetry carries no halo.* attributes.

The gateway attributes a request (x-halo-experiment, x-halo-variant) to a client-axis experiment only when no traffic-axis experiment applies to it, so a client-axis experiment never displaces a traffic experiment’s routing. Client-axis experiments never route. halod status and fleet reports carry experiment and variant.

Walkthrough: A/B a CLI upgrade. Design: ADR-0010.

Metrics and stopping

metrics:
  primary: {metric: halo.edit.accept_rate, direction: increase}
  guardrails:
    - {metric: halo.cost.usd_per_session, direction: decrease, maxRegression: 0.10}
stopping: {method: msprt, alpha: 0.05, minSamples: 300, maxDays: 14, maxSpendUSD: 500}

• This snippet is for a client-axis experiment. A traffic-axis experiment must use gateway-sourced metrics (halo.api.error_rate, halo.latency.p50_ms, halo.latency.p95_ms); halo.task.success is offline-eval only.
• msprt: mixture sequential probability ratio test, valid under continuous monitoring (no peeking penalty). fixed: fixed horizon.
• Guardrail maxRegression is relative worsening that aborts the experiment (0.10 = 10%). When the control’s value is 0 (an error rate with no control errors), relative worsening is unbounded: any significant increase fails the guardrail, and two arms that are both exactly 0 pass it.
• maxDays and maxSpendUSD are hard ceilings so a non-converging experiment stops spending.
• Analysis: mSPRT for sequential decisions and bootstrap confidence intervals (internal/stats), with CUPED variance reduction available in the stats package.

Lifecycle

halo exp start|pause|conclude <name> edit the status field of the experiment YAML in place (use --policy-dir for the policy repo). Only running experiments affect traffic.

halo exp list and halo exp show <name> read them. The status change is a git change like any other: commit it, review it, merge it. The gateway reloads the compiled policy (halo gateway compile) on its next snapshot.

Verdicts

halo exp analyze opus-5-5-canary --policy-dir policy --clickhouse http://clickhouse:8123 \
  --verdicts-file verdicts.json

The verdict is one of promote, rollback, continue or expired (a maxDays or maxSpendUSD ceiling was hit without a decision). --verdicts-file upserts the result into a JSON file that halo-server serves (--verdicts-file). ClickHouse credentials come from --user and the HALO_CLICKHOUSE_PASSWORD environment variable.

A promote or rollback report carries source: gateway or cli, the evidence plane the deciding metric came through (halo exp analyze --output json shows it, and so does each metric). gateway is exact cohort attribution measured by halo-proxy and delivered over the authenticated gateway receiver; cli is telemetry from developer machines, which anyone who can reach the collector can forge. Only gateway evidence can trip the kill switch (evidence trust). Sources are never mixed within one analysis: a metric uses gateway rows when any exist, and falls back to CLI events otherwise.

halo exp promote <name> --ring <ring> --release <digest> requires a promote verdict and opens a PR that points the ring at the release (--dry-run prints the patch). It never merges.

The controller loop

halo exp analyze is one evaluation you run by hand. The controller runs it for every status: running experiment on a timer and acts on the result. It is built into halo-server (--controller) and also available as a one-shot for CI or cron (halo controller run --once).

Verdict	Controller action	Notes
rollback	If the deciding evidence is gateway-sourced and a kill list is served, trips the kill switch at once; always opens a PR setting status: paused with the evidence in the body and notifies	The PR is never merged. On CLI-sourced evidence nothing is auto-killed (killOutcome: not_gateway_evidence): a human decides
promote	Opens a PR that sets status: concluded, notifies	It does not roll the treatment out: before merging, a human adds the ring release (client axis) or model route (traffic axis). Concluding returns everyone to control
expired	Opens a PR that sets status: concluded, notifies	The PR is never merged
continue	Nothing

Rules that matter in operation:

• Each action happens once per experiment run. The record is controller-state.jsonl in the data dir. When the experiment is next seen not running (someone merged the pause or conclusion), the record resets, so a restarted experiment starts clean. Ticks never duplicate PRs or notifications.
• Kill first, but only on trusted evidence and only if something enforces it. On rollback the kill happens before the PR, and the notification goes out even if the kill or the PR failed, because people must know. The webhook event’s killOutcome says what happened: enforced, recorded_only (written, enforcement not guaranteed), not_configured, not_gateway_evidence, or failed. The in-process controller kills only when halo-server has a kill key (-killswitch-key-file); without it nothing reads the list, so it does not pretend: the PR and notification say merge urgently. halo controller run always writes the kill to <data-dir>/killswitch.jsonl, and says it is enforced only with --killswitch-served.
• No evaluation after a rollback, or while killed. Once a run rolled back, the controller only finishes that rollback’s pending PR/notifications; a later promote verdict cannot conclude it. A running experiment found killed (admin kill, or a kill left from an earlier run) is not evaluated (its evidence is all-control); humans are told once per run to unkill before resuming (webhook verdict killed).
• Notifications are tracked per channel (notify:slack, notify:webhook): a failing channel is retried on later ticks alone, the others are not re-posted. Each delivery has a 10-second timeout and does not follow redirects.
• The controller never merges. Promotion and rollback PRs wait for a human. Automatic rollback (the kill switch) is the only automatic action that changes live traffic.
• Errors retry sooner than the interval: 15s, 30s, and so on, capped at the tick interval. Each tick is bounded to 2 minutes. The interval is at least 1 minute (default 5).
• Run one controller per data dir. Two controllers would race on the action log.
• Notifications go to a Slack incoming webhook and/or a generic webhook. Generic webhook bodies are signed and timestamped: X-Halo-Timestamp: <unix seconds> and X-Halo-Signature: sha256=<hex HMAC-SHA256(secret, timestamp + "." + body)>, with the secret from --notify-webhook-secret-file. Receivers must verify the signature and reject timestamps more than 5 minutes off; verification snippets. Webhook URLs are secrets: read from files, never logged.
• Audit is fail-closed for privileged actions. A controller kill is appended to the hash-chained audit.jsonl (actor halo-controller) with fsync. If the append fails the kill stays applied (fail safe) and the tick reports the audit gap as an error. halo controller run appends through the same chain, so it is a single writer: do not point it at a data dir a live halo-server is appending to; use the server’s in-process controller there.

Running it

Inside halo-server (the usual way; kills reach gateways through the same data dir):

halo-server -policy-dir /srv/policy -token-file fleet.token -data-dir /var/lib/halos \
  -controller -clickhouse-url http://clickhouse:8123 -clickhouse-user halo -clickhouse-password-file ch.pass \
  -policy-repo-dir /srv/policy-writer -policy-repo-base main \
  -killswitch-key-file killswitch.key -gateway-token-file gateway.token \
  -notify-slack-url-file slack.url -metrics-listen 127.0.0.1:9100

--controller requires -clickhouse-url and -data-dir. -policy-repo-dir must be a clone separate from -policy-dir; it may be the portal’s policyRepoDir (one clone and lock for console proposals and controller PRs). It is hard-reset to origin/<base> before each PR. Without it, the controller still evaluates, kills and notifies but opens no PRs. PRs need git and an authenticated gh.

As a one-shot (CI, cron):

HALO_CLICKHOUSE_PASSWORD=... halo controller run --once --policy-dir . \
  --clickhouse http://clickhouse:8123 --user halo --data-dir /var/lib/halos

Check: exit code is non-zero if any experiment failed. PRs are opened from the git checkout that contains --policy-dir. Kills reach gateways only when --data-dir is the same directory halo-server uses as -data-dir (a shared volume); otherwise the kill is recorded but nothing serves it. Pass --killswitch-served only when that is true: without it, rollback PRs and notifications say the kill is merely recorded. Use --no-pr for verdicts, kills and notifications only. Flags: CLI reference.

Kill switch

Merging a pause PR takes a review cycle plus a policy snapshot roll-out. The kill switch is the fast path for a bad treatment: a signed list of killed experiment names that gateways poll from halo-server. A killed experiment is treated as not running: its users get control routing and its shadow mirroring stops, at the gateway’s next poll (default every 10 seconds), with no PR and no merge.

An admin can kill or unkill by hand (POST /api/v1/experiments/{name}/kill and /unkill; the console shows a Kill switch button when the server has one). Setup and the wire format: API reference, security model, ADR-0009.

Failure policy

Situation	What gateways do
halo-server unreachable, or the list fails verification	Keep the last accepted list. A kill never lapses because the control plane is down
The gateway has never fetched a valid list	Kill nothing. The policy snapshot’s status fields alone decide
A list is older than 10 minutes or more than 1 minute in the future	Rejected (last list stays)
A list is not strictly newer (issuedAt) than the current one	Rejected as a replay, so an old envelope cannot undo a kill

Kills do not expire. Pausing or concluding the experiment in policy does not lift a kill. Unkill it explicitly before you restart the experiment, or it stays on control:

curl -X POST https://halo.acme.example/api/v1/experiments/opus-5-5-canary/unkill \
  -H 'content-type: application/json' -d '{"reason":"fixed; restarting"}'   # admin session required

Limits: the client axis

The kill switch lives at the gateway, so it fully undoes a traffic-axis experiment (model routes). A client-axis experiment changes the release on the machine. Killing it always stops gateway-side effects (routing, mirroring and attribution). Whether it reaches machines depends on halod:

• killSwitch configured in halod.yaml: halod polls the same signed list (GET /api/v1/fleet/killswitch, device token, default every 60 seconds) and treats a killed experiment as not running. The device applies the ring release (control) within one poll, without a republish, and reports killed: true. Unkill returns it to its variant. The list is verified, persisted and replay-protected like the gateway’s; a failed fetch keeps the last list. Config: halod.yaml.
• Not configured (the default for hand-written and MDM configs; portal enrollment adds it whenever halo-server has a kill key): devices keep their variant until the experiment is paused or concluded in policy and halo release publish --ring <R> republishes, or halo rollback points the ring and its channels at an earlier release. See client-axis delivery.

The controller’s notification text (“gateway traffic only; client-axis variants stay until the pause PR merges”) describes the second case.

Killed experiments also get no x-halo-experiment or x-halo-variant stamp (gateway headers).