Evals and the measure loop

Evals answer one question before a change reaches a ring: is the candidate at least as good as what runs today? halo eval run replays real tasks in containers, grades each trial, and returns a verdict: ship, hold or block. halo eval online keeps asking that question after rollout by grading shadow pairs.

A change is evaluated offline, exposed to a ring, measured through OTEL in ClickHouse and judged by mSPRT. A pass opens a promotion PR a human merges; a breach rolls back automatically and fires the signed kill switch.

What gets measured

pass@1, pass@k, pass^k. Each task runs repeats times. pass@k is the unbiased per-task chance that any of k trials passes, so it measures capability. pass^k is the chance that all k pass, so it measures reliability. Both are averaged over tasks.
Cost, tokens, wall time, tool calls, per task, as the harness reports them.
Graders run on every trial: the task’s own check, plus suite-wide diff limits and judge rubrics.

The judge

A pinned model (exact id, no latest aliases) reached through your gateway at temperature 0. It grades against a versioned rubric (id@version is recorded in the scorecard). The reply must be one strict JSON object with a score for every criterion and a non-empty rationale. Anything else is a grader error: never a pass, and it forces the gate to hold. Verdicts are cached by rubric, model and input hash, so reruns don’t pay twice.

The gate: ship, hold, block

The candidate is compared with the control paired by task, using bootstrap 95% CIs on the delta of pass rate, cost, tokens, wall time and tool calls.

Verdict	When
ship	Pass-rate CI lower bound ≥ −`max_pass_drop`, no regressed task over the limit, every set cost/latency limit inside its CI
hold	Inconclusive: a CI straddles a margin, too few non-flaky tasks (`min_tasks`), or any grader error
block	Pass rate significantly worse, more regressed tasks than `max_regressed_tasks`, or a cost/latency limit confidently exceeded

Flake detection. A task whose baseline passes some trials and fails others is flagged flaky and left out of every gated statistic, with the reason in the scorecard. A task that flips on unchanged code can’t attribute a flip to the candidate.

Example suite

# evals/suites/upgrade-gate.yaml (excerpt)
name: upgrade-gate
tasks: [fix-failing-go-test, add-cli-flag, refactor-rename]
control: current
repeats: 5          # trials per task
k: 3                # pass@3 and pass^3
gate:
  max_pass_drop: 0.05          # ship needs the pass-rate CI lower bound >= -5 pp
  task_regression: 0.5         # a task losing >= 50 pp has regressed...
  max_regressed_tasks: 0       # ...and any regressed task blocks
  max_cost_increase: 0.20      # relative to baseline; only gated when set
  min_tasks: 2                 # fewer non-flaky tasks -> hold
judge: {url: https://ai.acme.example, wire: anthropic-messages, model: claude-sonnet-5-5, api_key_env: HALO_EVAL_GATEWAY_TOKEN}
variants:
  - {name: current,   harness: claude, version: "2.1.312", model: sonnet}
  - {name: candidate, harness: claude, version: "2.1.330", model: sonnet}
matrix:
  baseline: claude@2.1.312/sonnet/orchestrator
  harnesses:
    - {harness: claude, version: "2.1.312", models: [sonnet]}
    - {harness: codex,  version: "0.100.0", models: [codex-default]}
    - {harness: gemini, version: "0.35.0",  models: [gemini-default]}
  providers:
    - {name: orchestrator}
    - {name: anthropic-direct, aliases: {sonnet: sonnet-direct}}

The matrix

--matrix expands the suite into cells named <harness>@<version>/<model>/<provider> and gates every cell against the baseline cell. Providers are gateway model aliases, so the CLI side stays provider-agnostic: the gateway routes each alias to its upstream.

VARIANT                                 PASS@1  PASS@2  PASS^2    $/TASK      P50  TOOLS  VERDICT
claude@2.1.312/sonnet/anthropic            89%    100%     78%    $0.200    36.0s   13.0  baseline
claude@2.1.312/sonnet/bedrock              89%    100%     78%    $0.210    36.0s   13.0  ship
codex@0.100.0/gpt-5/anthropic              67%     67%     67%    $0.350    36.0s   13.0  block
gemini@0.35.0/gemini-2.5-pro/anthropic    100%    100%    100%    $0.120    72.0s   13.0  ship
gate: BLOCK

The numbers come from the matrix test fixture (internal/eval/matrix_test.go); the table is a text rendering of them that no test pins, not the golden Markdown PR-comment report (internal/eval/testdata/golden/matrix.md) and not the output of upgrade-gate.yaml, whose cells are named after its own providers and models. You produce the real thing with halo eval run evals/suites/upgrade-gate.yaml --matrix --report report.md --fail-on block. A live run needs Docker and the harness images in evals/images/. --report writes a Markdown scorecard ready for a PR comment. --fail-on block|hold exits 3, which makes it a CI gate. Trials run with no network egress by default.

The online loop

$ halo eval online --pairs /var/lib/halo-shadow/pairs.jsonl --pair-key-file key.b64 \
    --rubric evals/rubrics/code-change-quality.yaml \
    --judge-url https://ai.acme.example --judge-model claude-sonnet-5-5 \
    --history scorecards.jsonl --otlp https://otel.acme.example:4320 --otlp-token-file eval-token

This samples halo-shadow pairs per experiment (a seeded shuffle, so reruns grade the same pairs) and judges both sides. For each experiment it reports the candidate − control score delta with a bootstrap CI, plus a win rate. Results go to the scorecard history (already-graded pairs are skipped) and are emitted as halo.eval.* metrics to the collector’s dedicated eval receiver (halo telemetry collector-config --eval-receiver, port 4320, bearer token from HALO_OTLP_EVAL_TOKEN via --otlp-token-file). They are stamped halo.source=eval. halo.eval.judge.score is a registered metric (higher is better) that shadow experiments and dark-launch rollout steps can use as a primary metric or guardrail: halo exp analyze and the controller read it from source=eval rows, and a breach opens a pause or rollback PR. Eval evidence never trips the kill switch; only gateway-measured evidence does. Pairs where either side errored are not judged: that is a reliability signal, not a quality one, so it is read separately. Over every pair, errored ones included, the run also reports each arm’s error rate (non-200 or failed replies, which is where refusals and overloads show up), mean latency and mean output tokens, printed below the score table and emitted as halo.eval.shadow.error_rate, halo.eval.shadow.latency_ms and halo.eval.shadow.output_tokens per halo.variant. That is the latency, refusal and cost comparison ADR-0004 promises; format regressions are the rubric’s job. It also reads every pair at the step level without a judge: each arm’s next action on the real prefix, agreement with the control, fidelity to the action production actually took (the control’s fidelity is the noise floor), invalid tool calls, premature stops and divergence by depth. All intervals resample sessions, not pairs. With turns: all this covers every step of real multi-turn sessions, including the steps that answer terminal output. See shadow traffic and ADR-0014.