Skip to content
Halosv0.3.0

Components

Halos ships six binaries from cmd/. Two more surfaces run inside one of them or next to it: the MCP server and the Claude Code plugin. Every claim on this page links to the code it comes from. Defaults are the values in the flag definitions.

Component Plane Runs where Listens on (default)
halo Policy / release Laptop, CI None (the MCP server uses stdio)
halod Client Every managed machine, as root None (outbound only)
halo-proxy Traffic Cluster or VM :8088 traffic, 127.0.0.1:9090 admin
halo-kong Traffic Inside Kong Kong’s own listeners
halo-shadow Evidence Cluster 127.0.0.1:8090
halo-server Control / evidence Cluster 127.0.0.1:8080
MCP server Agents Agent host stdio
Claude Code plugin Agents Claude Code n/a

halo

Section titled “halo”

Purpose: the operator CLI. It validates policy, builds and publishes signed releases, moves ring pointers, drives experiments, rollouts and toggles, runs evals, and serves MCP.

  • Inputs:
    • a policy repo directory, loaded by policy.Load
    • an ed25519 signing key (--key)
    • an OCI registry
    • ClickHouse, for exp analyze and mcp serve --clickhouse
  • Outputs:
    • release tarballs (release build)
    • OCI artifacts and signed pointers (release publish, cmd/halo/cmd_release.go:181-236)
    • git branches and pull requests
    • JSON (--output json)
  • Key internal packages:
    • internal/policy: types, strict loader, validation and the default guardrails.
    • internal/release: renders every harness for every OS into a content-addressed tar.
    • internal/bundle: OCI push and pull, ed25519 and cosign signing, signed ring pointers.
    • internal/harness/*: adapters for Claude Code, Codex, Gemini CLI, Copilot CLI, Cursor CLI and OpenCode.
    • internal/promote, internal/rollout, internal/controller: verdicts, rollout state and PRs.
    • internal/mcpserver: halo mcp serve.
    • internal/eval, internal/stats: replay evals, scorecards and statistics.
  • State and storage: the signer’s pointer state file, $XDG_STATE_HOME/halos/pointers.json (~/.local/state/halos/pointers.json when unset) (internal/bundle/state.go:36-48). It refuses replayed or forked pointers on later writes.
  • Trust boundary:
  • Config: flags only. See the CLI reference.
  • Ports: none. halo mcp serve speaks MCP over stdio (cmd/halo/mcp.go:35).
  • Tests:
    • unit tests: cmd/halo/*_test.go, internal/{policy,release,bundle,harness/...}/*_test.go
    • e2e: TestPolicyReleaseDelivery, TestSimpleMode, TestRollout, TestRolloutSimpleModeCompletion, TestToggles and TestObsEvidencePlane in test/e2e
    • UAT: TestK8sUAT

halod

Section titled “halod”

Purpose: the fleet agent. It runs as root on each machine, pulls the ring’s signed release, verifies it, and applies it atomically to an allowlisted set of paths.

  • Inputs:
    • halod.yaml (cmd/halod/config.go:17-56)
    • the OCI registry, read-only
    • the ring endpoint GET /api/v1/fleet/ring
    • the signed kill list GET /api/v1/fleet/killswitch
  • Outputs:
    • harness config files under each adapter’s managed directories
    • verified CLI installs
    • one JSON status line per cycle on stdout, also POSTed to reportURL (cmd/halod/agent.go:660-692)
  • Key internal packages:
    • internal/bundle: PullRing verifies the pointer and the release.
    • internal/release: manifest and blobs.
    • internal/policy: ResolveVariant for client-axis experiments.
    • internal/gateway: KillSwitch, the same verifier the gateways use.
    • internal/toggle: client toggle evaluation.
    • internal/harness/all: managed paths come from the adapters.
    • internal/fsutil: atomic writes.
  • State and storage:
    • Fixed root-owned layout per OS (cmd/halod/paths.go:25-38). On Linux that is /etc/halos/halod.yaml, /var/lib/halos/state.json, /usr/local/lib/halos/{bin,npm} and shims in /usr/local/bin.
    • state.json holds the applied digest, the files it owns, the last pointer seq and digest per ring and channel (anti-rollback), installed artifacts, and the last accepted kill list (cmd/halod/agent.go:74-91).
  • Trust boundary:
  • Config:
    • registry, org, ring or ringEndpoint
    • pubkey / pubkeys / revokedKeys
    • interval (default 15m, cmd/halod/config.go:233-242)
    • reportURL, deviceTokenFile, subject, maxBundleBytes, allowShellInstall (default false)
    • killSwitch.{url,pubkey,interval} (interval default 60s, cmd/halod/config.go:70-71)
    • Flags: --config, --state, --install (default true), --log-level (cmd/halod/main.go:77-82).
  • Ports: none. halod makes outbound calls only.
  • Tests:
    • unit tests: cmd/halod/{agent,experiment,killswitch,toggles,npm,service,main}_test.go, plus trust_windows_test.go on Windows
    • e2e: TestPolicyReleaseDelivery, TestClientAxisExperiment, TestPortalEnrollment, TestToggles
    • UAT: TestCLIs (installs through the Dev Container Feature; test/uat/clis_env_test.go)

halo-proxy

Section titled “halo-proxy”

Purpose: the standalone gateway. It verifies the caller, decides ring, toggle and experiment per model alias, and routes to the provider with failover. It can also mirror requests to halo-shadow and export halo.gateway.* metrics. Every step is shown on the low-level design page.

  • Inputs:
    • model API calls (Anthropic Messages, Bedrock invoke, OpenAI Responses, Gemini)
    • the compiled policy snapshot, hot-reloaded (internal/gateway/snapshot.go:32-61)
    • the signed kill list
    • the issuer’s JWKS
  • Outputs:
    • upstream requests carrying the gateway’s own credential
    • streamed responses
    • x-halo-ring|release|experiment|variant headers set on the upstream request
    • shadow jobs
    • OTLP metrics to the collector’s gateway receiver
    • Prometheus text on /metrics
  • Key internal packages:
    • internal/identity: JWT via OIDC, trusted proxy header, or none.
    • internal/gateway: PrepareVerified, Decide, RouteOrder, Breaker, KillSwitch, BuildOutbound.
    • internal/gateway/upstreamauth: SigV4, Vertex tokens and API keys.
    • internal/shadow: the mirror queue.
    • internal/telemetry/gwmetrics: the OTLP exporter.
    • internal/bundle: kill-list key parsing.
  • State and storage: none on disk. In memory it keeps the policy snapshot, the last accepted kill list, breaker state and per-target token caches.
  • Trust boundary:
    • Cohort comes only from the verified subject.
    • Every client x-halo-* header and the identity and groups headers are dropped (cmd/halo-proxy/proxy.go:324-329).
    • Client credentials are stripped unless --forward-auth is set (cmd/halo-proxy/proxy.go:340-348).
    • Provider credentials come only from the gateway host, never from the client (cmd/halo-proxy/route.go:233-253). Vertex, Azure OpenAI, and OpenAI or Gemini upstreams with a policy credential get the gateway’s credential. Bedrock is SigV4-signed. Anthropic and orchestrator upstreams get only the operator’s upstreamHeaders (YAML, ${ENV}-expanded), or the caller’s own credential with --forward-auth.
    • SigV4 signs only for AWS hosts or a signHosts allowlist (cmd/halo-proxy/route.go:77-89).
  • Config: precedence is defaults, then YAML (--config, or env HALO_PROXY_CONFIG), then HALO_PROXY_<FLAG> env vars, then flags (cmd/halo-proxy/config.go:118-213). Some keys are YAML only: signHosts, upstreamHosts, allowInsecureUpstreams, upstreamHeaders, route.* and shadow.maxBytes. The flags (cmd/halo-proxy/config.go:145-186), including:
    • --policy
    • --identity-mode jwt|trusted_header|none, --issuer, --audience, --allow-anonymous
    • --killswitch-{url,token-file,pubkey-file,interval} (interval default 10s)
    • --halo-shadow-{url,token}
    • --telemetry-{otlp-endpoint,token-file,interval,unit-salt-file}
    • --max-body-bytes (32 MiB)
    • --upstream-header-timeout (10m)
    • Routing defaults: 3 attempts, breaker opens after 5 failures, 30 s cooldown (cmd/halo-proxy/config.go:113).
  • Ports:
  • Tests:
    • unit tests: cmd/halo-proxy/{proxy,route,bedrock,gemini,killswitch,telemetry,config}_test.go, internal/gateway/*_test.go, internal/gateway/upstreamauth/*_test.go
    • e2e: TestGatewayTrafficPlane, TestRoutes, TestGeminiWire, TestToggles
    • UAT: TestCLIs (real CLIs behind halo-proxy) and TestK8sUAT (Helm)

halo-kong

Section titled “halo-kong”

Purpose: the same gateway decision, packaged as a Kong Go plugin for the Access phase. It is a thin adapter over internal/gateway. Plugin version 0.1.0, priority 900 (cmd/halo-kong/main.go:64-67).

  • Inputs: the request via the Kong PDK (headers capped at 1000, path, raw body, peer IP) (cmd/halo-kong/main.go:314-320), the policy snapshot at policy_path, and the kill list.
  • Outputs:
    • The service request is rewritten: headers, scheme, target, path and body.
    • Or the request is answered with kong.response.exit for 401, 404, 413 or 400.
    • Shadow jobs.
  • Key internal packages: internal/gateway, internal/identity, internal/policy, internal/shadow, and internal/bundle (key parsing).
  • State and storage: none. Snapshots are cached in memory per policy_path (cmd/halo-kong/main.go:118-129).
  • Trust boundary: the same rules as halo-proxy.
    • JWT mode answers 401 to unverified model calls unless allow_unverified is set.
    • Trusted-header mode honours headers only from trusted_proxy_cidrs.
    • Caller credentials are removed unless forward_client_credentials is set (cmd/halo-kong/main.go:1-35).
    • Multi-target routes (weights, failover) are halo-proxy only. Kong uses each route’s first target (internal/policy/types.go:93-96).
  • Config: plugin JSON fields policy_path, identity_mode, issuer, audience, trusted_proxy_cidrs, shadow_url / shadow_token, and killswitch_url / killswitch_token / killswitch_pubkey. Secrets can be {vault://env/NAME} references (cmd/halo-kong/main.go:70-113).
  • Ports: none of its own. It runs as a Kong plugin server process (cmd/halo-kong/main.go:376).
  • Tests:
    • unit tests: cmd/halo-kong/{access,killswitch,main}_test.go, internal/gateway/kong
    • UAT: TestKong with subtests routing, identity, experiments, limits, killswitch, latency and kong, run inside real Kong OSS 3.9 (test/uat/kong/kong_test.go:327-637, make uat-kong)

halo-shadow

Section titled “halo-shadow”

Purpose: receives mirrored requests (first turns, or every step of a sampled session with turns: all), replays each against control and candidate, and stores the pairs for the LLM judge.

  • Inputs: POST mirror jobs from halo-proxy or halo-kong, carrying a shared bearer token, and the compiled policy snapshot.
  • Outputs: pairs.jsonl, mode 0600, optionally AES-256-GCM encrypted (--pair-key-file), and counters.
  • Key internal packages: internal/shadow (queue, workers, replay, budget) and internal/gateway (snapshot).
  • State and storage:
    • --out pairs.jsonl
    • --retention 720h (30 days)
    • --budget-usd stops intake at the estimated spend
  • Trust boundary:
    • A job carries no URL or model. halo-shadow resolves both upstreams from its own policy, so the token only lets a caller spend, within the budget, on upstreams the policy already names.
    • Credentials are per upstream (--upstream-headers), and redirects are not followed (cmd/halo-shadow/main.go:1-9).
    • HALO_SHADOW_TOKEN is required (cmd/halo-shadow/main.go:79).
  • Config: flags with HALO_SHADOW_* env defaults (cmd/halo-shadow/main.go:56-69):
    • --policy, --upstream-headers, --out, --pair-key-file, --retention
    • --queue 64, --workers 4
    • --price-in 3, --price-out 15
  • Ports:
    • 127.0.0.1:8090
    • optional --metrics-listen, which serves /metrics without the token
  • Tests:
    • unit tests: cmd/halo-shadow/main_test.go, internal/shadow/*_test.go
    • e2e: TestGatewayTrafficPlane (mirror path)

halo-server

Section titled “halo-server”

Purpose: the control plane. It serves the console and the API, enrollment, the fleet ring and report endpoints, and the signed kill list. With --controller it also runs the automated experiment and rollout loop.

  • Inputs:
    • the served policy directory, reloaded on SIGHUP (cmd/halo-server/main.go:244-256)
    • OIDC logins
    • halod reports
    • ClickHouse evidence, when --controller is set
  • Outputs:
    • JSON API (routes in internal/server/server.go:166-218)
    • signed kill lists
    • policy PRs, from the portal writer and the controller
    • Slack and webhook notifications
    • halo_controller_* metrics
  • Key internal packages:
    • internal/server: HTTP handlers, OIDC sessions, devices, audit and portal.
    • internal/controller: the loop, the kill store and the state log.
    • internal/promote: verdicts, ClickHouse source and PRs.
    • internal/rollout: rollout evaluation and hash-chained state.
    • web: the console, embedded with -tags=webdist.
      • Operator pages (read role): Fleet (per-ring compliance, a drift count per device and an “in sync” filter; the device page lists the drifted settings from halod’s last report), Releases, Rollouts (#/rollouts[/name]: step timeline with the live step, per-gate status, bake countdown, halt and rollback events, and the ring-to-release pointers; the controller state is shown only when halo-server --controller keeps it in the data dir) and Cost & budgets (#/budgets, see cost budgets).
  • State and storage: files under --data-dir:
  • Trust boundary:
  • Config: flags (cmd/halo-server/main.go:59-135):
    • --policy-dir, --token-file, --data-dir
    • --portal-config
    • --killswitch-key-file together with --gateway-token-file (or --gateway-killswitch-token-file); optional per-capability --gateway-{killswitch,posture,budget,directory}-token-file
    • --controller, --interval (5m, minimum 1m)
    • --clickhouse-*
    • --notify-*
    • --policy-repo-*
    • --device-ttl, --trusted-proxy-cidrs
    • --dev-insecure-user (demo only)
  • Ports:
    • 127.0.0.1:8080
    • optional --metrics-listen for controller /metrics
  • Tests:
    • unit tests: cmd/halo-server/main_test.go, internal/server/*_test.go, internal/controller/*_test.go
    • e2e: TestPortalEnrollment, TestClientAxisExperiment, TestRollout, TestToggles
    • UAT: TestK8sUAT

Controller

Section titled “Controller”

This is the same loop as halo controller run. Every --interval it evaluates each running experiment and active rollout against ClickHouse. When the evidence says rollback, it trips the kill switch, but only on gateway-sourced evidence. It then opens a PR and notifies. It never merges. See the controller loop.

Console

Section titled “Console”

web/ is a React, Vite and TypeScript SPA (tech stack), served by halo-server at / (internal/server/server.go:218). Admin routes are wrapped in s.admin(...) and developer routes in s.user(...).

MCP server

Section titled “MCP server”

Purpose: lets coding agents read and propose Halos changes. It is halo mcp serve, not a separate binary.

  • Inputs: --policy-dir, and optionally --clickhouse, which enables analyze_experiment (cmd/halo/mcp.go:38-43).
  • Outputs: tool results, halos:// resources and prompts. With --allow-writes, it can also create local git branches.
  • Tools:
    • Read: validate, plan, render_preview, whoami, list_rings, list_experiments, show_experiment, harness_matrix, explain_release_diff, eval_scorecard (internal/mcpserver/tools_read.go:103-117), plus list_rollouts, rollout_status, list_toggles, evaluate_toggle, eval_matrix, upgrade_candidates.
    • Write, only with --allow-writes: start_experiment, pause_experiment, conclude_experiment, propose_promotion, propose_rollback (internal/mcpserver/tools_write.go:56-72), propose_toggle_change, propose_rollout_advance, propose_rollout_rollback. Each defaults to dry_run and requires a reason. None of them pushes or merges.
  • Key internal packages: internal/mcpserver, internal/policy, internal/release, internal/promote, internal/rollout, internal/toggle, internal/eval, internal/upgrade.
  • State: none. Ports: none (stdio).
  • Tests: unit tests in internal/mcpserver/*_test.go; e2e TestMCPServe.

Claude Code plugin

Section titled “Claude Code plugin”

Purpose: packages the MCP server and Halos workflows for Claude Code.

plugins/claude-code contains:

  • .mcp.json, which runs halo mcp serve --policy-dir ${HALOS_POLICY_DIR:-.} without --allow-writes.
  • Skills: halos-author-policy, halos-autopilot, halos-onboard, halos-rollout, halos-triage.
  • Commands: /halos:halo-onboard, /halos:halo-autopilot, /halos:halo-rollout, /halos:halo-status.
  • Agent: halos-release-manager, which proposes but never publishes, retags, merges or pushes.
  • No hooks.

The plugin has no code of its own, so it has no tests. TestMCPServe covers the server it starts.

Package dependency map

Section titled “Package dependency map”

Every package under cmd/ and internal/, with its direct, non-test imports of other Halos packages. The graph is built from go list across GOOS=linux, darwin and windows. Rows and columns are in dependency order: a dot in row r, column c means r imports c. Because Go forbids import cycles, every dot falls below the diagonal.

Dependency structure matrix of the 45 Halos packages under cmd and internal, ordered by layer; every import falls below the diagonal. Dependency structure matrix of the 45 Halos packages under cmd and internal, ordered by layer; every import falls below the diagonal.

The data is in assets/src/pkgdeps.json. TestPackageGraph fails when that file no longer matches go list. To regenerate it and the diagram:

Terminal window
go test ./internal/docgen -run TestPackageGraph -update
python3 assets/src/build.py