Low-level design

Each diagram on this page traces one code path in the order the code runs. The numbered list under each diagram gives the file and line range for every stage, so you can check the drawing against the source. For what each binary does as a whole, see Components.

Request path inside halo-proxy

Everything below runs inside Proxy.ServeHTTP (cmd/halo-proxy/proxy.go:187-363). The reverse proxy is built once, in newProxy (cmd/halo-proxy/proxy.go:74-122).

1. Snapshot. p.snap.Get() returns the compiled policy and reloads it when the file changes (internal/gateway/snapshot.go:32-61). If the policy is missing and there is no --next-hop or --default-upstream, the answer is 503 policy_unavailable (cmd/halo-proxy/proxy.go:244-249).

2. Identity. authenticate picks a verifier from the configured mode (internal/identity/resolve.go:68-83):

   • OIDC JWT: go-oidc against the issuer’s JWKS (internal/identity/oidc.go:71-91). With the default user claim email, the token must also carry email_verified=true unless trustUnverifiedEmail is set (internal/identity/oidc.go:93-110).
   • Trusted header: honoured only when the TCP peer is in the trusted proxy CIDRs (internal/identity/identity.go:72-100).
   • None.

   Gemini CLI’s x-goog-api-key is verified like a bearer token. A failure is 401 with WWW-Authenticate, an identity header sent twice is 400, and --allow-anonymous downgrades a failure to default routing (cmd/halo-proxy/proxy.go:391-429).

3. Body bound. Every body is wrapped in MaxBytesReader. Model calls are read in full for inspection, and oversize is a hard 413 (cmd/halo-proxy/proxy.go:257-279). The default limit is 32 MiB (cmd/halo-proxy/config.go:104).

4. Kill list. KillSwitch.Apply returns a copy of the policy with every killed experiment set to paused and every killed toggle removed. Killed experiments therefore route to control and mirror nothing (internal/gateway/killswitch.go:280-305). The list is polled every 10 s by default and accepted only if its ed25519 signature verifies, it is at most 10 minutes old, and it is newer than the current list (internal/gateway/killswitch.go:60-82, internal/gateway/killswitch.go:159-245).

5. Decision. PrepareVerified collects every client x-halo-* header name for removal, builds the request info from the verified subject only, and calls Decide (internal/gateway/prepare.go:139-180). Decide resolves the subject’s ring. It then applies a traffic toggle’s route for the alias (internal/gateway/toggles.go:12-26), and the first traffic-axis experiment by name whose variant routes this alias. A client-axis experiment is attributed but never routes (internal/gateway/decision.go:60-144). admit enforces the model allowlist: 404 for unknown paths, 400 for models that are not a policy alias (internal/gateway/prepare.go:103-137). The request body is rewritten to the upstream model.

6. Route order. RouteOrder sorts targets into priority tiers. Within a tier the weighted pick goes first, chosen by assign.Pick over the sticky key user + session (internal/gateway/route.go:14-58, internal/assign/assign.go:28-53). assign is the single source of hashing (AGENTS.md invariant 3).

7. Attempts. buildAttempts drops a target, with a log line, when:

   • its upstream kind cannot serve the client’s wire protocol
   • its URL fails CheckURL
   • it would SigV4-sign for a host that is neither AWS nor in signHosts
   • translation fails in BuildOutbound

   (cmd/halo-proxy/route.go:48-97, internal/gateway/upstreamauth/apikey.go:99-121, internal/gateway/translate.go:39-81). If no target is left, the answer is 502 bad_upstream (cmd/halo-proxy/proxy.go:308-313).

8. Header hygiene. Hygiene runs after the decision. That is safe because the decision never read client x-halo-* headers.

   • The client x-halo-* headers and the identity and groups headers are deleted.
   • The gateway’s own x-halo-ring|release|experiment|variant are stamped.
   • Client credentials are stripped unless --forward-auth is set (cmd/halo-proxy/proxy.go:324-355, internal/gateway/upstreamauth/apikey.go:47-51).

9. Mirror. For shadow experiments, shadow.Jobs builds jobs for eligible steps (first turns, or every step with turns: all) that carry only allowlisted headers, and the mirror queues them asynchronously (cmd/halo-proxy/proxy.go:431-441, internal/shadow/mirror.go:130-150).

10. Failover. routeTransport walks the attempts (cmd/halo-proxy/route.go:105-167):

    • At most 3 attempts by default.
    • A target whose breaker is open is skipped. With more than one target, the breaker opens after 5 consecutive failures and allows one half-open probe after 30 s (internal/gateway/breaker.go:46-92).
    • A connect error, a 5xx or a 429 moves to the next target.
    • Retries happen only before any byte reaches the client.

    Each attempt in try installs that target’s own credential (cmd/halo-proxy/route.go:178-253):

    • Vertex: a metadata-server or service-account token (internal/gateway/upstreamauth/vertex.go:205-214).
    • Azure OpenAI, and OpenAI or Gemini with a policy credential: the secret from the env var or file it names, read at use time (internal/gateway/upstreamauth/apikey.go:21-45, internal/gateway/upstreamauth/apikey.go:55-98).
    • Anthropic and orchestrator upstreams: no credential of the gateway’s own; only the operator’s per-upstream upstreamHeaders from the YAML config, or the caller’s credential with --forward-auth (cmd/halo-proxy/route.go:190-192).
    • Bedrock: SigV4 in the transport (internal/gateway/upstreamauth/bedrock.go:139-194, internal/gateway/upstreamauth/bedrock.go:206-208).

    If every target fails, the answer is 503 no_available_target (cmd/halo-proxy/proxy.go:135-153).

11. Stream. FlushInterval: -1 flushes every write (cmd/halo-proxy/proxy.go:107-111). A Bedrock event stream is converted to SSE when the client spoke the Anthropic protocol (cmd/halo-proxy/route.go:221-226, internal/gateway/eventstream.go:18-20).

12. Deferred metrics. A defer records the request after the response completes (cmd/halo-proxy/proxy.go:195-240):

    • halo.gateway.requests and halo.gateway.latency_ms, sent over OTLP/HTTP to the collector’s gateway receiver, for decided model calls only (not auth failures or count-tokens). The unit is the verified subject, never a client header. The served attempt sets halo.gateway.provider, halo.gateway.target and halo.gateway.failover (cmd/halo-proxy/proxy.go:205-209, cmd/halo-proxy/proxy.go:286-288, internal/telemetry/gwmetrics/gwmetrics.go:192-218).
    • The Prometheus histograms on the admin listener.
    • One access-log line. Bodies, headers and query strings are never logged.

Release pipeline

halo release publish --ring R --release-version V --key K (cmd/halo/cmd_release.go:181-236):

1. Load. policy.Load reads halos.yaml and every *.yaml under the directory in sorted order, decoding strictly so unknown fields are errors (internal/policy/load.go:79-153).

2. Validate. loadValid refuses to continue on any validation error (cmd/halo/root.go:184-198). Org.Validate runs the semantic checks, then the 16 built-in guardrails, then any ExtraGuardrails (which can only add issues, ADR-0007) (internal/policy/validate.go:56-79, internal/policy/guardrails.go:18-40), among them guardNoBypass (internal/policy/guardrails.go:48-69).

3. Build ring. release.BuildRing builds one release per variant of every running client-axis experiment that enrolls the ring, then the ring release itself. The ring release’s manifest lists those experiments with their salt, weights and channels (internal/release/channels.go:24-59).

4. Render. release.Build calls each enabled adapter’s Render for darwin, linux and windows. Warnings land in the manifest; nothing is silently dropped (internal/release/release.go:110-189).

5. Backstop. CheckRendered fails the build if any rendered file contains a permission-bypass token (policy.BypassToken, case-insensitive: modes such as bypassPermissions and danger-full-access and the harnesses’ bypass CLI flags). It scans each decoded JSON/TOML value (the error names the field) and then the raw bytes of every file, so a file that does not parse, or is not JSON/TOML, is still covered. Policy validation, halo ci config and vendor admin sync call the same function. It also runs per-harness checks (internal/release/check.go:17-83). It is called once per harness and OS (internal/release/release.go:160-162).

6. Pack. pack writes a deterministic USTAR tar (manifest.json plus blobs/<sha256>, epoch timestamps). The release digest is sha256(tar) (internal/release/release.go:191-223).

7. Sign. The signer is ed25519, or a MultiSigner{ed25519, Cosign} when --cosign-key or --keyless is set (cmd/halo/cmd_release.go:104-123). publish signs the tar digest before pushing anything. Co-signatures are stored as extra manifest annotations (internal/bundle/bundle.go:50-73). cosign runs as an external cosign sign-blob, without a transparency-log upload unless keyless (internal/bundle/sign.go:152-171). halod verifies only ed25519, so a cosign-only signer is refused (internal/bundle/pointer.go:82-92).

8. Push and tag. Config and layer blobs are pushed if missing, and the OCI manifest is packed with the signature annotations. v<version> is tagged, and moving an existing tag is an error (internal/bundle/bundle.go:79-123). PublishRing publishes the variant channels before the ring, and runs every pointer preflight before any pointer moves (internal/bundle/channels.go:53-95).

9. Ring pointer. writePointer builds {org, ring, digest, manifest, seq, issuedAt, expiresAt}:

   • seq is max(current+1, unix now, recorded+1).
   • The time to live is 7 days.
   • The signature is ed25519 over "halos.pointer.v1 " + sha256(payload).
   • It is pushed as its own artifact under ring-<r>.pointer, and the informational ring-<r> tag is moved (internal/bundle/pointer.go:21-58, internal/bundle/pointer.go:239-300).

   Rollback is PromoteRing to an older version: a new pointer with a higher seq (internal/bundle/channels.go:103-150).

Release binaries and images are signed separately. GoReleaser signs keyless with cosign (.goreleaser.yaml:159-171) in the tag-triggered release workflow, and its verify job checked the v0.3.0 signatures; see tech stack.

halod apply loop

halod run loops: Once every interval (default 15m), and with a kill switch, PollKill every killSwitch.interval (default 60s) in between. Both run on one goroutine, so state.json has a single writer (cmd/halod/main.go:163-190, cmd/halod/config.go:70-71, cmd/halod/config.go:233-242).

1. Kill list. refreshKill restores the persisted list, fetches and verifies the current one with the kill public key (not the release key), and persists it once accepted. A restart therefore cannot be fed an older envelope (cmd/halod/agent.go:319-337, cmd/halod/keys.go:68-86).

2. Ring. A fixed ring wins. Otherwise halod calls GET ringEndpoint?user=&host= with the device token, and validates the returned ring, subject and groups. If that fails, the last-known ring is used, except on the first run (cmd/halod/agent.go:214-226, cmd/halod/agent.go:400-438).

3. Pointer. bundle.PullRing reads ring-<r>.pointer and checks its type and size and the ed25519 signature over the domain-separated statement. It then requires org = config, seq ≥ the last applied seq, the same digest when the seq is unchanged, and an unexpired pointer (internal/bundle/pointer.go:96-135, internal/bundle/pointer.go:390-430). halod accepts any configured key that is not listed in revokedKeys (cmd/halod/keys.go:40-65).

4. Release. fetchRelease fetches the manifest by the digest the pointer names, never through a tag. It verifies the signature over the layer digest before downloading, caps the layer size (256 MiB by default), and checks the digest while reading. It then requires release digest = pointer digest and release org = pointer org (internal/bundle/pointer.go:467-506, internal/bundle/pointer.go:416-429).

5. Variant. If the ring release lists client-axis experiments, selectVariant computes the device’s variant with policy.Experiment.ResolveVariant, the same hash the gateway uses. It skips experiments on the kill list and pulls the variant channel with the same pointer checks. If that channel fails, the whole cycle fails: falling back would silently move treatment devices to control (cmd/halod/agent.go:269-317).

6. Toggles. activeToggles evaluates the client toggles in the signed manifest against the ring, subject and groups, minus the killed ones. withToggles merges their per-harness fragments (cmd/halod/toggles.go:27-96).

7. Check targets. For every file, checkTarget requires (cmd/halod/agent.go:537-547, cmd/halod/paths.go:65-83, cmd/halod/paths.go:119-141):

   • a path inside the adapter’s managed directories or files
   • mode 0644 or lower
   • every existing parent root-owned, not group- or world-writable, and not a planted symlink

   A file that fails is refused and the rest still apply.

8. Write. ensureFile writes only when the content hash differs, and flags drift when the release did not change. fsutil.WriteAtomic writes a temp file, fsyncs it and renames it over the target (cmd/halod/agent.go:549-583, internal/fsutil/fsutil.go:15-59).

9. Stale files. Files the previous release owned that this one does not ship are removed, but only inside managed locations (cmd/halod/agent.go:505-520).

10. State and report. state.json records the digest, owned files, pointer marks, installs and kill list. The status is printed as one JSON line and POSTed to reportURL with the device token (cmd/halod/agent.go:523-532, cmd/halod/agent.go:660-692).

Between pulls, PollKill reacts to kill-list changes without a registry round trip (cmd/halod/agent.go:352-394):

• If the device’s experiment was just killed, halod applies the cached, already-verified ring release.
• If the experiment was unkilled, or toggle kills changed while the device is on a variant, halod runs a full cycle.

Controller loop

halo-server --controller runs the loop every --interval (5m by default, at least 1m) (cmd/halo-server/main.go:76). After a failed tick it retries sooner, from 15 s up to the interval (internal/controller/controller.go:356-376). halo controller run runs the same loop.

1. Tick. One tick has a 2-minute budget. It loads the served policy and walks the running experiments by name (internal/controller/controller.go:85-125).

2. Rollout-owned. An experiment that backs an active Rollout is skipped here; the rollout drives it (internal/controller/rollouts.go:61-70).

3. Killed. A killed running experiment is not evaluated, because its evidence is all control. Humans are told once (internal/controller/controller.go:127-138, internal/controller/controller.go:310-322).

4. Evidence. promote.ClickHouse reads halo_metrics over ClickHouse’s HTTP interface with server-side bound parameters (internal/promote/source.go:38-136). Samples returns one value per (variant, unit) and the evidence source of the row (internal/promote/source.go:219-253).

5. Guardrails. Each guardrail is tested with stats.SeqGuardrail, an always-valid mSPRT confidence sequence, at alpha divided by the number of guardrails. Any failure is a rollback (internal/promote/promote.go:117-137, internal/stats/guardrail.go:37-71).

6. Budgets. Exceeding maxDays or maxSpendUSD is expired (internal/promote/promote.go:139-146).

7. Sample floor. Fewer than minSamples per arm is continue (internal/promote/promote.go:148-152).

8. Primary. A significant regression is rollback. A significant improvement with every guardrail passing is promote (internal/promote/promote.go:154-171).

9. Rollback. The kill switch is tripped first, but only when the deciding evidence is gateway-sourced, because CLI and eval telemetry can be forged or are opinion (internal/controller/controller.go:182-213). The kill appends to killswitch.jsonl with an fsync (internal/controller/killstore.go:112-149). Every gateway sees it at its next kill-list poll (10 s by default). The controller then opens a pause PR and notifies, even if the kill failed (internal/controller/controller.go:215-232).

10. Promote or expired. The controller opens a PR concluding the experiment, then notifies (internal/controller/controller.go:234-250). PRs are opened by GitWriter, which commits on a new branch in a temporary worktree, pushes it and runs gh pr create (internal/controller/writer.go:31-58, internal/promote/pr.go:311-328). Nothing merges. Each action is recorded in controller-state.jsonl, so it happens at most once per experiment run (internal/controller/controller.go:252-268).

11. Rollouts. For each Rollout (internal/controller/rollouts.go:42-136):

    • its state is loaded from rollouts/<name>.json, and a file whose hash chain does not verify is never acted on (internal/rollout/state.go:154-186)
    • evidence and scorecards are gathered, and rollout.Evaluate decides advance, hold, rollback, pause or complete (internal/rollout/rollout.go:84-151)

    rollback and pause halt the rollout. A traffic-axis rollback on gateway evidence also trips the kill switch. Both then open a PR (internal/controller/rollouts.go:138-181). advance and complete only open a PR (internal/controller/rollouts.go:183-195).

UNVERIFIED: GitHub PRs

GHOpener is marked UNVERIFIED against a real GitHub in its own source. Its tests use real git and a recording gh (internal/promote/pr.go:285-291).

Data model

Every document is apiVersion: halos.dev/v1 (internal/policy/types.go:6-13). policy.Load joins them into one Org (internal/policy/types.go:34-53). The JSON Schemas are in schemas/.

Kind	Go type	Key fields	References
halos.yaml (root)	Root (internal/policy/load.go:21-33)	org, identity (issuer, clientID, audience, claims, adminGroups), selfService, simple-mode fields	none
Gateway	Gateway (internal/policy/types.go:81-97)	baseURL, protocols, auth, models (alias to ModelRoute), upstreams, engine	Routes name upstreams; a route is one upstream+model or weighted, prioritized targets (internal/policy/types.go:109-129)
Profile	Profile (internal/policy/types.go:223-240)	extends, harnesses (exact version pin, model, overrides), models, permissions, mcp, hooks, telemetry, egress	extends another Profile; harnesses.*.model must be a Gateway alias
Ring	Ring (internal/policy/types.go:315-337)	order, profile, release (pinned digest), membership (users, groups, percent, default, optIn)	Profile
Experiment	Experiment, Variant (internal/policy/types.go:377-402)	type (ab, canary, shadow), axis (client, traffic), status, rings, variants, metrics, stopping, salt	Rings; traffic variants override Gateway alias routes; client variants name a Profile
Toggle	Toggle (internal/policy/toggle.go:28-84)	owner, expires, default, axis, ordered rules (rings, groups, users, percent, effect), client or traffic.routes	Rings in rules; traffic toggles override Gateway alias routes
Rollout	Rollout, RolloutStep (internal/policy/rollout.go:58-129)	axis, status, step, experiment, change (release or alias), baseline, steps (strategy, ring, percent, bake, gates, onFailure)	The backing Experiment; Rings per step; baseline.release for rollback

Ring and variant assignment both go through internal/assign (internal/policy/resolve.go:22-74).

Trust zones and keys

Secret	Held by (private part)	Verified by	Code
Release key (ed25519)	The publisher: halo release publish --key	halod (pubkey/pubkeys minus revokedKeys)	internal/bundle/sign.go:52-67, cmd/halo/cmd_release.go:104-123, cmd/halod/keys.go:40-65
Kill-list key (ed25519, a separate key)	halo-server --killswitch-key-file	halo-proxy --killswitch-pubkey-file, halo-kong killswitch_pubkey, halod killSwitch.pubkey	cmd/halo-server/main.go:207-218, internal/gateway/killswitch.go:44-75, internal/server/killswitch.go:162-197
Gateway token	halo-server --gateway-token-file (random hex or base64, at least 32 characters and 128 bits; compared as sha256, in constant time)	Presented by gateways on GET /api/v1/gateway/killswitch	cmd/halo-server/main.go:268-300, internal/server/killswitch.go:130-148
Fleet token	halo-server --token-file	MDM and devcontainer fleets on POST /api/v1/fleet/report	internal/server/server.go:273-277, internal/server/server.go:296-300
Device token	Minted at POST /api/v1/enroll from a one-time enrollment token; halo-server stores only its hash; expires after --device-ttl	halod on fleet ring, report and kill-list calls	internal/server/portal.go:510-608, internal/server/devices.go:110-165, internal/server/devices.go:302-322
Developer JWT	The IdP	halo-proxy and halo-kong (go-oidc, JWKS)	internal/identity/oidc.go:71-91
Console session	halo-server (OIDC code flow, state and nonce checked, sealed cookie)	halo-server	internal/server/auth.go:111-132, internal/server/auth.go:322-400
OTLP gateway token	Collector env HALO_OTLP_GATEWAY_TOKEN	Gateway receiver :4319, which stamps halo.source=gateway; halo-proxy sends it from --telemetry-token-file	internal/telemetry/collector.go:71-102, cmd/halo-proxy/config.go:183
OTLP eval token	Collector env HALO_OTLP_EVAL_TOKEN, which must differ from the gateway token	Eval receiver :4320 (halo.source=eval)	internal/telemetry/collector.go:92-95
CLI receivers	Optional token file	:4317 gRPC and :4318 HTTP (halo.source=cli)	internal/telemetry/collector.go:71-86
Shadow token	HALO_SHADOW_TOKEN	halo-shadow /mirror	cmd/halo-shadow/main.go:79
Provider credentials	The gateway host: the env var or file a policy credential names (Azure OpenAI, OpenAI, Gemini), read at use time; the AWS default chain (Bedrock); GOOGLE_APPLICATION_CREDENTIALS or the metadata server (Vertex); operator upstreamHeaders (Anthropic, orchestrator)	Never taken from the client, except with --forward-auth	internal/gateway/upstreamauth/apikey.go:21-45, internal/gateway/upstreamauth/bedrock.go:108-125, internal/gateway/upstreamauth/vertex.go:202-214

Only gateway-sourced evidence may trip the kill switch automatically. CLI telemetry can be forged by anyone who can reach the collector, and eval evidence is a model’s opinion; both open the pause PR, and a human decides (internal/controller/controller.go:193-200).