Skip to content
Halosv0.3.0

ADR-0009: Signed, poll-based kill switch for experiments

ADR-0009: Signed, poll-based kill switch for experiments

Section titled “ADR-0009: Signed, poll-based kill switch for experiments”
  • Status: accepted (extends ADR-0003 and the guardrail rollback in the experiments design)
  • Date: 2026-09-30

Amended 2026-10-21 (third security review). A valid gateway token is accepted even from an IP blocked for failed attempts (ADR-0022), which tells a blocked guesser when a guess is right. halo-server now refuses to start with any gateway token, shared or per capability, shorter than 32 characters or under 128 bits of estimated entropy (length times log2 of its distinct characters), and names openssl rand -hex 32 in the error (test: TestReadGatewayTokens).

Amended 2026-10-22 (final security review). The length times log2(distinct characters) estimate accepted passphrases such as halos-gateway-token-production-2026 and abcdefghijklmnopqrstuvwxyz012345. A gateway token must now look random: 32 or more hex or base64/base64url characters only, at least 128 bits by charset bits times length, no run of 8 repeated or consecutive characters, at least 8 (hex) or 16 (base64) distinct characters, and, for base64, upper case, lower case and digits. The error still names openssl rand -hex 32 (test: TestReadGatewayTokens).

Context and problem statement

Section titled “Context and problem statement”

Experiments on the traffic plane (ADR-0003) change what model real developers get. A guardrail breach (cost, error rate, latency) means the treatment should stop now. The only path that existed was a policy PR setting status: paused: review, merge, halo gateway compile, snapshot roll-out. That takes minutes to hours, which is too slow for a regression that burns money or breaks every request, and it is the wrong shape for an automated controller that detects the breach at 3am.

At the same time, anything that can switch routing without a merge is a new control channel to the gateways. It must not become a way to attack them, and it must not weaken the rule that a human owns every irreversible step.

Decision drivers

Section titled “Decision drivers”
  • Rollback of a bad experiment in seconds, without a merge.
  • The new channel can only move traffic back to control; it cannot roll anything out.
  • A compromised network path, a stolen release key or a replayed response must not be able to forge or undo a kill.
  • The control plane being down must never silently re-enable a killed experiment.
  • Reuse what exists: halo-server already holds admin sessions, an audit log and a data dir; gateways already poll policy snapshots.
  • Small enough to reason about: one signed document, one key, one token.

Considered options

Section titled “Considered options”
  1. Policy PR only. Keep status: paused in git as the sole mechanism; make the controller open the PR faster.
  2. Unsigned kill endpoint. Gateways poll halo-server over TLS for a plain JSON list.
  3. mTLS only. Authenticate the gateway-to-server channel with client certificates and trust the body.
  4. Signed kill list with a dedicated key (chosen). halo-server serves an ed25519-signed list; gateways verify it and apply it on top of their policy snapshot.

Decision outcome

Section titled “Decision outcome”

Chosen: option 4.

  • halo-server keeps an append-only killswitch.jsonl (who, when, why, per experiment; last record wins). An admin (POST /api/v1/experiments/{name}/kill) or the controller on a rollback verdict writes it, and each change is audit-logged.
  • GET /api/v1/gateway/killswitch returns {payload, signature}. The payload is {version, experiments, issuedAt}; the signature is ed25519 over "halo-killswitch-v1\n" + payload. It is enabled only when both --killswitch-key-file and --gateway-token-file are set.
  • A dedicated key, generated with halo keys generate --name killswitch, separate from the release key. The gateway holds only the public half.
  • A bearer gateway token (random hex or base64, at least 32 characters and 128 bits since the final review; 128 bits of estimated entropy since round-3 hardening; originally 16+ characters) controls who may read the list; the signature controls what a gateway believes.
  • halo-proxy and halo-kong poll every 10 seconds. A killed experiment is applied as status: paused on the gateway’s in-memory policy: control routing, no shadow mirroring, no experiment headers.
  • Verification rules: signature valid; issuedAt no more than 10 minutes old and no more than 1 minute in the future; issuedAt strictly newer than the list already held.
  • Failure policy: a failed fetch or verification keeps the last accepted list. A gateway that has never accepted a list kills nothing and the policy snapshot decides. Kills never expire and are lifted only by an explicit unkill (pausing or concluding in policy does not lift them).
  • The controller still opens the pause PR on rollback. The kill is the fast, reversible stopgap; the merged PR is the durable state.

Consequences

Section titled “Consequences”
  • Good: a rollback verdict takes effect on gateways within one poll interval, with no merge.
  • Good: the channel is one-directional in effect. It can only return users to control; it has no way to add a route or start an experiment.
  • Good: a stolen release key cannot forge kills, and a stolen kill key cannot sign a release or ring pointer. Replayed or stale envelopes are rejected, so an attacker cannot un-kill by replaying an old list.
  • Good: a control-plane outage does not re-enable a killed experiment.
  • Bad: a second key and a second secret (the gateway token) to generate, store and rotate. There is no rotation protocol; rotating means updating every gateway’s public key and halo-server together.
  • Bad: a gateway that has never fetched a valid list kills nothing, so a kill issued while halo-server is unreachable takes effect only after the gateway can fetch. A fresh gateway starts unprotected until its first successful poll.
  • Bad: whoever holds the kill key and controls a gateway’s path can kill experiments (a denial of the experiment, not of service: users keep working on control). Anyone who can write killswitch.jsonl on the server can un-kill.
  • Bad: kills do not expire, so a forgotten kill keeps an experiment off after the policy is fixed; operators must unkill before restarting.
  • Limit: it acts on gateways only. A client-axis experiment changes profiles on machines, which a kill cannot reach; those variants stay until the pause PR merges and halod pulls the release. (Later: halod can poll the same signed list on an opt-in basis; see client-axis delivery.)
  • Neutral: halo gateway deck does not yet generate the killswitch_* plugin fields; Kong users add them by hand.

Pros and cons of the options

Section titled “Pros and cons of the options”

1. Policy PR only

Section titled “1. Policy PR only”
  • Good: one mechanism, git is the source of truth, nothing new to secure.
  • Bad: minutes to hours to take effect, and it needs a human in the loop at the moment speed matters. Rejected as too slow; kept as the durable path.

2. Unsigned kill endpoint

Section titled “2. Unsigned kill endpoint”
  • Good: trivial to build.
  • Bad: TLS protects the transport, not the content. A compromised proxy, a misconfigured ingress cache or an on-path attacker can serve an empty list (hiding a kill) or a full one (switching off every experiment), and a replayed old response silently un-kills. Rejected.

3. mTLS only

Section titled “3. mTLS only”
  • Good: strong channel authentication with existing tooling.
  • Bad: authenticates the pipe, not the message; replay and a compromised server-side TLS terminator are not covered, and it requires a client-certificate PKI on every gateway, which heterogeneous stacks (Kong, Envoy, nginx, API Gateway) do not share. It is not a substitute for signing, but remains a compatible hardening on top, and halo-proxy does not provide mTLS today. Rejected as the sole control.

More information

Section titled “More information”
  • Concepts and failure policy: experiments
  • Wire format and endpoints: API reference
  • Threat rows T22 to T26: threat model
  • Code: internal/gateway/killswitch.go, internal/server/killswitch.go, internal/controller/killstore.go

Addendum 2026-09-30: who may trip it, and what happens without it

Section titled “Addendum 2026-09-30: who may trip it, and what happens without it”

A security review tightened how the controller reaches the switch. The wire format and trust chain above are unchanged.

  • Only trusted evidence kills. The controller trips the switch on a rollback verdict only when the deciding evidence is gateway-sourced (the collector’s authenticated receiver stamped halo.source=gateway). CLI telemetry is client-controlled, so a CLI-sourced rollback opens the pause PR and notifies (killOutcome: not_gateway_evidence) and a human decides. See evidence plane.
  • No key, no pretence. halo-server serves a kill list and accepts kill and unkill only with --killswitch-key-file (which requires --data-dir and --gateway-token-file). Without it, the in-process controller does not record a kill nobody reads: it reports the rollback as not enforced and asks humans to merge the pause PR urgently, and GET /api/v1/capabilities reports killSwitch: false. halo controller run writes the kill to the data dir and says it is enforced only with --killswitch-served.
  • Killed means held. The controller does not evaluate a running experiment that is already killed; it tells humans once to unkill before resuming.
  • A kill needs a reason (HTTP 422 without one) and is audited fail-closed: the kill stays applied if the audit append fails (fail safe), a failed unkill is reverted, and the request returns 500.
  • No cleartext token. Gateways refuse a plain http:// kill URL to a non-loopback host unless killSwitch.allowInsecureInCluster (halo-kong: killswitch_allow_insecure_in_cluster) is set. A misconfigured halo-kong keeps serving, logs ERROR once a minute and sets x-halo-killswitch: misconfigured on the upstream request.
  • Webhooks are replay-resistant. The generic webhook carries X-Halo-Timestamp and an HMAC over timestamp + "." + body; receivers reject timestamps more than 5 minutes off.

Addendum 2026-10-02: two limits lifted

Section titled “Addendum 2026-10-02: two limits lifted”
  • halo gateway deck generates the killswitch_* fields. Pass --killswitch-url (https, or --killswitch-allow-insecure-in-cluster for an in-cluster http:// URL) and the decK output carries killswitch_url, killswitch_pubkey and killswitch_token references (internal/gateway/kong/deck.go; golden testdata/kong-killswitch.golden.yml). The “Neutral” consequence above no longer applies.
  • halod polls the same signed list, opt-in. With killSwitch in halod.yaml (portal enrollment sets it when halo-server has a kill key), a device whose client-axis experiment is killed reverts to the ring release within one kill-poll interval; the same verification rules apply (signature, freshness, monotonic issuedAt), and a failed poll keeps the last accepted list. Without it the device waits for the pause PR and republish, as the “Limit” above says. See ADR-0010.

Implementation status (2026-10-02)

Section titled “Implementation status (2026-10-02)”
Commitment Code Test Status
Append-only killswitch.jsonl; admin kill/unkill endpoints; controller writes on rollback; each change audit-logged internal/server/killswitch.go; internal/controller/killstore.go, controller.go server TestKillEndpointsAuthz, TestToggleKillEndpoints, TestKillRejectsInvalidNames; controller TestKillStore, TestRolloutBreachKillsOnce Done
GET /api/v1/gateway/killswitch returns {payload, signature} over "halo-killswitch-v1\n" + payload; enabled only with key and gateway token internal/server/killswitch.go; internal/gateway/killswitch.go (SignKillList, VerifyKillList) gateway TestKillListSignVerify; server TestGatewayKillswitchEndpoint, TestKillWithoutKeyNotImplemented, TestFleetKillswitchEndpoint Done
Dedicated key (halo keys generate --name killswitch), separate from the release key cmd/halo/cmd_release.go (keys generate); gateways hold the public half only TestKillListSignVerify (release key cannot sign a kill) Done
Bearer gateway token (random hex or base64, 32+ chars, 128+ bits) gates reads internal/server/killswitch.go, server.go TestGatewayKillswitchEndpoint (401 without token) Done
halo-proxy and halo-kong poll every 10 s; killed = status: paused (control routing, no mirroring, no experiment headers) internal/gateway/killswitch.go (KillSwitch, poller); cmd/halo-proxy, cmd/halo-kong/main.go TestKillSwitchPoller, TestKillSwitchApplyRoutesControl; halo-kong TestAccessKillSwitch; server TestKillSwitchEndToEnd Done
Verification: signature, issuedAt ≤ 10 min old and ≤ 1 min in the future, strictly newer than held internal/gateway/killswitch.go (VerifyKillList, refresh) TestKillSwitchPoller (stale, future, replayed envelopes) Done
Failure policy: keep last accepted list; never-fetched kills nothing; kills never expire; only unkill lifts internal/gateway/killswitch.go; internal/server/killswitch.go TestKillSwitchPoller; TestToggleKillEndpoints (pause in policy does not lift) Done
Controller still opens the pause PR on rollback internal/controller/controller.go TestRollbackSurvivesRestartAndFailures, TestRollbackKillFailureStillNotifies Done
No rotation protocol (documented limit) production deployment n/a Accepted
Only gateway-sourced evidence kills; CLI-sourced rollback notifies not_gateway_evidence internal/controller/controller.go, rollouts.go, notify.go TestRollbackOnNonGatewayEvidenceDoesNotKill, TestRolloutBreachOnCLIEvidenceDoesNotKill, TestEvalEvidenceNeverKills, TestKillOutcomeWording Done
No key, no pretence: capabilities.killSwitch, “not enforced” reporting, halo controller run --killswitch-served internal/server/server.go; internal/controller; cmd/halo/cmd_controller.go server TestKillWithoutKeyNotImplemented (capabilities); controller TestRollbackKillNotEnforced; cmd/halo TestControllerRunFlags Done
Killed means held internal/controller/controller.go TestKilledExperimentIsHeld, TestNoEvaluationAfterRollback Done
Kill needs a reason (422); audited fail-closed (kill stays on audit failure, unkill reverted, 500) internal/server/killswitch.go TestKillEndpointsAuthz (no/blank reason), TestKillAuditFailure Done
No cleartext token: refuse plain http:// to non-loopback unless allowInsecureInCluster; halo-kong flags x-halo-killswitch: misconfigured internal/gateway/killswitch.go (NewKillSwitch); cmd/halo-kong/main.go TestNewKillSwitchValidates; halo-kong TestKillSwitchURLMustBeHTTPS; halo-proxy TestKillSwitchConfig; test/uat/kong (misconfigured header, cannot be spoofed) Done
Webhooks replay-resistant (X-Halo-Timestamp, HMAC over timestamp.body, 5-minute skew) internal/controller/notify.go TestWebhookSignature, TestWebhookVerify Done
halo gateway deck generates killswitch_* fields internal/gateway/kong/deck.go kong TestGenerateKillswitchGolden, TestKillswitchURLMustBeHTTPS Done (was Neutral/open, see addendum)
Client-axis devices: halod polls the signed list (opt-in) cmd/halod/agent.go (kill poll), config.go (KillSwitch) cmd/halod TestKillSwitchRevertsClientExperiment, TestKillSwitchUnrelatedOrUnreachable, TestLoopPollsKillSwitch Done (was a stated limit, see addendum)