Testing

make test-e2e runs Go suites against a live local stack over HTTP, with a real JWT login and real Postgres rows, no mocks. Some suites call a real LLM, so every test agent gets a hard-fail daily budget from a fixed set of named agents. If a cap is hit the test fails on purpose: the guard works, and a bad loop cannot run up a bill. A second tier replays recorded LLM and connector responses, so CI needs no provider key and spends nothing.

Run the suites

make dev-infra   # Postgres + Redis + MinIO (once)
make run-api     # control plane on :8080 (terminal 1)

export PAL_E2E_API_URL=http://localhost:8080
make test-e2e    # terminal 2

The target runs make e2e-lint, then the e2e/runtime, e2e/quickstart, e2e/examples and e2e/deploy suites. They are Go with build tag e2e, in standalone modules, so plain go test ./... never runs them. make test-e2e-llm is an alias. It targets the single-tier topology (stub scheduler, no KVM needed), and the quickstart suite needs an LLM provider configured for the dev tenant.

On a two-tier stack (PAL_SCHEDULER_GRPC_ADDR plus a Docker-backend runtime-manager), set PAL_E2E_IMAGE_DIGEST to a locally pullable image. With the placeholder default, schedule returns 500.

If the stack is down, every test fails, so make test-e2e exits non-zero and you know nothing ran. E2E_ALLOW_SKIP=1 turns that into a skip with exit 0. A reachable API with wrong credentials is always a failure.

Danger

Never point the suites at a stack with real tenant data: they delete every agent named e2e-* in the logged-in tenant.

Suite	Covers
e2e/runtime	/v1/runtime/* control path: auth gates, schedule (201 + vm_id), tenant-scoped list, detail and audit, logs SSE, terminate, quota 402.
e2e/quickstart	The examples/quickstart/ agents: create from spec.json, post run.json, poll to succeeded, assert an output shape. incident-summarizer also issues and verifies a receipt. Real LLM calls.
e2e/examples	The fixture-tier example flows (digest, Telegram, voice). Needs the fixture stack that make test-e2e-examples builds.
e2e/deploy	The deploy CLI against a control plane (the deploy-cli CI job).

Environment

Variable	Default	Purpose
PAL_E2E_API_URL	http://localhost:8080	Target API. Always set it explicitly.
PAL_API_URL	unset	Alias, used only when the canonical variable is unset
PAL_E2E_EMAIL	admin@pal.local	Login email (seeded dev user)
PAL_E2E_PASSWORD	pal	Login password
PAL_E2E_IMAGE_DIGEST	placeholder digest	Image to schedule in the runtime suite; needed on two-tier stacks
E2E_ALLOW_SKIP	unset	1 turns a down stack into a skip

Spend policy

Enforced by e2e/internal/e2ekit, the only way a suite creates agents or posts runs.

• Reserved names. Agent names must be one of 8: e2e-quickstart, e2e-support, e2e-pr-triage, e2e-incident, e2e-digest, e2e-analyst, e2e-assistant, e2e-voice. At most 8 e2e-* agents exist at once.
• Budget first. Each agent gets a hard-fail budget before any run: maxCostUsdPerDay 0.50, maxRunsPerDay 15, hardFail true. The kit reads it back and fails the test if it is absent.
• Cheapest model. The model is pinned to reasoning-small if Anthropic is configured, else chat-small, else fast (env-configured keys). Never a large model.
• No search. web_search is stripped from every spec except e2e-assistant.
• A 402 fails the test with e2e daily budget exhausted for <name> (cap working; rerun after local midnight).
• Tenant cap untouched. make e2e-lint fails the build if anything under e2e/ references /v1/budgets/tenant.

The quickstart suite posts 3 runs per invocation against one slot, and the cap is 15 runs a day, so expect about 5 full runs per day.

Cleanup

Tests delete their agents with t.Cleanup, and the runtime suite terminates its VMs and restores its temporary quota row even when assertions fail. The quickstart suite also sweeps leftover e2e-* agents in TestMain.

After an interrupted run, recover manually. make e2e-clean runs go run ./cmd/sweep in e2e/internal/e2ekit, lists GET /v1/agents, and issues DELETE /v1/agents/{name} for every agent whose name starts with e2e-. Nothing else is deleted, and failures are printed, never fatal. Use it when the kit refuses to create an agent because 8 e2e-* agents already exist.

PAL_E2E_API_URL=http://localhost:8080 make e2e-clean

Fixture LLM tier

llm-replay is a small standard-library server that speaks the OpenAI chat-completions and Anthropic messages wire formats (JSON and SSE streaming, text and tool calls). Setting PAL_LLM_BASEURL_OPENAI and/or PAL_LLM_BASEURL_ANTHROPIC sends every control-plane provider call there instead of the real API. CI and kiosk mode use it so answers are identical every time and need no key. Token usage is still reported, so budgets and cost metering run for real.

The default reply is fixture: plus the first 60 characters of the last user message. A rules file scripts the rest: the first rule whose match is a substring of the system prompt or the last user message wins. This is how the demo kiosk tells its agents apart.

[
  {"match": "refund", "reply": "Refund issued."},
  {"match": "weather", "tool": {"name": "get_weather", "args": {"city": "Austin"}}}
]

A tool rule fires only when the request declares tools. When the next request carries the tool result, the reply is fixture: tool result: <first 60 chars>, so a scripted tool loop always ends.

cd services/control-plane
go run ./cmd/llm-replay -rules rules.json      # 127.0.0.1:9192

PAL_LLM_BASEURL_OPENAI=http://127.0.0.1:9192 \
PAL_LLM_BASEURL_ANTHROPIC=http://127.0.0.1:9192 \
OPENAI_API_KEY=dummy make run-api

curl -s localhost:8080/v1/completions -H "Authorization: Bearer $TOKEN" \
  -d '{"messages":[{"role":"user","content":"hello"}]}'   # "fixture: hello"
curl -s 127.0.0.1:9192/__requests                         # what the control plane sent

• Dev and test only. The control plane refuses to start with either variable set when PAL_ENV is prod, production or staging, and ignores them at request time.
• A provider key must still exist (database or OPENAI_API_KEY / ANTHROPIC_API_KEY); any dummy value works.
• Only /v1/chat/completions and /v1/messages are served. Embeddings, speech and transcription return 404.
• It echoes and follows rules. It has no model quality and cannot validate prompts.

Example suites on a fixture stack

make test-e2e-examples runs scripts/e2e-examples.sh. It creates a throwaway database, starts llm-replay, connector-replay (Connectors), the control plane (pointed at both) and the surface gateway, runs e2e/examples, and tears everything down. It never touches :8080 or your dev database. It needs Postgres and Redis from make dev-infra (a random Redis DB 1 to 14 is used).

Test	Flow
TestDigest_*	A schedule with America/New_York returns a 09:00 New York nextFireAt. Made due, the scheduler creates the run; GitHub is read and exactly one multi-line chat.postMessage reaches the Slack fixture with the digest intact; the schedule advances to the next 09:00. A bad timezone is a 400.
TestTelegram_*	A missing or wrong X-Telegram-Bot-Api-Secret-Token is a 401 and sends nothing. A signed update runs the workflow (Google Calendar fixture read) and a sendMessage with the answer reaches the Telegram fixture for the right chat; a second message gets its own reply. WhatsApp X-Hub-Signature-256 negatives live in services/surface-gateway/tests.
TestVoice_*	Unsigned and wrong-token webhooks and status callbacks are 403 and change nothing. A signed inbound call gets TwiML and reserves $0.05 in agent_usage_daily; the status callback (61 s) reconciles it to $0.06, replays are no-ops, and a never-connected call refunds to zero. Over a hard-fail budget the call gets <Reject/> with no row and no spend.

make dev-infra
make test-e2e-examples                 # ~70 s, mostly the scheduler's 60 s poll

E2E_EXAMPLES_SERVE=1 scripts/e2e-examples.sh   # leave the stack up, no tests (debugging)
scripts/e2e-examples.sh -run TestVoice          # extra args go to go test

• Connector and LLM replies are recorded fixtures. This proves the wiring and the security gates, not model quality or vendor API drift. Real-account runs are in each example’s README.
• The gateway creates runs over gRPC, which the control plane’s recovery sweep drives. The script sets PAL_RECOVERY_INTERVAL=2s so a reply takes seconds, not up to 30.
• It needs go, cargo, psql/createdb and python3 on the PATH.

Browser tier

apps/web/e2e drives the real dashboard in headless Chromium with Playwright: sign-up, forgot and reset password (the reset link is read from the log mailer), onboarding (template gallery and the free-text path through the fixture LLM to a succeeded first run), the settings danger zone, and the light and dark theme.

cd apps/web
npm ci && npx playwright install chromium
npm run test:e2e

It boots its own stack on spare ports (web 3411, API 8411, llm-replay 9412): a throwaway Postgres database, a private Redis db index, the control plane (with PAL_CORS_ORIGINS set to the dashboard origin), llm-replay and next dev. It never touches :8080, :3001 or the shared dev database, and tears everything down afterwards. It needs Go, psql, and a PRIVATE Postgres and Redis you start yourself (e.g. -p 55511:5432, -p 56511:6379); E2E_PG_ADMIN_URL and E2E_REDIS_PORT are required and the shared :5432/:6379 are refused. Details: apps/web/e2e/README.md.

CI

.github/workflows/sdlc-e2e.yml runs the fixture tier on every pull request and push to master, with Postgres (pgvector) and Redis service containers and no provider key. Three jobs: examples (scripts/e2e-examples.sh), browser (Playwright; report and traces upload on failure) and deploy-cli (e2e/deploy against a control plane started in the job). A skipped test fails the job. The real-LLM suites (make test-e2e-llm) are a local gate only, because they need a provider key and spend money.

Not covered

• True cross-tenant isolation (a second tenant probing the first) needs a second seeded tenant.
• PUT /v1/runtime/quota clamps non-positive ceilings, so the 402 test pins maxConcurrentVms to the current live-VM count instead.