End-to-end tests

make test-e2e runs Go suites against a live local stack over HTTP: real JWT login, real Postgres rows, no mocks. Because some tests call a real LLM, a spend policy is enforced in code.

The tests get a small, fixed set of named test agents, each with a hard daily cost cap. If a cap is hit the test fails on purpose: it proves the guard works, and it stops a bad loop from running up a bill.

Run it

make dev-infra   # Postgres + Redis + MinIO (once)
make run-api     # control-plane on :8080 (terminal 1)

export LANTERN_E2E_API_URL=http://localhost:8080
make test-e2e    # terminal 2

The target runs make e2e-lint, then the e2e/runtime and e2e/quickstart suites (Go, build tag e2e, standalone modules, so plain go test ./... never runs them). make test-e2e-llm is an alias. It targets the single-tier topology (stub scheduler, no KVM needed). Quickstart needs an LLM provider configured for the dev tenant.

On a two-tier stack (LANTERN_SCHEDULER_GRPC_ADDR plus a Docker-backend runtime-manager) set LANTERN_E2E_IMAGE_DIGEST to a locally pullable image; with the placeholder default, schedule returns 500.

Suites

Dir	Covers
e2e/runtime	/v1/runtime/* control path: auth gates, schedule (201 + vm_id), tenant-scoped list, detail and audit, logs SSE, terminate, quota 402.
e2e/quickstart	The examples/quickstart/ agents: create from spec.json, post run.json, poll to succeeded, assert an output shape; incident-summarizer also issues and verifies a receipt. Real LLM calls.

Spend policy

Enforced by e2e/internal/e2ekit, which is the only way a suite creates agents or posts runs:

• Agent names must be one of 8 reserved names: e2e-quickstart, e2e-support, e2e-pr-triage, e2e-incident, e2e-digest, e2e-analyst, e2e-assistant, e2e-voice. At most 8 e2e-* agents exist at once.
• Each agent gets a hard-fail budget before any run: maxCostUsdPerDay 0.50, maxRunsPerDay 15, hardFail true. The kit reads it back and fails the test if it is absent.
• Model is pinned to the cheapest tier: reasoning-small if Anthropic is configured, else chat-small, else fast (env-configured keys). Never a large model.
• web_search is stripped from every spec except e2e-assistant.
• A 402 on POST /v1/runs fails the test with e2e daily budget exhausted for <name> (cap working; rerun after local midnight).
• make e2e-lint fails the build if anything under e2e/ references /v1/budgets/tenant; the tests never touch the tenant-wide cap.

The quickstart suite posts 3 runs per invocation against one slot, and the cap is 15 runs/day, so expect about 5 full runs per day. The pinned model is advisory until the run executor honors the agent’s model field; the budget is what actually bounds spend.

Environment variables

Variable	Default	Purpose
LANTERN_E2E_API_URL	http://localhost:8080	Target API. Canonical; always set it explicitly.
LANTERN_API_URL	unset	Alias, used only when the canonical var is unset
LANTERN_E2E_EMAIL	admin@lantern.dev	Login email (seeded dev user)
LANTERN_E2E_PASSWORD	lantern	Login password
LANTERN_E2E_IMAGE_DIGEST	placeholder digest	Image to schedule in the runtime suite; needed on two-tier stacks
E2E_ALLOW_SKIP	unset	1 turns a down stack into a skip

The runtime suite prints the resolved target URL at the start of a run.

E2E_ALLOW_SKIP

Every test probes GET /healthz first. If the stack is unreachable the test fails, so make test-e2e exits non-zero and you know nothing ran. With E2E_ALLOW_SKIP=1 it skips and exits 0. A reachable API with wrong credentials is always a failure.

E2E_ALLOW_SKIP=1 make test-e2e   # exits 0 when the stack is down

What it creates

• Agents named from the reserved e2e-* set above, each with a budget.
• Runs on those agents (quickstart), and scheduled VMs plus a temporary quota row (runtime).

Tests delete their agents via t.Cleanup, and the runtime suite terminates its VMs and restores the quota row even when assertions fail. The quickstart suite also sweeps leftover e2e-* agents in TestMain before running.

make e2e-clean

Manual recovery after an interrupted run. It runs go run ./cmd/sweep in e2e/internal/e2ekit, logs in with the same URL and credentials variables, lists GET /v1/agents, and issues DELETE /v1/agents/{name} for every agent whose name starts with e2e-. Nothing else is deleted; failures are printed, never fatal. Use it when the kit refuses to create an agent because 8 e2e-* agents already exist.

LANTERN_E2E_API_URL=http://localhost:8080 make e2e-clean

Browser tier (Playwright)

apps/web/e2e drives the real dashboard in headless Chromium: sign-up, forgot/reset password (the reset link is read from the log mailer), onboarding (template gallery and the free-text path through the fixture LLM to a succeeded first run), the settings danger zone, and the light/dark theme.

cd apps/web
npm ci && npx playwright install chromium
npm run test:e2e

It boots its own stack on spare ports (web 3411, API 8411, llm-replay 9412): a throwaway Postgres database, a private Redis db index, the control-plane (with LANTERN_CORS_ORIGINS set to the dashboard origin), llm-replay and next dev. It never touches :8080, :3001 or the shared lantern database, and tears everything down afterwards. Needs Go, psql, and Postgres and Redis reachable (the make dev-infra defaults work; override with E2E_PG_ADMIN_URL, E2E_REDIS_*). Details and kiosk-spec hooks: apps/web/e2e/README.md.

CI

.github/workflows/sdlc-e2e.yml runs the fixture tier on every pull request and push to master, with Postgres (pgvector) and Redis service containers and no provider key. Three jobs: examples (scripts/e2e-examples.sh), browser (Playwright; report and traces are uploaded on failure) and deploy-cli (e2e/deploy against a control-plane started in the job). There is no skip path: a skipped test fails the job. The real-LLM suites (make test-e2e-llm) are a local gate only, because they need a provider key and spend money.

Danger

Never point the suites at a stack with real tenant data: they delete every agent named e2e-* in the logged-in tenant.

Fixture LLM tier

What. llm-replay is a small stdlib server that speaks the OpenAI chat-completions and Anthropic messages wire formats (JSON and SSE streaming, text and tool calls). Setting LANTERN_LLM_BASEURL_OPENAI and/or LANTERN_LLM_BASEURL_ANTHROPIC sends every control-plane provider call there instead of the real API.

Why. CI and kiosk mode need the same answer every time and must not need a provider key or spend money. Token usage is still reported, so budgets and cost metering run for real.

How. The default reply is fixture: plus the first 60 characters of the last user message. A rules file scripts the rest; the first rule whose match is a substring of the system prompt or the last user message wins (this is how the demo kiosk tells its agents apart). GET /healthz returns 200.

[
  {"match": "refund", "reply": "Refund issued."},
  {"match": "weather", "tool": {"name": "get_weather", "args": {"city": "Austin"}}}
]

A tool rule fires only when the request declares tools. When the next request carries the tool result, the reply is fixture: tool result: <first 60 chars>, so a scripted tool loop always ends. Usage is about one token per 4 characters of the request body and the reply.

Try it

cd services/control-plane
go run ./cmd/llm-replay -rules rules.json      # 127.0.0.1:9192

LANTERN_LLM_BASEURL_OPENAI=http://127.0.0.1:9192 \
LANTERN_LLM_BASEURL_ANTHROPIC=http://127.0.0.1:9192 \
OPENAI_API_KEY=dummy make run-api

curl -s localhost:8080/v1/completions -H "Authorization: Bearer $TOKEN" \
  -d '{"messages":[{"role":"user","content":"hello"}]}'   # "fixture: hello"
curl -s 127.0.0.1:9192/__requests                         # what the control-plane sent

Limits

• Dev and test only. The control-plane refuses to start with either variable set when LANTERN_ENV is prod, production or staging, and ignores them at request time.
• A provider key must still exist (database or OPENAI_API_KEY/ANTHROPIC_API_KEY); any dummy value works.
• Only /v1/chat/completions and /v1/messages are served. Other provider endpoints (embeddings, speech, transcription) return 404.
• No model quality: it echoes and follows rules. It cannot validate prompts.

Example suites on a fixture stack

What. make test-e2e-examples runs scripts/e2e-examples.sh: it creates a throwaway database, starts llm-replay, connector-replay, the control-plane (pointed at both) and the surface-gateway, runs e2e/examples, and tears everything down. It never touches :8080 or your dev database. Needs Postgres and Redis from make dev-infra (a random Redis DB 1-14 is used).

What it asserts.

Test	Flow
TestDigest_*	Schedule with America/New_York returns a 09:00 New York nextFireAt; made due, the scheduler creates the run; GitHub is read and exactly one multi-line chat.postMessage reaches the Slack fixture with the digest intact; the schedule advances to the next 09:00. A bad timezone is a 400.
TestTelegram_*	Missing or wrong X-Telegram-Bot-Api-Secret-Token is a 401 and sends nothing; a signed update runs the workflow (Google Calendar fixture read) and a sendMessage with the answer reaches the Telegram fixture for the right chat; a second message gets its own reply. WhatsApp X-Hub-Signature-256 negatives live in services/surface-gateway/tests.
TestVoice_*	Unsigned and wrong-token webhooks and status callbacks are 403 and change nothing; a signed inbound call gets TwiML and reserves $0.05 in agent_usage_daily; the status callback (61 s) reconciles it to $0.06, replays are no-ops, a never-connected call refunds to zero; over a hard-fail budget the call gets <Reject/> with no row and no spend.

Try it.

make dev-infra
make test-e2e-examples                 # ~70 s, mostly the scheduler's 60 s poll

E2E_EXAMPLES_SERVE=1 scripts/e2e-examples.sh   # leave the stack up, no tests (debugging)
scripts/e2e-examples.sh -run TestVoice          # extra args go to go test

Limits.

• Connector and LLM replies are recorded fixtures; this proves the wiring and the security gates, not model quality or vendor API drift. Real-account runs are in each example’s README.
• The gateway creates runs over gRPC, which the control-plane’s recovery sweep drives; the script sets LANTERN_RECOVERY_INTERVAL=2s so a reply takes seconds, not up to 30.
• Needs go, cargo, psql/createdb and python3 on the PATH. Runs in CI as the examples job.

Not covered

• True cross-tenant isolation (a second tenant probing the first) needs a second seeded tenant.
• PUT /v1/runtime/quota clamps non-positive ceilings, so the 402 test pins maxConcurrentVms to the current live-VM count instead.