End-to-end tests

make test-e2e runs Go suites against a live local stack over HTTP: real JWT login, real Postgres rows, no mocks. Because some tests call a real LLM, a spend policy is enforced in code.

Run it

make dev-infra   # Postgres + Redis + MinIO (once)
make run-api     # control-plane on :8080 (terminal 1)

export LANTERN_E2E_API_URL=http://localhost:8080
make test-e2e    # terminal 2

The target runs make e2e-lint, then the e2e/runtime and e2e/quickstart suites (Go, build tag e2e, standalone modules, so plain go test ./... never runs them). make test-e2e-llm is an alias. It targets the single-tier topology (stub scheduler, no KVM needed). Quickstart needs an LLM provider configured for the dev tenant.

On a two-tier stack (LANTERN_SCHEDULER_GRPC_ADDR plus a Docker-backend runtime-manager) set LANTERN_E2E_IMAGE_DIGEST to a locally pullable image; with the placeholder default, schedule returns 500.

Suites

Spend policy

Enforced by e2e/internal/e2ekit, which is the only way a suite creates agents or posts runs:

• Agent names must be one of 8 reserved names: e2e-quickstart, e2e-support, e2e-pr-triage, e2e-incident, e2e-digest, e2e-analyst, e2e-assistant, e2e-voice. At most 8 e2e-* agents exist at once.
• Each agent gets a hard-fail budget before any run: maxCostUsdPerDay 0.50, maxRunsPerDay 15, hardFail true. The kit reads it back and fails the test if it is absent.
• Model is pinned to the cheapest tier: reasoning-small if Anthropic is configured, else chat-small, else fast (env-configured keys). Never a large model.
• web_search is stripped from every spec except e2e-assistant.
• A 402 on POST /v1/runs fails the test with e2e daily budget exhausted for <name> (cap working; rerun after local midnight).
• make e2e-lint fails the build if anything under e2e/ references /v1/budgets/tenant; the tests never touch the tenant-wide cap.

The quickstart suite posts 3 runs per invocation against one slot, and the cap is 15 runs/day, so expect about 5 full runs per day. The pinned model is advisory until the run executor honors the agent's model field; the budget is what actually bounds spend.

Environment variables

The runtime suite prints the resolved target URL at the start of a run.

E2E_ALLOW_SKIP

Every test probes GET /healthz first. If the stack is unreachable the test fails, so make test-e2e exits non-zero and you know nothing ran. With E2E_ALLOW_SKIP=1 it skips and exits 0. A reachable API with wrong credentials is always a failure.

E2E_ALLOW_SKIP=1 make test-e2e   # exits 0 when the stack is down

What it creates

• Agents named from the reserved e2e-* set above, each with a budget.
• Runs on those agents (quickstart), and scheduled VMs plus a temporary quota row (runtime).

Tests delete their agents via t.Cleanup, and the runtime suite terminates its VMs and restores the quota row even when assertions fail. The quickstart suite also sweeps leftover e2e-* agents in TestMain before running.

make e2e-clean

Manual recovery after an interrupted run. It runs go run ./cmd/sweep in e2e/internal/e2ekit, logs in with the same URL and credentials variables, lists GET /v1/agents, and issues DELETE /v1/agents/{name} for every agent whose name starts with e2e-. Nothing else is deleted; failures are printed, never fatal. Use it when the kit refuses to create an agent because 8 e2e-* agents already exist.

LANTERN_E2E_API_URL=http://localhost:8080 make e2e-clean

CI

No workflow under .github/workflows runs make test-e2e today. A pipeline without a live stack should set E2E_ALLOW_SKIP=1, otherwise the run fails by design. Never point the suites at a stack with real tenant data: they delete every agent named e2e-* in the logged-in tenant.

Fixture LLM tier

What. llm-replay is a small stdlib server that speaks the OpenAI chat-completions and Anthropic messages wire formats (JSON and SSE streaming, text and tool calls). Setting LANTERN_LLM_BASEURL_OPENAI and/or LANTERN_LLM_BASEURL_ANTHROPIC sends every control-plane provider call there instead of the real API.

Why. CI and kiosk mode need the same answer every time and must not need a provider key or spend money. Token usage is still reported, so budgets and cost metering run for real.

How. The default reply is fixture: plus the first 60 characters of the last user message. A rules file scripts the rest; the first rule whose match is a substring of the system prompt or the last user message wins (this is how the demo kiosk tells its agents apart). GET /healthz returns 200.

[
  {"match": "refund", "reply": "Refund issued."},
  {"match": "weather", "tool": {"name": "get_weather", "args": {"city": "Austin"}}}
]

A tool rule fires only when the request declares tools. When the next request carries the tool result, the reply is fixture: tool result: <first 60 chars>, so a scripted tool loop always ends. Usage is about one token per 4 characters of the request body and the reply.

Try it

cd services/control-plane
go run ./cmd/llm-replay -rules rules.json      # 127.0.0.1:9192

LANTERN_LLM_BASEURL_OPENAI=http://127.0.0.1:9192 LANTERN_LLM_BASEURL_ANTHROPIC=http://127.0.0.1:9192 OPENAI_API_KEY=dummy make run-api

curl -s localhost:8080/v1/completions -H "Authorization: Bearer $TOKEN"   -d '{"messages":[{"role":"user","content":"hello"}]}'   # "fixture: hello"
curl -s 127.0.0.1:9192/__requests                         # what the control-plane sent

Limits

• Dev and test only. The control-plane refuses to start with either variable set when LANTERN_ENV is prod, production or staging, and ignores them at request time.
• A provider key must still exist (database or OPENAI_API_KEY/ANTHROPIC_API_KEY); any dummy value works.
• Only /v1/chat/completions and /v1/messages are served. Other provider endpoints (embeddings, speech, transcription) return 404.
• No model quality: it echoes and follows rules. It cannot validate prompts.

Not covered

• True cross-tenant isolation (a second tenant probing the first) needs a second seeded tenant.
• PUT /v1/runtime/quota clamps non-positive ceilings, so the 402 test pins maxConcurrentVms to the current live-VM count instead.