Evaluations

Lantern has a built-in evaluation system for testing agents against declarative test cases, gating CI on regressions, running A/B experiments, and collecting human feedback — all from the same API the dashboard uses.

In plain termsHow do you know your agent is actually good — and stays good after you change its prompt? Evaluations are the answer: a set of example questions with known-good answers that you run against the agent like a test suite. Pin today's score as your baseline, and if a future change makes the agent worse, Lantern blocks it in CI before your users ever see the regression. A/B experiments and thumbs-up/down feedback feed the same loop.

Eval suites

An eval suite is a named collection of test cases for an agent. Each case defines an input and expected output (or a scoring function). Suites are upserted by (tenant, agent, name).

Create or update a suite

POST /v1/eval-suites
{
  "agentName": "research-agent",
  "name": "factual-accuracy",
  "cases": [
    {
      "id": "case-1",
      "input": { "question": "What year did WWII end?" },
      "expectedOutput": "1945",
      "scoreThreshold": 0.9
    }
  ]
}

Response: 200 OK
{ "id": "suite-uuid", "agentName": "research-agent", "name": "factual-accuracy" }

Other suite endpoints

GET    /v1/eval-suites              — list all suites (bare array)
GET    /v1/eval-suites?agentName=  — filter by agent
GET    /v1/eval-suites/{id}        — get suite
DELETE /v1/eval-suites/{id}        — delete suite

Recording a run + CI gate

After running your agent against a suite's cases, post the results. If the score has regressed vs. the branch baseline, the API returns HTTP 422 — wire this into your CI pipeline to block merges on regression.

POST /v1/eval-runs
{
  "suiteId": "suite-uuid",
  "agentName": "research-agent",
  "agentVersion": "v2",
  "commitSha": "abc1234",
  "branch": "main",
  "passed": 9,
  "score": 0.91,
  "casesResult": [
    { "id": "case-1", "passed": true, "score": 0.95, "actual": "1945" }
  ]
}

Response: 200 OK — passed baseline check
{ "id": "run-uuid", "score": 0.91, "passed": 9, "regressed": false }

Response: 422 Unprocessable Entity — regressed vs. baseline
{ "error": "score 0.91 is below baseline 0.94 on branch main", "regressed": true }

Other eval-run endpoints

GET /v1/eval-runs?suiteId=&agentName=&branch=  — list runs (bare array)

Baselines

Pin a specific eval run as the baseline for an agent + branch. Future runs on that branch are compared against it.

POST /v1/eval-baselines
{ "agentName": "research-agent", "branch": "main", "evalRunId": "run-uuid" }

GET /v1/eval-baselines?agentName=research-agent&branch=main

Eval observability

Two read-only endpoints over existing eval_runs data for the dashboard quality-trend sparkline and top-failing-cases view.

Failure clusters

GET /v1/eval-observability/failures?agentName=research-agent&branch=main&limit=50

Response:
{
  "clusters": [
    {
      "case": "What year did WWII end?",
      "failures": 3,
      "seen": 5,
      "failRate": 0.6,
      "sampleError": "Agent said 1944 instead of 1945",
      "firstSeen": "2026-07-01T...",
      "lastSeen": "2026-07-20T..."
    }
  ],
  "runsScanned": 50
}

Score trend

GET /v1/eval-observability/trends?agentName=research-agent&branch=main

Response:
{
  "points": [
    {
      "runId": "run-uuid",
      "createdAt": "2026-07-01T...",
      "score": 0.94,
      "passRate": 0.9,
      "passed": 9,
      "costUsd": 0.12,
      "agentVersion": "v1",
      "commitSha": "abc1234"
    }
  ],
  "latestVsMean": -0.03,
  "regressing": true
}

A/B experiments

Run two agent versions side by side with deterministic traffic splitting (FNV-1a hash on a caller-supplied bucketing key). Record per-request outcomes and let Lantern auto-promote the winner when it has a statistically significant lift.

Create an experiment

POST /v1/experiments
{
  "agentName": "research-agent",
  "variantAVersion": "v1",
  "variantBVersion": "v2",
  "trafficSplitB": 0.2,     // 20% to variant B
  "autoPromote": true       // auto-flip to B if lift > 2%
}

Response: 201 Created
{ "id": "exp-uuid", "status": "running", ... }

Record an outcome

POST /v1/experiments/{id}/record
{
  "bucketingKey": "user-123",  // determines which variant this caller gets
  "score": 0.95                // 0–1; your metric (e.g. task success rate)
}

// Auto-promotion fires when:
// - variant B score - variant A score > 0.02
// - both arms have ≥ minRunsPerArm samples

Conclude manually

POST /v1/experiments/{id}/conclude
{
  "winner": "b",     // "a" | "b" | null (inconclusive)
  "promote": true    // flip agent's currentVersionId to the winner
}

Other experiment endpoints

GET /v1/experiments         — list (bare array)
GET /v1/experiments/{id}    — get with current a_score / b_score

Human feedback (RLHF)

Collect thumbs-up / thumbs-down reactions on individual runs. Score is 1–5; 4–5 is positive, 1–2 is negative. Negative runs feed the rehearsal queue automatically.

POST /v1/runs/{id}/feedback
{
  "score": 2,
  "comment": "The agent missed the follow-up question",
  "preferredOutput": "It should have asked for clarification first"
}

GET /v1/runs/{id}/feedback                   — per-run history (bare array)
GET /v1/agents/{name}/feedback               — aggregate summary
  → { avgScore, thumbsUp, thumbsDown, trend7d: [...] }

Rehearsals

Replay past failed or low-scoring runs as synthetic test cases against a candidate version before you flip production traffic. Uses the same CI-gate baseline machinery as eval runs.

POST /v1/runs/rehearse
{
  "agentName": "research-agent",
  "candidateVersion": "v3",
  "window": "7d",   // look-back window for source runs
  "limit": 20,      // max synthetic cases to generate
  "maxScore": 2     // only pull runs with feedback score ≤ this
}

OTel traces

Every HTTP request and gRPC call carries a span enriched with lantern.tenant_id, lantern.run_id, lantern.step_id, and lantern.user_id via middleware.EnrichSpan. Traces are no-op safe when LANTERN_OTEL_ENABLED is unset. Export to any OTel backend by setting OTEL_EXPORTER_OTLP_ENDPOINT.