Evaluations

An eval suite is a set of inputs with expected results for one agent. You run it like a test suite, pin today’s score as the baseline, and the API answers HTTP 422 when a later run scores lower. Everything here is on the REST API the dashboard uses; the exhaustive list is in the REST API.

Budgets, forecasts and the eval gateTwo gates keep agents safe to ship: spend is checked before a run starts, and quality is checked before a change merges.

Under budgetOver budgetEval passesRegression

Swipe sideways to pan.FitShow steps

⏮◀▶ Play▶

Scrub through steps1 / 4

1. Forecast

Optional, and cheap: what will this run cost, and would it break a budget?

Suites

A suite is a named list of cases. A case has a name, an input string, and an expected string and/or an assert object. Suites are upserted by (tenant, agent, name).

PAL stores expected and assert with the case. Your runner (or pal test) decides pass or fail and reports passed and score per case. The one assertion the server checks itself is {"grounded":true}, which forces a case to fail when the output claims an action nothing backed.

POST /v1/eval-suites
{
  "agentName": "research-agent",
  "name": "factual-accuracy",
  "cases": [
    { "name": "case-1", "input": "What year did WWII end?",
      "expected": "1945", "assert": { "contains": "1945" } }
  ]
}

Also GET /v1/eval-suites (optionally ?agentName=), GET /v1/eval-suites/{id} and DELETE /v1/eval-suites/{id}.

Recording a run and the CI gate

Run the agent on each case yourself, or use pal test, which waits for each run to finish (--timeout caps the wait, default 2m). Then post the per-case results. The server computes the score (mean of case scores) and compares it with the pinned baseline for the same agent and branch. A score more than 0.01 below the baseline returns 422, which fails a CI step. With no baseline pinned, nothing regresses.

POST /v1/eval-runs
{
  "suiteId": "suite-uuid", "agentVersion": "v2", "commitSha": "abc1234", "branch": "main",
  "caseResults": [ { "name": "case-1", "passed": true, "score": 0.95, "actual": "1945" } ]
}

200 OK
{ "id": "run-uuid", "passed": true, "score": 0.95, "regressed": false, "baselineScore": 0.94 }

422 Unprocessable Entity   (regressed against the baseline)
{ "id": "run-uuid", "score": 0.91, "regressed": true, "baselineScore": 0.94 }

List runs with GET /v1/eval-runs?suiteId=&agentName=&branch=. A full pipeline (run, post, stop on 422, deploy, receipt) is in Examples.

Baselines

Pin an eval run as the baseline for an agent and branch; later runs on that branch are compared against it.

pal test --agent research-agent --suite factual-accuracy --set-baseline   # run, then pin this run

Over REST, POST /v1/eval-baselines with { "agentName", "branch", "evalRunId" } pins one, and GET /v1/eval-baselines?agentName=&branch= reads it (404 {"error":"no baseline"} when unset).

Reading the history

Two read-only endpoints over existing eval_runs data feed the dashboard’s quality-trend sparkline and top-failing-cases view.

Endpoint	Returns
GET /v1/eval-observability/failures?agentName=&branch=&limit=	clusters: failing cases grouped across recent runs, worst first (case, failures, seen, failRate, sampleError, firstSeen, lastSeen), and runsScanned. limit is the number of runs scanned (default 50, cap 200)
GET /v1/eval-observability/trends?agentName=&branch=&limit=	points, oldest first (runId, createdAt, score, passRate, passed, costUsd, agentVersion, commitSha), plus latestVsMean and regressing (the latest score is below the mean of earlier ones)

Human feedback and rehearsals

Collect reactions on individual runs. Score is 1 to 5; 4 to 5 is positive, 1 to 2 is negative.

POST /v1/runs/{id}/feedback
{ "score": 2, "comment": "The agent missed the follow-up question",
  "preferredOutput": "It should have asked for clarification first" }

GET /v1/runs/{id}/feedback lists a run’s history; GET /v1/agents/{name}/feedback returns the aggregate (totalFeedback, avgScore, thumbsUp, thumbsDown, last7DaysAvgScore).

Rehearsals pull past failed runs and runs scored 2 or lower as synthetic test cases, so you can replay them against a candidate version before you flip traffic. The endpoint returns the cases; pal test --rehearse runs them.

POST /v1/runs/rehearse
{ "agentName": "research-agent", "window": "7d", "limit": 20,
  "includeFailures": true, "includeLowScore": true }

Defaults: window 7d, limit 25 (max 200), both include* flags true.

A/B experiments

An experiment records scores for two agent versions and can promote the winner. PickVariant implements a deterministic FNV-1a split, but nothing in the run path calls it, so experiments do not route traffic today. You assign variants yourself and record each outcome.

POST /v1/experiments
{ "agentName": "research-agent", "name": "prompt-v2-test",
  "variantAVersion": "v1", "variantBVersion": "v2",
  "trafficSplitB": 20, "autoPromote": true, "minRunsToPromote": 100 }

POST /v1/experiments/{id}/record      { "variant": "b", "score": 0.95 }     # running mean per variant
POST /v1/experiments/{id}/conclude    { "winner": "b", "promote": true }    # "a" | "b" | "tie"

trafficSplitB is stored, not applied. With autoPromote on, PAL promotes the leader once both arms have at least minRunsToPromote runs (default 100) and one leads by more than 0.02. That is a fixed margin, not a significance test. GET /v1/experiments and GET /v1/experiments/{id} list and read them.

Span attributes and traces: Observability.

CI gates for your agent repo

pal ci init writes a GitHub Actions workflow and a generic .pal/ci.sh that run, in order, manifest lint, a policy/security lint, the hermetic eval suite with the baseline regression gate, and a budget check. All pass/fail decisions are deterministic code; a model never grades the gate. The run writes pal-ci-report.json and a PR comment.

pal ci init --suite ci --dry-run   # preview the files
pal ci init --suite ci             # write them
pal ci lint                        # the two offline gates, no credentials

Set the repo secret PAL_API_KEY and variables PAL_API_URL and PAL_CI_STAGING_ACK=1. The key must belong to a staging workspace: the gate deploys the PR’s manifest there before evaluating it.