Skip to content
docs

Tech stack

Running other people’s agents is three jobs. Coordinate (who owns what, what ran, who is billed): Go, for its database and cluster libraries. Stay fast on every request (route tokens, hold sandboxes): Rust, with no garbage-collector pauses. Be easy to build against: TypeScript, where the AI ecosystem lives. One language does not win all three, so each layer gets its own, and protobuf fixes the contract between them.

PAL Runtime: containersEvery deployable service, grouped by tier. Hover or tap a box for what it owns.
Gateway:8443 · Rust/Axumcontrol-planeREST :8080 · gRPC :50051notifierbillingschedulercronmodel-router:50053Providersdata-plane-agentoutbound tunnelworkflow-engine:50052runtime-scheduler:50055runtime-manager:50054harnessPID 1 in the VMPostgresruns · journal · RLSRediscache · pub/sub · locksS3 / MinIObundles · snapshots
Swipe sideways to pan.
  • Hot path or API
  • Host
  • Inside the VM
  • Store

The rule

Section titled “The rule”

Pick the right tool per layer. Do not unify for unification’s sake. Adding a language requires an ADR (ADR 0001), and the bar is high: each language adds a CI matrix, a set of security advisories and something new for contributors to learn.

Languages and the job each holds

Section titled “Languages and the job each holds”
Layer Language Why this one
Control plane, workflow engine, scheduler, memory, billing Go 1.26 Kubernetes is written in Go, so every cluster library is first-class. It compiles to one static binary and has a mature gRPC and Postgres ecosystem. Goroutines suit work that mostly waits on a database or another service.
Gateway, model router, runtime manager, surface gateway Rust 2024 The hot path. With no garbage collector there are no unpredictable pauses while a token stream is in flight, and memory is freed deterministically for a process babysitting many sandboxes. Firecracker is itself Rust.
Dashboard, landing page, docs site TypeScript The dashboard is Next.js 15 (React Server Components and streaming), so the run waterfall can render tokens as they arrive. It shares a language with the primary SDK, so types flow through.
Primary SDK TypeScript Where the agent ecosystem already lives.
Secondary SDKs Python 3.11+, Go Python for AI and ML users, Go for infra users.
CLI (pal) Go / Cobra A static binary that cross-compiles to every OS and reuses the services’ gRPC client.
API contracts protobuf3 One source of truth for types that cross a service boundary. It generates Go and TypeScript, so a field rename cannot drift between two services.

Terms used across these pages

Section titled “Terms used across these pages”
Term In one sentence
gRPC How the services call each other: a request with a schema both sides agreed on in advance, so a typo is a compile error.
protobuf The schema language for those calls. You write the message shape once and generate code for every language.
Firecracker A small, fast virtual machine monitor built by AWS. Its documentation cites a boot of about 125 ms. Untrusted code gets its own kernel, so escaping means breaking the hardware virtualization boundary.
gVisor A sandbox that implements the operating system interface in user space and forwards only safe calls.
Kata Containers Containers that are full virtual machines. Broader compatibility than Firecracker; one of the manager’s isolation backends.
Harness PAL’s Rust program, the first process inside every microVM. It hands out secrets, enforces the egress allowlist and streams logs and heartbeats home.
Control plane / data plane The part that decides (tenants, runs, budgets) and the part that executes (in your cloud account, where agent code and data live).
pgvector A Postgres extension that stores embeddings and finds similar ones, with no separate vector database.
OTel The vendor-neutral standard for traces and metrics. Every service emits it, so one run can be followed across services in your own monitoring tool.
RLS Postgres row-level security, which filters rows by tenant inside the database. Enforced in production whenever PAL_APP_DB_PASSWORD is set (opt out with PAL_RLS_ENFORCE=0), so isolation survives an application bug.

Data stores

Section titled “Data stores”

There are three, and adding a fourth requires an ADR.

Store Holds Why not something else
Postgres (with pgvector) Everything durable: tenants, agents, runs, the event journal, budgets, receipts, embeddings Transactions, JSONB and row-level security in one engine. pgvector avoids a separate vector database.
Redis Caching, rate limiting, queues and the pub/sub that pushes live run events to the dashboard This data may be lost on restart; putting it in Postgres would pay durability costs for something disposable.
S3 / MinIO Large blobs: agent bundles, microVM snapshots, attachments Big binary objects do not belong in a relational database. MinIO is the S3-compatible local stand-in.

One sandbox run, by process

Section titled “One sandbox run, by process”

What sits between pal run agent.yaml and a result. Every row is a process boundary.

# Process Language, port Does
0 CLI, SDK or dashboard Go, TypeScript Sends REST or gRPC with a bearer token
1 Control plane Go, :8080 / :50051 Authenticates, resolves the tenant, checks budget and quota
2 Runtime scheduler Go, :50055 Picks a node: warm pool, region, fair share, cost, health
3 Runtime manager Rust, :50054 Spawns the workload in the declared isolation class (Firecracker, Kata, K8s Job, Wasmtime, Docker)
4 Harness Rust, PID 1 in the VM Enforces the egress allowlist, vends short-lived secrets, streams logs and heartbeats up

Results stream back the same way, from harness to manager to control plane, then over SSE to the dashboard or SDK, and into journal_events as the durable record for replay, receipts and audit. Workflow runs add the workflow engine (:50052), the only writer of run state, and the model router (:50053). The full map is in Runtime internals, and the request path is in Request lifecycle.

The rules that hold it together

Section titled “The rules that hold it together”

Breaking one of these causes an incident, not a bug.

  • The control plane never touches your code. Only the runtime manager talks to Firecracker, Kata or pods.
  • One writer for run state. Services emit events; the workflow engine writes the outcome. No service updates the runs table directly.
  • Anything slow is durable. If it can take more than 100 ms or calls a model, it is a journaled, idempotent, replayable step.
  • Streaming never buffers. No service collects a whole response and then forwards it.
  • Untrusted code gets a microVM. If a node cannot provide the isolation, the manager refuses the workload instead of downgrading it.
  • Models are named by capability. Your code says reasoning-large; the router resolves a vendor model.
  • Multi-tenant by default. Every row and every call carries a tenant, and row-level security backs it below the application in production.
  • Secrets never appear in logs, traces or run state. Only a pal.secret/... reference travels, resolved at the moment of use.

What this costs

Section titled “What this costs”
  • Four language toolchains in CI. Every pull request runs golangci-lint, cargo clippy -D warnings and eslint with tsc. The pipeline is slower and the supply chain wider.
  • Contract changes are two-step. Touching a proto means regenerating and then fixing both the Go and the Rust side. The friction is what stops silent drift.
  • A wider contributor surface. The boundaries are narrow enough to work in one layer without holding the others in your head.
  • Rust is slower to write. It is used only where latency or memory determinism pays for it, never for CRUD.