local-first · zero telemetry · Apache-2.0

Open weights are free.
Running them properly is the hard part.

One command. You never write a config. Agents drive it natively. Between "the weights are on Hugging Face" and "it's serving production" sits a project — clickllm collapses it into one decision, and prints the arithmetic behind every number in it.

91%the interval, not the point96%
your traffic, not a benchmark a rollback button, not a leap every number drillable to the prompt
clickllm — ~/payments-api 
$ clickllm fit --context 32k --concurrency 8   M4 Max · 128 GB · 546 GB/s       usable: 96 GB   model                 quant  total   ~tok/s  licence  ─────────────────────────────────────────────────  Qwen3 30B-A3B (MoE)   q8     56.2G   119     Apache-2.0  Phi-4 14B             q8     66.3G   27      MIT   NOT FEASIBLE  Kimi K3 (2.8T MoE)    weights alone need 1,467 GB at q4 —                        sparsity cuts compute, not memory $ clickllm guard receipt.json --traffic mix.json  RECEIPT VOID · 17cfed178112    [void] gpt-5: fingerprint changed behind its name    [void] workload: 45% of traffic in shapes never scored  → re-prove now: the model you proved is not the    model in production 
948
tests · Rust + Python
0
bytes of telemetry, by design
9
diagrams, palette-validated
5
engines planned for and emitted

The three ways a migration dies

Everyone can run the model.
Nobody can prove it's safe.

Deployment is solved six times over. The gap is the decision — and it is usually made on a vibe, a leaderboard, or a dashboard that has never seen your traffic.

📊wrong data

A benchmark isn't your workload

MMLU says nothing about whether a model can extract line items from your invoices. clickllm builds the eval set from traffic you actually served — redacted before it touches disk, clustered by task shape.

🎲no confidence

A score without an interval is a guess

"94%" over eleven samples is noise wearing a number. Every cell carries a Wilson interval, and a cell with too little evidence shows ? rather than a fabricated score.

🚪no way back

A cutover with no rollback is a bet

Shadow first, always. Rollback is automatic and easy to trigger; advancing is only ever a proposal a human applies. The gateway re-derives that rule from the numbers, so buggy automation still cannot escalate.

Prove it, then move it

Seven stages. Each one can say no.

Observe your traffic, distill it into task clusters, size what fits, prove equivalence, deploy native config, cut over behind a gate — then keep checking that the proof still holds.

The loop is the product. Everything else is an implementation detail of one of these stages.
🧾portable

Proof is an artifact, not a dashboard

A receipt is a file: every claim with its interval, the bar it was measured against, the judge and its human-agreement rate, and — required, never optional — the clusters that did not pass. Re-run the eval set and the digest must match. A signature says we said this; reproduction says and it's true.

🛡️keeps checking

Three kinds of stale, not one

The model changed behind its name → your proof is void. Your traffic moved → the eval set answers questions nobody asks. Something new shipped → your proof is still true. Only the first two mean you no longer know whether production is adequate, and collapsing them is how a watchdog gets muted.

The auditable core

Three ways sizing goes wrong.
All three are common.

Every number is arithmetic you can check with --explain. Getting any of these wrong rejects a model that fits, or accepts one that doesn't.

TrapWhat people doWhat it costs
MoE Size on active parameters, because that's the headline number. Weights need total params resident. Sparsity cuts compute, not memory — a 2.8T MoE needs 1,467 GB at q4 whatever its active count.
GQA Use attention heads in the KV formula instead of KV heads. Up to an 8× overestimate. The model fits; the tool says it doesn't.
MLA Apply the GQA formula to a DeepSeek-family model. ~50× overestimate. MLA stores a compressed latent, not per-head K/V.
KV cache per stream against context length for MHA, GQA and MLA on a logarithmic axis
Log scale on purpose: MLA is roughly fifty times smaller than MHA, and a linear axis would render three of four series as one flat line.
One engine working out of 132 — because a single word costs a read of the entire model, and batching only gets you to eighteen people before the cache fills.

Any catalogue entry with kv_scheme: mla must set kv_lora_rank — enforced by a test, because the failure is silent and the arithmetic looks fine either way.

Engine and flags, derived

Batch and real-time are tuned in
opposite directions.

A stack tuned for throughput and one tuned for tail latency disagree on almost every knob, and the defaults ship somewhere in between. Both are "running vLLM". Only one is running it correctly.

🧩picks the engine

Prefix sharing

Above 55% shared prompt tokens, RadixAttention reuses that prefix's KV instead of recomputing it per request — structural, and no vLLM flag reproduces it. Below, PagedAttention is the better-supported default.

🎯a function of load

Speculative decoding

Drafting is a bet that spare compute exists. At high batch there is none, the verify pass competes with real tokens, and throughput drops. Derived from concurrency — never a checkbox.

⚖️has a cost

fp8 KV cache

Halves KV memory — offered only when KV is what actually binds. It is the one knob with a real accuracy question, and the reason says so: vLLM publishes no quantitative claim. Prove it before you trust it.

Throughput change from speculative decoding against batch size, positive at low batch and negative beyond about 32
The quantity has a sign. That is why it is derived from your concurrency rather than offered as a switch.

Every engine flag is verified against published docs, never recalled. That rule exists because breaking it shipped a bug: --guided-decoding-backend had been renamed in vLLM, so every structured-output config generated was unrunnable. Where a flag can't be confirmed, the adapter reports a gap rather than a plausible guess.

Inference in a box

It is not a container.
A container can't reach Apple's GPU.

The box is an artifact plus a binding decision made on arrival — which is why the interesting content is the three outcomes, not the packaging.

One artifact landing on three machines and producing three outcomes: run as packed, re-solved with the changes reported, or refused
Run as packed, re-solved with every change reported, or refused. The re-solve is the product — anyone can ship an artifact.

Inference from first principles

If you've never sized a KV cache,
start here.

Nine diagrams on a palette validated for colour-vision deficiency against the real surface — computed, not chosen by eye. The full course, with the arithmetic, is in the docs.

Stacked memory use for a 32B model as concurrency rises from 1 to 32
Where the memory goes. Weights are a constant you can look up. The KV cache decides how many people you can serve — and it crosses the 80 GB line at 32 users, not the model.
Prefill vs decode. A long prompt with a short answer is a prefill problem; a short prompt with a long answer is a decode problem. Tuning the wrong one buys hardware that doesn't help.
Continuous batching. Same tokens, same GPU. The shaded time is what static batching throws away — the single biggest throughput win in modern serving.
Weight size for a 32B model at 16-, 8-, 6- and 4-bit
Quantization buys bandwidth. Decode re-reads every weight for every token, so a smaller format is directly faster — not just smaller.
Memory per TPU host for v5e, v6e and v5p against an H100 and an 8-way H100 node
TPUs. Per-chip memory is the number usually quoted and the wrong one to plan with. Multi-host serving is unsupported on GKE, so one host is a ceiling, not a starting point.
The degradation ladder: quantisation reduced first, then context halved repeatedly, then refusal below a 2k floor
Degrading honestly. Give something up rather than nothing — and name what was given up. Serving 8k when it was built for 32k is fine if you know, and an incident if you don't.

What the dashboard won't tell you

Telemetry that grades the plan
that produced it.

A planner emits a config on assumptions. Then nobody checks. clickllm reports only the predictions your deployment falsified.

clickllm — engine, right now 
  candidate-local   KV near full — scheduler is preempting
  KV cache   ████████████████████████░░  94%
  prefill    ██████░░░░░░░░░░░░░░░░░░░░  25%  decode-bound — tune bandwidth

  running 14  waiting 6  queued 60ms  TTFT 210ms  e2e 640ms
  prefix hits 62%  draft accepted 41%  truncated 12%  abandoned 3%

  THE DEPLOYMENT DISAGREES WITH THE PLAN
  │ speculative decoding was enabled assuming idle compute, but only
  │ 41% of drafted tokens are accepted. Below ~60% the draft pass costs
  │ more than it returns — turn it off or change the drafter.
  │ 12% of completions stop at the token limit rather than finishing.
  │ That reads as a quality problem and is almost always max_tokens.
  │ the card is at 98% memory but the engine's KV cache is only at 60%.
  │ Roughly 38% is held by another process — engine telemetry alone
  │ cannot show you this.

Every field is optional and renders as — when unreadable. A KV utilisation of "0%" reads as plenty of headroom, which is the exact opposite of we could not tell.

No login, no tokens, no account

Try it in ten seconds.

The sizing solver has zero runtime dependencies and runs offline. Nothing leaves your machine — captured traffic is the most sensitive data you have, and export is an explicit command, never a background sync.

The package is clickllm-cli; the command is clickllm. PyPI refused the bare name as too similar to the existing click-llm, so pip install clickllm and uvx clickllm both fail — the --from is not optional. npm allowed the name PyPI refused, so there the package and the command are both clickllm, and npx clickllm works. There is no Homebrew tab because there is no clickllm formula in the tap; a tab advertising one would be the same defect as before.

Then ask the inverse. clickllm where deepseek-v3 --context 128k answers what would I need to run this? across 20 hardware classes — including TPU v5e, v6e and v5p, sized per host because a TPU is not rented by the chip.