Reproduce the Numbers

Every headline number has a command. This page shows exactly how each one is produced — what is synthetic, what is live, and what we ran but chose not to ship as the default (E7). None of it requires you to trust our word; it requires you to run the script.

The three CI gates

Every push to main must pass all three gates before merging. They run in ~2 min combined, no API key.

Gate 1 — distil bench (offline decision-equivalence)

Runs the bundled 7-domain trajectory corpus through the decision-equivalence gate using the deterministic (structural) runner. The gate certifies that compression does not change the agent’s next action on any trajectory. This is the primary correctness gate.

# reproduce exactly — no key, no internet, ~5 s
distil bench

# or against the full 64-trajectory varied corpus (requires a clone)
python benchmarks/gen_corpus.py      # writes benchmarks/corpus_xl/ (seeded, deterministic)
distil bench --corpus benchmarks/corpus_xl

The deterministic runner is a structural stand-in: it checks whether the “decision-bearing” token sequence survives (not a live model call). This makes it reproducible by anyone in ~5 seconds. It is not a substitute for real-model grading — see Gate 3 and prove.py below for that.

Gate 2 — distil verify (byte-fidelity)

Checks that every Tier-0 and Tier-1 operation in the corpus is byte-reversible: compression followed by decompression returns the exact input bytes, and frozen history never mutates between turns. A failure here means a lossless claim is false.

# verify byte-fidelity across the bundled corpus
distil verify

# against a custom corpus
distil verify --corpus benchmarks/corpus_xl

Gate 3 — distil validate (adversarial real-path)

Drives the actual compression + proxy path against 12 hostile inputs (huge log blocks, unicode surrogates, deeply nested JSON, handle-injection strings, secret-looking content, malformed/None tool results, empty blocks) and asserts five load-bearing invariants on each:

12 cases × 5 invariants = 60 checks. All 60 must pass.

# run the adversarial gate
distil validate

Source: distil/harness.py. The gate is separate from distil verify (corpus byte-fidelity) and distil bench (non-inferiority): it exercises the code paths a corpus never hits — hostile inputs, streaming, marker injection.

Real-model grading — prove.py

benchmarks/prove.py removes the circularity of the deterministic runner by grading real agent traces with a real model. It runs four experiments (E1–E4) against actual trajectories:

Four grading backends are supported:

# offline plumbing check only — no key, no download
python benchmarks/fixtures/make_fixtures.py
python benchmarks/prove.py --dataset fixtures --runner smoke --alpha 0.2

# real-model grading via your Claude Code subscription (no API key)
python benchmarks/prove.py --dataset tau --path runs.json \
    --runner claude-cli --model claude-haiku-4-5-20251001 --samples 3

# local open model via vLLM (zero per-call cost)
# vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000
python benchmarks/prove.py --dataset swe --path swe_trajs/ \
    --runner openai --base-url http://localhost:8000/v1 \
    --model meta-llama/Llama-3.1-8B-Instruct

The smoke runner is a non-evidential stand-in for plumbing checks. It is not evidence about real agents. See benchmarks/PROVE.md for the full runner table and what each backend actually proves.

The head-to-head corpus

The head-to-head runs on 64 trajectories across 8 families, generated deterministically with a fixed seed so the numbers reproduce exactly. The corpus spans both regimes deliberately so no tool gets a home-turf advantage:

• Structured / repetitive: JSON record arrays, SQL rows, metrics logs — where structural compaction pays off.
• Diagnostic / prose: Kubernetes incidents, stack traces, RAG chunks, support transcripts — where conservative tools protect content and lossy methods mangle it.

Decisions are buried inside large tool outputs, as in real agents, so naive head/tail truncation drops them. A large reference document is re-read every turn (the recurring tool output prompt caching can’t reach), which distil’s cross-turn dedup collapses.

# generate the corpus (idempotent, seeded at 42)
python benchmarks/gen_corpus.py      # writes benchmarks/corpus_xl/

# run the head-to-head — distil + structural baselines, deterministic runner
distil bench --corpus benchmarks/corpus_xl

# add a real external compressor (list[str] -> list[str] over block texts)
PYTHONPATH=. distil bench --corpus benchmarks/corpus_xl \
  --external benchmarks.headroom_adapter:compress:Headroom

# live model grading
export ANTHROPIC_API_KEY=sk-ant-...
distil bench --corpus benchmarks/corpus_xl --runner anthropic --tokenizer anthropic

The full live head-to-head (distil vs. llmlingua 0.2.2 vs. headroom-ai 0.27.0, graded by claude-opus-4-8 majority-of-3) ran on 2026-07-05 and its raw output is committed at docs/paper/results/derc_live_compare.2026-07-05.log. Reproduce it:

pip install headroom-ai llmlingua
python benchmarks/derc_live_compare.py

The invariant scorecard

The same five invariants distil enforces on itself (distil validate) can be run against any compressor adapter. benchmarks/scorecard.py does this and produces a side-by-side table. It runs offline with no extra installs; headroom and llmlingua columns appear automatically if the packages are installed.

python benchmarks/scorecard.py      # prints table, writes benchmarks/scorecard.json

What the current run produces (distil + structural baselines; external adapters skipped when not installed):

                    distil  truncate@500  recency-w@500  keep-last-3  recomp-extr  sel-context
  ----------------  ------  ------------  -------------  -----------  -----------  -----------
  reversibility     PASS    FAIL          FAIL           PASS         FAIL         FAIL
  reject-if-bigger  PASS    PASS          PASS           PASS         PASS         PASS
  recency-exact     PASS    PASS          PASS           PASS         PASS         FAIL
  fail-open         PASS    PASS          PASS           PASS         PASS         PASS
  content-free      PASS    n/a           n/a            n/a          n/a          n/a

Fairness semantics used per cell:

• reversibility — Behavioral: does the compressor provide a recovery path? Distil stores originals in a local RestoreStore keyed by 8-hex handles embedded in the output; the harness confirms every handle recovers its exact bytes. Structural baselines are observably irrecoverable (truncated bytes are gone) → FAIL. keep-last-3-turns targets HISTORY blocks, not TOOL_OUTPUT; it is a no-op on the tool-result content used here → trivially PASS (no bytes changed).
• content-free — Architectural: the invariant tests whether on-disk telemetry holds zero prompt/tool text. Structural baselines and library-only adapters have no on-disk state at all; the invariant is inapplicable, not failing → n/a. This follows the rule: grade observable behavior only; an invariant a compressor never claimed (because it has no telemetry to check) is not scored as FAIL.
• sel-context recency-exact FAIL — Honest finding: the selective-context token-level pruner drops the recency sentinel when scoring salient tokens. This is the exact failure mode distil’s recency carve-out exists to prevent.

The negative result: E7

E7 is an internal SWE-bench Verified end-to-end experiment (benchmarks/swe_bench_e2e/) that ran the full agent loop — not a single-step proxy — under distil’s aggressive lossy compression tier. The result: 52% → 16% task resolution. The reversible tier held (56% vs. 52% full-context baseline). Per-step decision-equivalence had passed; end-to-end task success still cratered under the aggressive tier.

We publish E7 as evidence, not confession. It is the concrete reason the shipped default is the conservative/reversible tier rather than the tier that scores best on a raw compression-ratio leaderboard. The trajectory-risk certificate and the fail-safe gate exist because of E7.

# E7 harness (requires SWE-bench Verified setup)
bash benchmarks/swe_bench_e2e/run_all_agents.sh
bash benchmarks/swe_bench_e2e/run_all_scores.sh
python benchmarks/swe_bench_e2e/aggregate.py

Full methodology, the A/A nondeterminism baseline, trajectory-risk certificates, and what distil’s numbers do and don’t prove: docs/EVALUATION.md.

Synthetic vs. live: what each number actually is