Code skeletons from a real parse

When a tool result is source code, the reversible tier keeps its structure — imports, types, every function signature — and folds function bodies behind a handle the agent can expand. Python has always used the standard library's ast. Other languages used a brace-counting heuristic; with the optional [code] extra they get a real parse.

$ pipx install 'distil-llm[code]'     # or: pip install 'distil-llm[code]'

What changes

• Languages: Go, Rust, Java, C, C++, Ruby, TypeScript, JavaScript. Ruby is new ground: it has no braces, so the heuristic never saw it.
• Bodies elided, signatures kept. A function body becomes ...; the signature line, doc comments above it and the closing line stay. A body that holds a named function stays open so that name stays visible — a factory, a describe wrapper or an IIFE module does not swallow the file. Callbacks and closures go with their body.
• Conservative on errors. A subtree the grammar could not parse is left whole, and a text that is mostly parse errors is treated as not this language.
• Recoverable, deterministic. The byte-exact original sits behind the same handle as every other digest; same input, same skeleton.

Dependencies

The extra installs the official per-language tree-sitter grammar wheels: prebuilt, bundled, nothing downloaded when a file is parsed. tree-sitter-language-pack was considered and not used: its 1.x series fetches grammar bundles from the network on first use and needs Python 3.10, and distil runs on 3.9 and offline. Without the extra — or for a language it does not cover — distil uses the heuristic exactly as before.

Measured

On eleven real source files (eight pinned public files across the eight languages plus three of this repo's own), the skeletons went from 26.0% to 40.4% smaller than the source, and every named function's signature line survived; the heuristic dropped signatures on the Java and C++ files and could not skeleton Rust's string.rs or any Ruby at all. On the bundled corpus the bench, verify, validate and retention gates print identical results with and without the extra: its trajectories carry no non-Python source, so this path is not exercised there. Artifact: benchmarks/results/skeleton_treesitter.json; reproduce with:

$ uv run --extra code python benchmarks/skeleton_treesitter.py