LlamaIndex × Distil

LlamaIndex's own OpenAI/Anthropic LLM classes take an api_base like any OpenAI/Anthropic-compatible client, so the proxy route needs no code beyond that. A second, in-process module also ships for teams that don't want a sidecar at all — including a node postprocessor for compressing retrieved context.

Setup — the proxy route

Start the proxy against the provider this SDK talks to, then point the LLM at it. Nothing else in your code changes.

$ pip install llama-index
$ distil proxy --port 8788 --upstream https://api.openai.com &

from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-5", api_base="http://127.0.0.1:8788/v1")  # ← the only change
query_engine = index.as_query_engine(llm=llm)

In-process — no sidecar

distil.integrations.llamaindex is duck-typed and never imports llama_index, so it costs nothing to have installed either way. Three seams, matching LlamaIndex's own shapes (verified against the official API reference and the llama-index-core source):

from distil.integrations.llamaindex import DistilNodePostprocessor, DistilLLM, compressing_tool

# 1. Compress retrieved nodes before they reach the LLM's context window
query_engine = index.as_query_engine(
    node_postprocessors=[DistilNodePostprocessor()],
)

# 2. Compress a tool's return value before an agent sees it
def get_weather(city: str) -> str:
    return "... huge forecast ..."

tool = FunctionTool.from_defaults(fn=compressing_tool(get_weather))

# 3. Or compress every outgoing chat/completion call, transparently
llm = DistilLLM(OpenAI(model="gpt-5"))
query_engine = index.as_query_engine(llm=llm)

DistilNodePostprocessor drops straight into node_postprocessors=[...]: it exposes postprocess_nodes/apostprocess_nodes, the two methods a query engine actually calls, so nothing needs to subclass BaseNodePostprocessor. Retrieved node text gets the reversible Tier-1 digest, the same as tool output elsewhere in this package. DistilLLM hands back your LLM re-typed as a transparent subclass of its own class, with only the chat/completion methods overridden and every other attribute (metadata, callback_manager, ...) untouched — so isinstance still holds, which is what resolve_llm() and every Pydantic llm: LLM field actually check; chat messages and completion prompts get Tier-0 lossless, and the model's own assistant-role replies are never rewritten. Pass verbatim=True to any of the three for Tier-0 only.

What actually happens

The proxy intercepts only the compressible paths — /v1/messages, /v1/chat/completions, /v1/responses and the Gemini generateContent routes. Everything else passes through untouched, and your API key travels in the request headers exactly as normal: the proxy never logs or stores it.

Large tool results and retrieved nodes are replaced by reversible digests carrying a content handle, and the original stays on your machine. The agent can pull any of it back mid-task through the distil_expand tool, so nothing is permanently discarded — which is what lets the compression be aggressive without being a gamble.

Verify it is actually routing

The most common failure is silent: the client never reaches the proxy and everything still works, just uncompressed. Two checks:

$ curl -s localhost:8788/distil/health
{"status":"ok"}

$ distil dashboard          # live savings; zero here means nothing is routing

Every response also carries x-distil-tokens-saved, so a request that went through the proxy is identifiable from its headers alone.

Full matrix of every supported SDK, including the in-process hooks that need no proxy: Integrations. Hosting your own LLM-facing endpoint instead of using an SDK? See the ASGI middleware.