Microsoft AutoGen × Distil

OpenAIChatCompletionClient takes a base_url like any OpenAI-compatible client, so the proxy route needs no code beyond that. A second, in-process module also ships for teams that don't want a sidecar at all.

Setup — the proxy route

Start the proxy against the provider this SDK talks to, then point the client at it. Nothing else in your code changes.

$ pip install "autogen-agentchat" "autogen-ext[openai]"
$ distil proxy --port 8788 --upstream https://api.openai.com &

import os
from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient

client = OpenAIChatCompletionClient(
    model="gpt-4o",
    api_key=os.environ["OPENAI_API_KEY"],
    base_url="http://127.0.0.1:8788/v1",  # ← the only change
)
agent = AssistantAgent("assistant", model_client=client)

In-process — no sidecar

distil.integrations.autogen is duck-typed and never imports autogen_core, so it costs nothing to have installed either way. Two seams, matching AutoGen's own shapes (verified against the official docs, autogen-core 0.4+/0.7+):

from distil.integrations.autogen import DistilModelClient, compressing_tool

# 1. Compress a tool's return value before it becomes a FunctionExecutionResult
async def get_weather(city: str) -> str:
    return "... huge forecast ..."

tool = FunctionTool(compressing_tool(get_weather), description="Get the weather")

# 2. Or compress every outgoing model call, transparently
client = DistilModelClient(OpenAIChatCompletionClient(model="gpt-4o"))
agent = AssistantAgent("assistant", model_client=client)

DistilModelClient delegates every attribute it doesn't wrap (model_info, capabilities, count_tokens, ...) straight to the real client, so it drops in wherever a ChatCompletionClient is expected. FunctionExecutionResultMessage content gets the reversible Tier-1 digest; SystemMessage/UserMessage get Tier-0 lossless; AssistantMessage — the model's own words — is never rewritten. Pass verbatim=True to either helper for Tier-0 only.

What actually happens

The proxy intercepts only the compressible paths — /v1/messages, /v1/chat/completions, /v1/responses and the Gemini generateContent routes. Everything else passes through untouched, and your API key travels in the request headers exactly as normal: the proxy never logs or stores it.

Large tool results are replaced by reversible digests carrying a content handle, and the original stays on your machine. The agent can pull any of it back mid-task through the distil_expand tool, so nothing is permanently discarded — which is what lets the compression be aggressive without being a gamble.

Verify it is actually routing

The most common failure is silent: the client never reaches the proxy and everything still works, just uncompressed. Two checks:

$ curl -s localhost:8788/distil/health
{"status":"ok"}

$ distil dashboard          # live savings; zero here means nothing is routing

Every response also carries x-distil-tokens-saved, so a request that went through the proxy is identifiable from its headers alone.

Full matrix of every supported SDK, including the in-process hooks that need no proxy: Integrations. Hosting your own LLM-facing endpoint instead of using an SDK? See the ASGI middleware.