The cache contract

Prompt caching is the largest single lever on an agent's bill, and it is all-or-nothing: rewrite one byte at or before the provider's cache boundary and the entry for the whole prefix is discarded. This page states, precisely, what Distil guarantees about the bytes it forwards across turns — and what it does not. Every clause is enforced by tests/test_cache_contract.py, against all three provider shapes.

This is the contract. Prompt Caching is the operational guide — how to mark a prefix, what distil cache reports, and how to read a miss. Start there if you are debugging a bill; start here if you are asking what Distil promises.

Why this is written down

Distil has been on the wrong side of this once. A recency carve-out that counted back from the end of the message list slides forward as a conversation grows, so every block was protected while fresh and rewritten one turn later — after the client had already committed it to a cached prefix. Measured against the real API, that produced zero cache reads on every turn and 2× the cost of compressing nothing at all: the entire prefix was re-written at 1.25× instead of re-read at 0.1×.

The fix was to anchor the carve-out to the client's own cache_control breakpoint rather than to the end of the list. The reason it needs a contract and not just a fix is that the failure is silent. Every request still succeeds. The savings report still shows tokens removed. Nothing raises and nothing 5xxs. The only symptom is the invoice.

Where the boundary is

The contract is stated relative to the index the provider has cached through, and that differs by provider:

The empty carve-out on OpenAI and Gemini is a decision, not an oversight. Those providers cache what they are sent the moment they are sent, so a window counted back from the end would digest, one turn later, content already committed. _recent_chat_verbatim_indices and its siblings return the empty set for exactly that reason, and say so in their docstrings.

The binding form of the contract uses a high-water mark rather than the current turn's boundary: once the provider has cached through index i, rewriting i on any later turn invalidates the entry, even if this turn's marker sits earlier.

What is guaranteed

(a) Prefix stability

For every message the client re-sends byte-identical to the previous turn, Distil forwards it byte-identical, at every index at or before the cache boundary.

(b) Compression touches only the volatile suffix

Anything that does change lies strictly after the boundary. Two mechanisms enforce this and both are load-bearing. The recency carve-out anchors to the breakpoint. And query-aware salience is scoped to blocks after it — intent terms come from the newest user turn and change every turn by design, so letting them choose which lines survive inside an already-cached block would rewrite that block on every new question. That is the same cache bust from a second direction, and it is invisible to any single-request test.

(c) Digest determinism

A digest emitted for a block at turn N is byte-identical when the same block is forwarded at turn N+1. Handles are content-addressed — sha256(text)[:8], with no per-request nonce, counter, or timestamp. This was checked in the code rather than assumed, and it is asserted on its own, because a random handle would rewrite every digest stub on every turn and bust the cache by itself while every other clause still passed.

(e) Prefix determinism — the other way to satisfy (a)

Clause (b) is how most of Distil stays inside the contract, but it is not the only way and it is not always available. Under the client shape that actually bills — Claude Code pins its newest turn, so the entire history is cached — there is no volatile suffix, and a transform gated on one would simply never run.

The re-read delta takes the other route. Its plan is a pure function of the message prefix: what a block encodes to depends only on the blocks before it, never on where the boundary currently sits or on how long the conversation has grown. So the block's bytes never change at all, which is strictly stronger than (a) asks for — there is nothing to invalidate because nothing is ever rewritten. Cache-delta coding relies on the same construction. The recency carve-out still applies on top — the freshest tool output is never elided — and because that carve-out is itself anchored to the breakpoint, the one rendering change it causes lands strictly after the boundary, exactly as clause (d) describes.

A gate on the boundary would have been worse here, not safer: it flips a block from stub to verbatim exactly as the boundary advances past it, which is the failure clause (a) exists to catch. The decision record is ADR 0010.

(f) A canonically-equal prefix is forwarded as previously sent

Real agentic clients rewrite their own history on every turn without changing a token the model reads: the cache_control breakpoint advances to the newest block, an SDK stamps positional index fields, a string becomes a single text block or back again. Clause (d) below used to end the story there — the input is not the same input, so no promise — and that cost the whole prefix. Measured offline before the fix, a client doing nothing but renumbering an index field forwarded 0% of its prefix byte-identical, on every provider, on every turn.

So Distil remembers the previous turn's forwarded bytes per conversation lineage, and for the longest canonically-equal leading prefix it forwards the bytes it forwarded last turn rather than the bytes it would produce now. The canonical comparison ignores cache_control, index, the interchangeable spellings of a single text block, and JSON key order. It ignores nothing else; a tool payload — the arguments going out, the result coming back — is compared exactly as it arrived, because a key called index one level down inside a tool result is data, not SDK bookkeeping. The client's current breakpoint markers are re-placed at the client's current block positions, because the marker delimits the cached span and is not part of it.

The guard that makes this safe is narrower than it looks. Replay restores bytes, never decisions: a message is replayed only when what Distil would send this turn is itself canonically equal to what it sent last turn. Distil's compressor is not a pure function of one message — the exact-quote guarantee keeps a tool result verbatim because of an edit that arrives later in the list — so overlaying an older stub there would break the agent's next edit to buy a cache hit. That trade is not made. distil validate carries the invariant.

On by default in all three servers — the threaded proxy, the async proxy, and the multi-tenant gateway, which scopes the lineage per tenant. --no-prefix-replay forwards exactly what the compressor produced. Any exception falls open to the same. The decision record is ADR 0011.

What is not guaranteed

• A client that rewrites its own history semantically. If the client edits content, re-orders messages, or changes a tool call, the input is not the same input and the contract does not apply. Distil cannot make an unstable client stable. Non-semantic churn — a moved marker, an added index, a re-spelled text block, a re-ordered JSON object — is now repaired by clause (f); a real edit is still a real edit, and replay stops at it, at exactly its index.
• Provider TTL. Cache entries expire on the provider's schedule. Byte-stability is necessary for a hit, not sufficient.
• A client that sends no cache marker at all. With nothing marked there is no Anthropic prefix to protect, so the plain last-k recency window applies and a block does go verbatim on one turn and digested on the next. Nothing is being invalidated, so this is intended. It is bounded rather than unbounded, and the test asserts the bound in both directions: the churn must occur strictly after the boundary, and it must still occur at all — a silently-dead carve-out fails too.

How it is enforced

tests/test_cache_contract.py replays growing six-turn synthetic sessions through the same public entry points the proxy calls — compress_messages, compress_chat_completions, compress_responses_input, and compress_generate_request — and fails on any same-input byte drift at or before the boundary. Three Anthropic marker placements are covered: pinned at the head, moved forward each turn, and absent entirely.

Every transform that touches cross-block state is replayed the same way. The re-read delta gets its own assertion under the moving-marker (fully-cached) shape in tests/test_reread_delta.py, plus a test that fails if it stops firing — a silently-dead transform must not pass as a silently-safe one.

Clause (f) is driven the same way, with a control arm: each of the three rewrite shapes is replayed against every provider both with replay on and with it off, and the test fails unless the rewrite demonstrably shortened the byte-stable prefix without replay and demonstrably did not with it. A stability assertion whose control never destabilises proves nothing. benchmarks/prefix_replay_stability.py reports the same measurement as a table, offline and with no API calls.

Measured at adoption, the contract holds on all three providers. That includes the realistic Claude Code shape, where the client pins its newest turn so the entire history is cached: there is no uncached tail in that configuration and nothing moves at all. No bug was found. The test codifies a property that was true and undefended.

Checking it against the provider

The clauses above are what Distil intends. The provider's own accounting is what actually happened, and distil dissect now reports it per session:

cache-read share: 87.4% of billed input was served from the
                  provider's prompt cache (at ~0.1x)

Beside it, what Distil did to hold that share:

prefix replay: 812 messages forwarded as previously sent,
               96 compressed fresh; 143 client rewrites repaired

Repaired is the number that moved money — messages the client rewrote non-semantically and Distil forwarded as previously sent, so the prefix survived a rewrite it would otherwise have lost. Hits with zero repairs is the healthy steady state, not a dead feature, which is why the two are never added together. The line reads not recorded for sessions proxied before 1.52 or run with --no-prefix-replay.

The number is cache_read_input_tokens over total billed input, taken from the fields the ledger already recorded. A near-zero share on a long session means something rewrote the prefix, and it is the only signal that says so — the requests all still succeed either way.

When the records predate those fields the line reads not captured and the JSON field is null, never 0.0. “We did not measure this” and “the cache never hit” are opposite diagnoses and must not share a rendering. This matters for a specific shape: older sessions carry only the aggregate usage_cache_tokens and neither split field, so summing reads over them gives zero against a nonzero input total — a confident 0.0% for a session nobody measured. At least one record must carry a split field before the number is reported at all.

The decision records are ADR 0008 (the contract) and ADR 0011 (forwarded-bytes prefix replay). For the mechanics of what gets compressed, see Techniques; for the adversarial side of the same surface, see the Threat Model.