Cache the
prefix.

Every request re-sends the same big preamble: a long system prompt, tool definitions, a document you keep asking about. The model recomputes its internal representation of all that from scratch each time — wasted work you pay for on every call. Prompt caching stores that computation for a prefix, so later requests sharing it skip the recompute and pay only a small fraction of the input price for the cached part.

Mechanically it's a reuse of the model's KV cache for the shared prefix tokens: the keys and values were already computed once, so a matching request reads them instead of recomputing. That's why the savings are real — cached tokens skip the expensive attention math. But the reuse only works when the prefix matches, and "matches" here is unforgivingly literal. Understanding exactly what counts as a match is the whole skill.

Here's the rule that governs everything: a prompt cache is keyed on the exact prefix. When a new request arrives, it's compared to a cached one token by token from the very start, and the cache applies only up to the point where they still agree. The instant a token differs, the cache stops — every token from that position onward is recomputed at full price, even if the rest would have matched.

The consequence is stark and non-obvious: caching is a property of prefixes, not of content. Two prompts that share 5,000 identical tokens get zero cache benefit if those tokens don't start at the same place — if a single differing byte sits in front of them, the shared block is past the break and recomputed. It doesn't matter how much the prompts have in common; it matters how much they share from the front, uninterrupted. That single fact turns prompt caching from a billing feature into a prompt-structure problem.

If caching is about the shared prefix, then the design rule writes itself: put everything fixed at the front and everything variable at the back. The system prompt, tool schemas, and any static documents go first — identical on every request, so they form a long shared prefix that caches. The user's message, the current timestamp, the request id, anything that changes go last, after the cached region.

Get this backwards — put the user query or a timestamp near the top — and the prefix differs on every single request, so nothing after it caches, no matter how much static content follows. This is the number-one prompt-caching bug, and it's invisible: everything works, you just quietly pay full price forever. The same discipline is why context engineering and caching go together: a stable, well-ordered context is both easier to reason about and far cheaper to serve.

The cache benefit is exactly the length of the shared prefix between the new prompt and a cached one. Compare token by token; the hit ends at the first difference:

Only leading tokens that match uninterrupted count. A difference at position i caps the hit at i, regardless of what follows.

So ordering, not content overlap, sets the hit rate. With fixed content of length F first and a variable query last, every request shares the full F; with the variable part first, the shared prefix is essentially zero:

Same tokens, opposite outcomes — the only difference is where the variable content sits. The runnable version below computes the shared prefix and shows ordering flipping the hit rate from full to zero.

Two more mechanics complete the picture. TTL (time to live) is how long a cached prefix survives before it expires — prompt caches are typically short-lived (often minutes), so caching helps most when the same prefix is reused frequently within that window: a busy chat session, a burst of requests sharing one system prompt. If reuse is sparse relative to the TTL, the prefix expires between uses and you're recomputing anyway.

Some providers expose explicit cache breakpoints — you mark the boundary between the stable, cacheable head and the variable tail, so the system knows exactly what to store. Whether implicit or explicit, the mental model is the same: identify the longest stable prefix, keep it byte-for-byte identical across requests, and put everything that varies after it. The economics of when this pays off — the write premium versus the read discount — live in the cost handbook; here the job is just to make the hit rate high.

The silent killers all change the prefix. A timestamp or unique id near the top — "Current time: 14:03:22" in the system prompt — differs every second and disables caching for everything below it; move it to the end or drop it. Non-deterministic serialization — a dict whose key order varies, a JSON blob re-encoded differently — produces a different prefix even when the content is "the same"; serialize the fixed part deterministically. And reordering or editing tool definitions between calls shifts the prefix; keep them stable and in a fixed order.

Two more. Assuming a cache hit without measuring: instrument the cached-token counts your provider returns and confirm the hit rate is what you expect — a broken cache looks identical to a working one except on the bill. And over-caching volatile content: if the "fixed" prefix actually changes often (a document that updates per user), caching it wastes the write premium for little reuse. The craft is simple to state and easy to get wrong: find your longest genuinely-stable prefix, keep it identical, put it first, and measure. Do that and a huge system prompt becomes nearly free to send again and again.