Prompt caching
A system prompt gets sent to the model on every single turn of a conversation, in full — including the parts that never changed since the last message. Prompt caching is what stops a business from paying full price for those unchanged parts, over and over.
3 min read
The cost problem, stated plainly
Every request to an LLM sends the entire context again — system prompt, tool definitions, and the conversation so far — because the model has no memory between API calls. For an agent with a substantial tool catalog and a detailed system prompt, that means the same several-thousand-token preamble gets billed again on every single turn of a ten-message conversation, even though nothing in it changed.
What prompt caching actually does
Model providers offering prompt caching let a request mark a prefix of its input as cacheable — the provider stores that exact prefix (given a stable input) and, on a subsequent request with the identical prefix, charges a small fraction of the normal input-token rate for it instead of the full rate. A concrete, real figure: roughly 10% of the normal cost for a cached prefix hit, after the first turn establishes the cache.
Turn 1: [system prompt + tool catalog] + [user message 1]
-> full price for the system prompt + tool catalog
Turn 2: [system prompt + tool catalog] + [conversation so far] + [user message 2]
-> ~10% price for the system prompt + tool catalog,
IF it's byte-for-byte identical to turn 1's cached prefix
Why "byte-for-byte identical" is the entire game
Caching works on an exact-match prefix — not "similar enough," not "semantically the same." If a single character differs anywhere within what was meant to be the cached portion, the cache misses entirely for that request, and the full price applies again. This is why the design of what goes into the cached prefix versus what doesn't is the actual skill here, not an implementation detail to ignore.
Separating the stable part from the volatile part
STABLE (belongs in the cached prefix):
- the core system prompt instructions
- the full tool catalog / schema definitions
- anything that's the same for every request, regardless of who's
asking or when
VOLATILE (must NOT be inside the cached prefix):
- today's actual date
- the specific user's name, role, or permissions
- a specific building/account/session identifier
- anything computed fresh per-request
A system prompt that interleaves stable instructions with per-request facts — "You are a helpful assistant for , today is , this user's role is ..." all as one contiguous block — breaks caching entirely, because that entire block changes every single request, defeating the purpose even for the genuinely-stable instructions sitting right next to the volatile ones. The fix is structural: build the prompt so the large, genuinely-static portion (the instructions, the tool catalog) comes first and stays byte-identical across requests, with the volatile, per-request facts appended or injected separately, outside the cached region.
This directly connects to an earlier lesson in this domain
The lesson on rebuilding system prompts per request (current date, permissions) and this lesson aren't in tension — they're two constraints on the same design that both have to be satisfied. The prompt genuinely does need fresh, per-request facts (that's non-negotiable, covered earlier). Prompt caching just adds a second requirement on top: structure where those fresh facts go, so the volatile part is cleanly separated from the large stable part, instead of the two being tangled together in a way that silently defeats caching for the entire prompt.
The practical takeaway
Prompt caching isn't a switch to flip — it's a constraint on how a prompt is assembled. Getting it right means treating "what's stable" and "what's volatile" as a real design question with real cost consequences, not an afterthought, especially for any agent whose system prompt or tool catalog is large enough that the difference between full price and ~10% price matters at real usage volume.
Further reading
Check your understanding
A quick comprehension check — not tracked, not graded, just for you.
1. Why does an agent with a large system prompt and tool catalog end up paying for the same content repeatedly across a multi-turn conversation without prompt caching?
2. A cached prompt prefix differs from the previous request by a single character. What happens to the cache?
3. A system prompt is written as one contiguous block: "You are an assistant for {building}, today is {date}, this user's role is {role}..." mixing stable instructions with per-request facts. What does this do to caching?
4. How should a system prompt be restructured to get both per-request freshness (date, permissions) AND the cost benefit of prompt caching?