In 50 First Dates, Lucy wakes up every morning with no memory of the day before. So every morning, the courtship starts over. Same introduction, same jokes, same walk through the entire history of the relationship.
Eventually Henry stops re-living it and records a videotape. Each morning Lucy watches ten minutes of tape and gets caught up. The expensive part happened once. The replay is cheap.
Every API call to a language model is Lucy waking up.
The model remembers nothing between requests. Your system prompt, your tool definitions, your reference documents, the entire conversation so far: all of it gets sent again, and processed again, on every single call. And you pay for every token of it. Every time.
Prompt caching is the videotape. Most teams have never pressed record.
How the Meter Actually Runs
Language models are stateless. That one fact drives most of the API bill.
Say your agent has a 15,000-token system prompt: role instructions, business rules, tool definitions, a few reference documents. A user asks a question that adds 200 tokens. The model processes 15,200 input tokens to answer.
The user asks a follow-up. The model processes the system prompt again, plus the history, plus the new question. Every turn of every conversation re-reads the same 15,000 tokens at full price.
Run that agent through 1,000 requests a day and you have paid to process 15 million tokens of pure repetition. The model learned nothing new from them. You just paid the meter to re-read the manual.
What Caching Does
When a model processes your prompt, it builds an internal representation of it before generating a single output token. That computation is the expensive part of input processing.
Prompt caching stores that computed state on the provider's side. When your next request starts with the same prefix, the provider skips the recomputation and picks up where the cache left off. You get billed a fraction of the normal input price for those tokens.
The mechanics differ by provider.
| Provider | How It Works | Cached Read Price |
|---|---|---|
| Anthropic | Explicit cache breakpoints you set in the request. Writing to the cache costs slightly more than normal input; reads are the discount. | ~10% of base input price |
| OpenAI | Automatic for prompts over 1,024 tokens. No configuration, no cache write premium. | ~50% of base input price |
| Google Gemini | Implicit caching happens automatically. Explicit context caching gives more control but adds a storage fee. | Varies by model and mode |
One important detail: caches expire. On Anthropic the default lifetime is around five minutes, refreshed each time the cache is hit, with a longer one-hour option at a higher write cost. If your traffic is sparse enough that the cache dies between requests, you pay the write premium repeatedly and collect the discount rarely.
Caching a prompt does not change the model's output. Same tokens in, same behavior. The only thing that changes is the bill.
What Breaks the Cache
Caching works on exact prefix matching. The provider compares the start of your new request against what it has stored, byte for byte. The match stops at the first difference, and everything after that point gets processed at full price.
This is where most teams silently lose the discount.
A timestamp in the system prompt kills it. "The current date is May 26, 2026, 9:14 AM" means every request has a different prefix. Cache hit rate: zero. Move the date to the end of the prompt, or truncate it to the day, and the cache survives.
Reordering kills it. If your tool definitions get serialized in a different order on different requests (a common bug when tools live in an unordered data structure), the prefix changes even though nothing meaningful did.
Per-user content placed early kills it. If the user's name or account details appear before your 15,000 tokens of stable instructions, no two users ever share a cache.
None of these show up as errors. The requests succeed. The answers are fine. The only symptom is a bill that is two to five times larger than it needs to be, and nobody is looking at the line item that would reveal it.
Structuring for Cache Hits
The rule: order your prompt from most stable to most volatile.
System instructions and tool definitions first. These change when you deploy, not when a user asks a question. This is the deep-frozen layer that should be cached across every user and every session.
Reference documents next. Product docs, policy text, schema descriptions. Stable for days or weeks at a time.
Conversation history after that. It grows during a session but each turn only appends, which means the previous turns stay cacheable.
The new user message last. This is the only part that should be genuinely unique per request.
Then verify. Every major provider reports cached token counts in the API response and the usage dashboard. If you have never looked at that number, look this week. A healthy agent workload should show the large majority of input tokens hitting the cache. If yours shows zero, something in your prefix is changing on every call, and finding it is probably an afternoon of work.
Who This Matters For
Caching is not equally valuable everywhere.
If you run occasional one-off calls with short prompts, skip it. The cache will expire between requests and there is not much repeated content to discount anyway.
If you run agents, the math is hard to ignore. Agents are the worst-case repetition scenario: large system prompts, large tool registries, and many model calls per task, often several per minute. The same goes for RAG systems that prepend the same instructions to every query, and for any chat product with real traffic.
One of our clients was spending roughly 70% of their input token budget on a system prompt that had not changed in two months. Restructuring the prompt order and adding cache breakpoints took a day. The input bill dropped by more than half. Nothing about the product changed.
That is the general shape of this opportunity. It is not a redesign. It is not a migration. It is prompt hygiene plus a config change, and the discount is sitting there whether you claim it or not.
Open your usage dashboard and find the cached token count. If it is near zero and you are running anything with a real system prompt, you have found the cheapest cost reduction available to you this quarter.
