Professor Dumbledore kept a Pensieve on his desk.
Not because his memory was failing. He had centuries of knowledge, and none of it was going anywhere. He used the Pensieve because he understood something most AI builders have not figured out yet: there is a limit to how much any mind can hold in active focus at once. Even a brilliant one.
The Pensieve was not a crutch. It was a strategy. Offload what you do not need right now. Retrieve it when you do. Keep the working space clear for what actually matters.
Your AI has the same problem. And if you are not managing it deliberately, it is quietly breaking your tools and inflating your costs.
The context window
Every AI model has a context window. Think of it as the model's desk: the total amount of text it can hold in active memory during a single task. Your system instructions, the conversation history, the documents you uploaded, the response it is generating. All of it has to fit on that desk at once.
Context windows are measured in tokens. Roughly three-quarters of a word each. A 200,000-token window sounds enormous. For a single query, it is. But enterprise AI deployments are not single queries. They are pipelines. Assistants processing dozens of documents. Chatbots maintaining conversations across hundreds of turns. Workflows where the model needs to hold a 50-page report, a full conversation history, and your system prompt simultaneously.
When the desk gets full, one of two things happens.
The model truncates. Silently dropping the oldest content to make room for new. Your system instructions from the start of the conversation. The client context you established three messages ago. Gone.
Or the model degrades. Still has access to everything technically. But the volume of competing information dilutes its focus. Answers get less precise. Reasoning gets sloppier. It starts to feel like talking to someone who has read too much and retained too little.
The failure is almost never dramatic. The AI does not crash. It just quietly gets worse. And most teams never connect the degraded outputs to the underlying problem.
This is also a cost problem
AI APIs charge by the token. Both input and output. Every time you make a call, you pay for every token in the context window. Not just the new message you sent. All of it.
If your system prompt is 5,000 tokens, your conversation history is 40,000 tokens, and you are processing a 30,000-token document, you are paying for 75,000 tokens of input on every single call. Even if the only new information is a 20-word question.
For a team running a few hundred queries a day, this is rounding error. For an enterprise deployment running tens of thousands of queries, it compounds fast.
One client had a support chatbot stuffing their entire product documentation (80,000 tokens) into every single API call. Not because they needed all of it every time. Because nobody had thought about what the model actually needed for each specific query. Their monthly AI costs were six times what they needed to be.
Manage what goes on the desk. Everything else is just money sitting there.
Chunking: don't load what you don't need
This is the most immediate fix for most teams, and it connects directly to how your documents live in a vector database. (Our post on RAG covers the mechanics of retrieval if you need the foundation.)
Instead of loading an entire document into every AI call, you split it into meaningful chunks and retrieve only the chunks relevant to the specific query. Ask about the refund policy, get the refund policy section. Ask about shipping, get the shipping section. The rest stays in storage until needed.
The context window stays lean. Costs drop. Answer quality often improves because the model has less noise to filter through.
Audit what you are loading into every AI call right now. If you are sending entire documents when you only need sections, this is the fix.
Summarization: compress what you must keep
Not everything can be chunked. Sometimes the full conversation history matters. Sometimes the document is dense and every section is potentially relevant.
In those cases, summarization does what Dumbledore's Pensieve did: you offload the detail, keep the essential. Instead of keeping every prior exchange verbatim in the context window, you periodically summarize what has been discussed and replace the detailed history with the compressed version. The model retains the key information. The token cost drops sharply.
A 10,000-token conversation history can usually be compressed to 500-1,000 tokens without losing anything that matters. That is a 90% reduction in cost for that portion of the window.
| Scenario | Before summarization | After summarization |
|---|---|---|
| 10-turn customer support chat | ~8,000 tokens of history | ~400 tokens of summary |
| Research assistant after 30 min | ~25,000 tokens accumulated | ~1,200 tokens of summary |
| Cost impact at $15/M input tokens | $0.12 per call | $0.006 per call |
This matters especially for long-running conversations and multi-turn workflows. Set a threshold, say once history exceeds 8,000 tokens, and automate the compression. Most teams never bother. The ones who do wonder why they waited.
Prompt caching: pay full price once
This is the most underused optimization in enterprise AI right now.
Most platforms, including Anthropic's Claude API, offer prompt caching. If part of your context never changes between requests (your system instructions, stable reference documents, base product knowledge), you pay full price the first time you send it, and roughly 90% less on every subsequent call that reuses those cached tokens.
For a company running a customer-facing assistant where the system prompt and product knowledge base are the same on every call, this is not a marginal saving. It is a structural cost reduction that compounds with every query you run.
The audit: identify which parts of your context are stable versus dynamic. Stable content is anything that does not change between calls. That is your caching candidate. The user's specific question, the current conversation turn: those change every time and cannot be cached. But for most enterprise deployments, a significant portion of what gets sent on every call is identical.
The teams saving the most on AI infrastructure right now are not the ones who found cheaper models. They found cheaper calls.
Four questions worth answering before your next build
Walk through these for any AI deployment you are running or planning:
What are you loading into every call? List it out. System prompt, conversation history, documents, examples. Put a rough token estimate next to each.
What of that content is actually needed for every single query? Anything that is not is a chunking or selective retrieval opportunity.
What content is stable and never changes between calls? That is your caching opportunity. Calculate what you are currently paying to send it each time.
When does the conversation history get long enough that summarization makes sense? Set a threshold now. Automate the compression before you need it.
Most teams who run this exercise find 40-70% savings on API costs. Not from switching models. Not from clever engineering. From paying attention to what they are putting on the desk.
The degraded outputs you noticed two weeks after launch. The API bill that keeps creeping upward. The assistant that seemed sharp in testing but softer in production.
Context drift is the diagnosis most teams never run.
Run it.