The AI's working memory, measured in tokens. When it fills up, things fall out or get missed.
3 min to watch2 min to read
Based on research, not a single incident: the "Lost in the Middle" study (2023),1 Anthropic's context window docs2 and its long-document tips.4 The chat, the contract and its page numbers are made up.
Read the walkthrough as text
All 300 pages
Friday, 5 pm. You need one answer from a long supplier contract, fast.
You paste in the whole contract and ask your question.
The AI says yes and gets to work.
Behind the scenes, all 300 pages go into the AI's memory: everything it can see at one time.
This is the AI's context window: its working memory. It's counted in tokens, chunks of text about ¾ of a word each.
The AI starts reading. Page 1: who signed, for how long, at what price.
Page 147, deep in the middle, holds the answer: either side can end the contract with 60 days' notice.
The AI skims right past it.
At the end, it pays attention again: signatures and dates.
Here's what happened inside: AIs use facts at the start and end of a long input best, and miss more in the middle.
So it answers with confidence, and gets it wrong.
Monday, the legal team finds the rule on page 147. It was there all along.
Lost in the middle.The whole contract fit in the AI's context window, but the answer sat in the middle. A 2023 study found AIs use facts at the start and end of a long input best, and miss more in the middle.
Just the right pages
Same question, same contract. This time the app picks the pages before the AI reads.
A search finds the 3 pages about ending the contract early. The other 297 stay out.
Only those 3 pages go into memory: about 2,000 tokens instead of 200,000.
The order matters too: the key page goes first and the question goes last, where the AI reads best.
The AI reads page 147 closely and finds the rule.
It answers correctly, and says where it found the answer.
Less in, key facts first.Send only the pages that matter. Put the key facts at the start and the question at the end, where the AI reads best.
Keep going:
Why it matters, what it costs, and what to ask your team.
How it works
Working memory, measured in tokens.
Everything the AI can see right now sits in its context window: your instructions, the chat so far, any files, and its own reply. Anthropic calls it the model's "working memory".2
It's measured in tokens, chunks of text about ¾ of a word each.3 Fill it up and see what happens:
Window size
Tap to add things to the AI's memory:
0
Rough sizes: about 500 words a page, and a token is about ¾ of a word. The two window sizes are real: 1M tokens on many Claude models, 200K on others.2 This shows a chat app that drops the oldest parts first.
A bigger window holds more. But more isn't automatically better: as it fills, recall drops. Anthropic calls this context rot.2
So the more you give me, the more I might miss?
The real evidence
Where did the fact go?
In 2023, researchers hid one answer among 20 search results and moved it around.1
Move the answer. Watch how often the AI gets it right.
Model (2023)
20 search results. One holds the answer. Tap a numbered one to move it:
of questions answered right
Real results from the "Lost in the Middle" study, Tables 1 and 6: 20 search results (about 3,000 tokens), one holding the answer. These are 2023 models.1
Start and end: best. Middle: worst. With the answer in the middle, GPT-3.5-Turbo did worse than with no documents at all.1
So: send less, and don't bury the important part.
Ask your team
Three questions for your next meeting.
You don't need to read the code. These show whether your AI is reading the right things.
How full is the context on a typical request?If nobody knows, start counting tokens. Near the limit, recall drops and the bill climbs.
Do we send everything, or only what's relevant?A search can pick the few pages that answer the question instead of pasting the whole pile.
Where do the most important facts sit in the prompt?At the start or the end. The middle is where things get lost.
Under the hood, step by step: build a contract helper that finds the one page that matters.
Build along
Build a contract helper that finds the right page, one step at a time
We start with the simplest version and ask it the same question about the same 300-page contract. Press Run it after each step and watch the answer change.
Before you ship
A short checklist.
The build-along covers the code. These keep it working as your documents and chats grow.
Set a token budget per requestWell under the model's limit. Log every time you hit it.
Count with the model you'll useNewer Claude models count about 30% more tokens for the same text.6
Retrieve, don't stuffSend the pages that answer the question, with their source.
Key facts first, question lastRules and must-know facts at the top. The question at the very end.
Summarize or clear old turnsKeep decisions and numbers. Drop chatter and old tool results.
Save notes outside the windowAgents can write notes to a file and load them back when needed.5
References
"Lost in the Middle: How Language Models Use Long Contexts"· Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni and Liang, arXiv, July 2023 (v3 Nov 20, 2023). U-shaped accuracy; Table 6 (20 documents) and Table 1 (no documents: 56.1% for GPT-3.5-Turbo, 48.3% for Claude 1.3).
Context windows· Anthropic docs. "Working memory", context rot, up to 1M tokens, oldest-first in chat apps, "prompt is too long" errors, compaction.
Pricing· Anthropic docs, FAQ: 1 token is about 4 characters or 0.75 words in English.
Prompting best practices· Anthropic docs, "Long context prompting": long documents at the top, queries at the end, up to 30% better in tests.
Token counting· Anthropic docs. Free to use, an estimate; newer models count about 30% more tokens for the same text.
The chat, the contract and its page numbers are made up. In the walkthrough, the "read closely" and "skimmed" bars picture the study's pattern; they aren't measurements. The memory-filling widget uses rough sizes; the accuracy chart uses the study's real numbers.