A larger context window does not give you more usable context. Models retrieve reliably from the beginning and end of long inputs and considerably worse from the middle. Filling the window therefore degrades the answers it was supposed to improve, which makes placement a first-class engineering decision.
The finding that should reshape how you build prompts is positional. Liu et al. documented that performance on long-context tasks is highest when relevant information sits at the very start or very end of the input, and drops noticeably when it sits in the middle (arXiv:2307.03172).
Read that alongside the marketing for million-token windows and the tension is obvious. Capacity grew far faster than the ability to use it evenly.
At a glance
- Window size is capacity, not usable attention
- Position matters: start and end outperform the middle
- Every irrelevant token competes with relevant ones
- Cost and latency both scale with input length
- Most bloat comes from conversation history nobody prunes
- Measure retrieval quality at your real context length, not at a small one
Why does packing the window backfire?
Three mechanisms compound, and only the first is widely discussed.
Figure 1 — Same information, two placements
Relevant chunk placed last
Ten retrieved chunks, the correct one immediately before the question.
Recency and proximity to the instruction both work in your favour, and the model attends to it reliably.
High recall
Relevant chunk placed sixth of ten
Identical content, identical token count, buried mid-input.
The same model now frequently answers as though the information were absent, with no error and no signal that anything went wrong.
Degraded recall
Nothing about the retrieval changed. The correct chunk was fetched and supplied in both cases. Ordering alone moved the outcome, which is why reranking to put the best result closest to the question is one of the cheapest wins available.
The second mechanism is dilution. Attention is finite and distributed, so each irrelevant token takes a small share from relevant ones. Ten well-chosen chunks routinely outperform fifty mediocre ones containing the same answer.
The third is cost. Input tokens are charged and processed on every call, so a bloated prompt is a recurring tax paid on every request forever.
The twelve techniques
| Technique | What it fixes | Effort |
|---|---|---|
| Rerank before prompting | Best result buried mid-context | Low |
| Place key content last | Positional degradation | Very low |
| Cut k aggressively | Dilution from marginal chunks | Very low |
| Deduplicate retrieved chunks | Overlap wasting the window | Low |
| Summarise old conversation turns | History bloat | Moderate |
| Sliding window over history | Unbounded session growth | Low |
| Extract rather than include | Long documents where one field matters | Moderate |
| Compress system prompts | Instruction sprawl | Low |
| Move rules into fine-tuning | Repeated instructions at high volume | High |
| Filter by metadata first | Retrieving from the wrong document set | Low |
| Chunk on structure | Fragments that waste tokens without meaning | Moderate |
| Cache the static prefix | Repeated cost of a fixed preamble | Low |
The first three cost almost nothing and deliver most of the available gain. Reranking, ordering, and simply retrieving fewer chunks will outperform elaborate compression schemes in most systems.
Where does the bloat actually come from?
Rarely from the documents. Almost always from history and instructions.
Conversation history grows without bound unless something prunes it, and a session that started at 200 tokens can pass 20,000 after forty turns. Most of that is irrelevant to the current question.
System prompts accumulate by a similar ratchet. Each edge case adds a sentence, nobody removes any, and a year later the preamble runs to a thousand words that get billed on every call.
Watch for this
Audit your actual prompts in production rather than reasoning about them from the code. Log total input tokens broken down by component: system prompt, history, retrieved context, user message. Teams routinely discover that retrieved context, the part they spent months optimising, is a minority of the window, while unpruned history dominates it.
How should you handle conversation history?
Three strategies, escalating in effort.
- Sliding window. Keep the last n turns verbatim, discard the rest. Trivial to build and adequate for most assistants.
- Rolling summary. Periodically compress older turns into a running summary, keeping recent turns verbatim. Preserves long-range context at the cost of a summarisation call.
- Retrieval over history. Treat past turns as a searchable corpus and retrieve only the relevant ones. Most capable, most complex, and the right answer for long-lived assistants.
Whichever you pick, always keep the first turn or two verbatim. Early messages usually establish the task framing, and losing them degrades everything downstream in ways that look like model failure rather than context management failure.
Does a bigger window remove the need for retrieval?
Not on the evidence. It changes the trade rather than settling it.
Long-context prompting removes chunking, indexing and retrieval failure. In exchange it charges per query rather than amortising an index, and it exposes you to positional degradation across a large input.
The practical middle ground most teams land on: retrieve to narrow the candidate set, then use a generous window for the survivors. That gets retrieval’s precision and long context’s tolerance for imperfect boundaries, without depending fully on either.
Where do teams go wrong?
Raising k to fix poor retrieval
When the right answer is not being retrieved, increasing k from 5 to 50 sometimes drags it in. It also buries it mid-context and dilutes attention, so quality often falls even as recall technically improves. Fix ranking instead.
Testing at short context and shipping at long
A prompt validated with three chunks behaves differently with thirty. Positional effects only appear at scale, so evaluations must run at the context length you actually deploy.
Compressing before measuring
Summarisation and compression are lossy and add a generation call. Teams reach for them before checking whether trimming k and pruning history would have sufficed, which it usually would.
What do experienced teams do differently?
They budget the window explicitly, like memory.
A fixed allocation — so many tokens for system instructions, so many for history, so many for retrieved context, with a reserve for output — turns an invisible creeping problem into a visible constraint. When one component wants more, it has to take it from another, and that forces the conversation.
They also run a positional sanity check before launch: place a known fact at the start, middle and end of a realistic prompt and ask about it. If middle placement fails, you know your ordering strategy matters before users find out.
A short glossary
- Context window
- The maximum number of tokens a model can accept as input, including instructions, history and retrieved content.
- Lost in the middle
- The documented tendency for models to use information less effectively when it sits in the middle of a long input.
- Dilution
- Loss of effective attention on relevant content caused by the presence of irrelevant content.
- Rolling summary
- Periodic compression of older conversation turns into a running summary while recent turns stay verbatim.
- Prefix caching
- Reusing the processed representation of a fixed prompt prefix across calls to avoid recomputing it.
The caveat that qualifies all of this
Positional effects are model-specific and they change between releases. The finding that information in the middle is used less effectively was measured on particular models at a particular time, and successive generations have improved on it to varying degrees.
So treat everything above as a set of hypotheses to test on the model you actually run, not as fixed laws. The technique that survives every model change is the meta-technique: measure retrieval quality at your real context length, with your real prompt structure, and re-measure whenever you change models.
Everything else in this article is downstream of that habit.
Key takeaways
- Window size is capacity, not usable attention, and the two diverge as inputs grow.
- Information placed at the start or end of a long input is used more reliably than information in the middle.
- Reranking, ordering and cutting k are the cheapest interventions and usually the most effective.
- Most bloat comes from unpruned conversation history and accumulated system instructions, not from documents.
- Always keep the first turn or two verbatim, since early messages establish task framing.
- Evaluate at the context length you deploy, because positional effects only appear at scale.
- Positional behaviour is model-specific and changes between releases, so re-measure after every model change.
Frequently asked questions
Does a larger context window solve prompt engineering problems?
No. Capacity grew faster than the ability to use it evenly, and models use information in the middle of long inputs less effectively than information at either end. A larger window raises the ceiling on what you can supply without guaranteeing the model will use it well.
Where should I place the most important information in a prompt?
At the end, immediately before the question, or at the very start. Both positions outperform the middle. Where you have a reranker, use it to place the highest-scoring retrieved chunk closest to the instruction, which costs nothing and reliably improves answers.
How many chunks should I retrieve?
Fewer than instinct suggests. Ten well-ranked chunks typically beat fifty mediocre ones containing the same answer, because irrelevant content dilutes attention and pushes relevant content toward the middle. Raise k only when measurement shows the correct chunk is genuinely being missed.
What is the best way to manage conversation history?
Start with a sliding window keeping the last several turns verbatim, since it is trivial to implement and sufficient for most assistants. Move to a rolling summary when long-range context matters, and to retrieval over history for long-lived assistants. Always preserve the first turn or two.
Should I use long context instead of RAG?
They trade differently rather than one replacing the other. Long context removes chunking and retrieval failure but charges per query and exposes you to positional degradation. Most teams retrieve to narrow the candidate set, then use a generous window for the survivors.
Why did my answers get worse when I retrieved more documents?
Dilution and position. Extra chunks consume attention that relevant content needs, and adding results pushes the correct one deeper into the middle of the input where it is used less effectively. Improve ranking rather than increasing quantity.
How do I find out what is filling my context window?
Log input tokens in production broken down by component: system prompt, conversation history, retrieved context and user message. Teams frequently discover that retrieved context, the part they optimised hardest, is a minority of the window while unpruned history dominates it.
Is prompt compression worth doing?
Only after cheaper options are exhausted. Compression is lossy and adds a generation call. Trimming k, deduplicating chunks and pruning history usually recover more tokens with less risk, so measure what those achieve before building a compression stage.
References
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172.
- Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401.
- Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB: Massive Text Embedding Benchmark. arXiv:2210.07316.
