Prompt caching reuses the processed prefix of a repeated prompt; batching amortises fixed overhead across many requests. Neither changes what the model outputs, which makes them the two safest cost optimisations available. Everything depends on one rule: the cached portion must be byte-identical and must come first.
Most cost work on LLM systems trades quality for money. Smaller models, shorter context, fewer retrieved documents — each saves something and risks something.
These two do not. The output is identical either way, which is why they belong at the top of any optimisation list and why it is odd how often they come last.
- 0Change to output quality from either technique, correctly applied
- 1 ruleStatic content first, variable content last, byte-identical prefix
- 2 axesCaching cuts repeated work; batching cuts per-request overhead
- PrefixCaching matches from the start only, never mid-prompt
- LatencyThe cost batching trades away, and the reason it suits offline work
What does prompt caching actually reuse?
The computed internal representation of tokens the model has already processed, not the text itself.
When a prompt begins with the same long preamble on every call, the model would normally recompute that preamble’s representation every time. Caching stores it after the first pass and resumes from the boundary on subsequent calls.
The critical property is that matching runs from the start of the prompt forward, stopping at the first difference. A single changed character early in the prompt invalidates everything after it.
Figure 1 — Why prompt order decides whether caching works
Variable content first
Prompt opens with a timestamp, session ID or the user’s question, then the long shared instructions and documents.
The prefix differs on every call, so the match fails at token one. The cache exists and never hits.
Zero reuse
Static content first
Prompt opens with the fixed system instructions and reference documents, and the user’s question comes last.
The long shared prefix matches every time, and only the short tail is processed fresh.
Full reuse
This is the single highest-value paragraph in the article. Reordering a prompt so static content precedes variable content is a few minutes of work, changes no output, and is the difference between a cache that never hits and one that hits on nearly every call.
What silently breaks a cache?
Small things, and none of them raise an error. Your calls simply cost what they always did.
- A timestamp or request ID injected near the top of the system prompt
- Retrieved documents placed before the static instructions rather than after
- Non-deterministic serialisation, where a dictionary or JSON object renders its keys in a different order between calls
- Trailing whitespace differences from template rendering
- Per-user personalisation inserted into the preamble instead of appended near the question
- Version strings that change on every deploy
Watch for this
Caching failures are invisible without instrumentation. Nothing errors, output is correct, and only the bill reflects the problem. Log cache hit rate as a first-class metric from day one, and alert when it drops. A deploy that reorders a template can take a system from near-total reuse to zero, and nobody notices for a month.
How should you structure a cacheable prompt?
Order components from most stable to least stable. That single principle covers nearly every case.
- System instructions. Changes on deploy at most.
- Tool and schema definitions. Stable across a release.
- Few-shot examples. Fixed until deliberately revised.
- Shared reference documents. Stable per corpus version.
- Retrieved context. Varies per query.
- Conversation history. Grows per turn, so it appends cleanly after the stable block.
- The current user message. Always last.
Conversation history is the case worth understanding properly, because it appends rather than mutates. Each turn extends the prefix rather than changing it, so a multi-turn conversation can reuse everything up to the previous turn if history sits after the static block and nothing rewrites earlier turns.
Summarising old turns breaks that property, since it rewrites the middle of the prompt. Where you must compress, compress and then pin the summary so it stops changing.
How is batching different?
Caching removes repeated work. Batching removes repeated overhead.
Generation is typically memory-bandwidth bound, so the weights must be read for each forward pass regardless of how many sequences are being processed. Running many sequences together spreads that fixed read across all of them, raising throughput substantially.
| Dimension | Prompt caching | Batching |
|---|---|---|
| Saves | Recomputation of a shared prefix | Per-request fixed overhead |
| Requires | Byte-identical prefix across calls | Multiple requests available at once |
| Latency effect | Improves it | Worsens individual request latency |
| Suits | Any repeated preamble | Offline and asynchronous work |
| Output change | None | None |
| Main failure | Prefix mismatch, silent | Queueing delay, visible |
The two compose well. A batch of requests sharing a long system prompt benefits from both at once, which is why bulk classification and document-processing jobs are the cheapest workloads to run.
When should you not batch?
Whenever a human is waiting.
Batching works by holding requests until enough accumulate or a timeout fires. That wait is pure added latency for the first request in the window, and interactive users feel it directly.
The useful split is by who consumes the output. Interactive chat, autocomplete and anything with a visible cursor should not batch. Overnight enrichment, bulk classification, evaluation runs and document pipelines should batch aggressively, because nothing perceives the delay.
Where a system serves both, separate the queues rather than compromising with a middling batch window that serves neither well.
Where do teams go wrong?
Optimising the model before the prompt structure
Teams move to a smaller model, accepting a quality loss, while their prompt remains uncacheable because a session ID sits in line three. Fix the free thing before the costly one.
Treating cache hit rate as a vanity metric
Unlike semantic caching, prefix caching has no correctness risk, so a high hit rate here is unambiguously good. It deserves a dashboard and an alert, and it is one of the few metrics where more is simply better.
Letting templates drift
Prompt templates get edited by many hands. Without a test asserting that the static prefix is byte-stable across renders, someone will eventually add a variable near the top and quietly double your costs.
What do experienced teams do differently?
They write a test for prefix stability and run it in CI.
The test renders the prompt twice with different variable inputs and asserts the first n characters are identical. It is a handful of lines, and it converts an invisible recurring cost regression into a build failure.
They also treat the static block as a versioned artefact. When it must change, it changes once per release rather than continuously, so the cache warms after deploy and stays warm.
A short glossary
- Prefix caching
- Reusing the computed representation of a prompt’s opening tokens across calls that share that opening exactly.
- Cache boundary
- The token position where a cached prefix stops matching and fresh computation begins.
- Batching
- Processing several requests together in one forward pass to spread fixed overhead across them.
- Memory-bandwidth bound
- A workload limited by how fast weights can be read from memory rather than by arithmetic throughput.
- Batch window
- The interval a scheduler waits for additional requests before dispatching what it has.
Key takeaways
- Prompt caching and batching both cut cost without changing output, which makes them the safest optimisations available.
- Caching matches from the start of the prompt forward and stops at the first difference.
- Order prompt components from most stable to least stable, with the user message always last.
- A timestamp or session ID near the top of a prompt reduces cache reuse to zero, silently.
- Conversation history appends rather than mutates, so it caches well unless you rewrite earlier turns.
- Batching raises throughput and worsens individual latency, so separate interactive and offline queues.
- Assert prefix stability in CI, or template drift will quietly undo the whole optimisation.
Frequently asked questions
Does prompt caching change model output?
No. It reuses the computed representation of tokens already processed rather than altering how generation proceeds. The result is identical to an uncached call, which is what makes it unusually safe compared with optimisations that trade quality for cost.
Why is my prompt cache not being hit?
Almost always because something variable sits near the start of the prompt. Timestamps, session identifiers, request IDs, per-user personalisation and non-deterministic key ordering in serialised objects all break the prefix match at the first differing character, invalidating everything after it.
Where should the user question go in a cacheable prompt?
Last. Order components from most stable to least stable: system instructions, tool definitions, few-shot examples, shared reference documents, retrieved context, conversation history, then the current user message. That ordering maximises the length of the shared prefix.
Does caching work with multi-turn conversations?
Yes, and unusually well, because history appends rather than mutating. Each turn extends the prefix instead of changing it, so everything up to the previous turn can be reused. Summarising older turns breaks this by rewriting the middle of the prompt.
Should I batch requests in an interactive application?
No. Batching works by waiting for requests to accumulate, and that wait is added latency a user directly perceives. Reserve batching for offline work such as bulk classification, enrichment jobs and evaluation runs, and keep a separate unbatched path for interactive traffic.
Do batching and caching work together?
Yes, and they compose particularly well. A batch of requests sharing a long system prompt benefits from both simultaneously, which is why bulk document processing and classification jobs tend to be the cheapest workloads to operate at scale.
How do I know caching is working?
Instrument it. Log cache hit rate as a first-class metric and alert on drops. Caching failures raise no errors and produce correct output, so the only symptom is cost. A template change can take a system from near-total reuse to none without anything appearing broken.
Is it worth restructuring an existing prompt for caching?
Usually yes, because it is among the highest-return changes available. Moving static content ahead of variable content takes minutes, alters no output, and can convert a cache that never hits into one that hits on nearly every call.
References
- Kwon, W., et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180. Background on prefix sharing and batched serving.
- Dettmers, T., et al. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314. Context on memory as the binding constraint in model serving.
- Liu, N. F., et al. Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172. Relevant when reordering prompts for cacheability.
