A semantic cache stores answers keyed by meaning rather than exact text, returning a cached response when a new query is similar enough. It converts a slow expensive generation into a fast vector lookup. The complication is that “similar enough” is a threshold you set, and setting it wrong serves confidently wrong answers.
Traditional caching keys on an exact string. In an LLM application that almost never hits, because “how do I reset my password” and “password reset help” are different strings expressing one intent.
Semantic caching embeds the query, searches previous queries by vector similarity, and returns the stored answer when the nearest neighbour clears a threshold. Hit rates jump from near zero to something worth having.
That threshold is the entire engineering problem.
- ExactMatch rate of traditional string caching on natural-language queries: effectively nil
- 1 vectorLookup replacing a full generation on a cache hit
- 2 knobsThreshold and TTL account for most cache behaviour
- 0Personalised or permission-scoped responses that should share a cache
- 2 costsEvery hit saves tokens and latency; every false hit costs trust
How does the threshold actually behave?
It trades hit rate against correctness, and the curve is not symmetric.
Figure 1 — Two queries, one threshold decision
Genuinely equivalent
“How do I reset my password?” against “password reset steps”
Same intent, same correct answer. A cache hit here is pure saving with no downside.
Serve from cache
Similar but not equivalent
“How do I reset my password?” against “How do I reset my API key?”
High embedding similarity, completely different answer. The user receives fluent, confident, wrong instructions.
Must miss
The asymmetry that should drive your threshold: a false miss costs one generation. A false hit costs a wrong answer delivered with full confidence and no error anywhere in your logs. Tune conservatively.
Negations and small qualifiers are where naive thresholds fail hardest. “Can I cancel after 30 days” and “Can I cancel before 30 days” sit close together in embedding space and have opposite answers.
What must never be cached?
Five categories, and the first two are correctness bugs waiting to happen.
- Anything personalised. “What is my balance” is the same question from every user and a different answer for each. Cache keys must include identity, or this must not be cached at all.
- Anything permission-scoped. A cached answer built from documents one user could see may be served to another who cannot. This is a data leak, not a performance bug.
- Anything time-sensitive. Today’s status, current pricing, live inventory.
- Anything stateful. Responses depending on conversation history are not reusable across sessions.
- Anything the user expects to vary. Creative generation, brainstorming, rewrites. Returning an identical answer reads as a broken product.
Watch for this
Permission-scoped caching is the failure that turns a performance optimisation into a security incident. If retrieved context depends on who is asking, the cache key must include the permission scope, not just the query embedding. Partition caches per tenant by default and treat a shared cache as a deliberate exception requiring review.
Where should you put the cache?
| Layer | Caches | Saves | Risk |
|---|---|---|---|
| Full response | Query to final answer | Retrieval and generation both | Highest; a false hit returns a wrong answer |
| Retrieval results | Query to retrieved chunks | Vector search only | Low; the model still generates fresh |
| Embedding | Text to vector | Embedding calls | None; deterministic for a fixed model |
Embedding caching is free correctness-wise and should be switched on without deliberation, since the same text always produces the same vector for a fixed model version. Retrieval caching is nearly as safe. Full-response caching is where the savings and the danger both live.
Many teams get most of the benefit from the two safe layers and never need the risky one.
How do you measure whether it is working?
Hit rate alone is a vanity metric. A cache with a 90% hit rate and a loose threshold is a machine for serving wrong answers quickly.
- Sample hits and grade them. Take 100 cache hits weekly and check whether the returned answer genuinely answers the new query. This is the only metric that matters.
- Track the false hit rate as a first-class number alongside hit rate. Report them together or the pair is misleading.
- Log the similarity score on every hit. When a false hit is found, its score tells you exactly where the threshold should have been.
- Shadow-test threshold changes. Run the candidate threshold in parallel without serving from it, and compare what it would have returned.
Start with a deliberately tight threshold and loosen it only while the graded sample stays clean. Loosening is easy to reverse; a period of wrong answers is not.
How long should entries live?
Long enough to be useful, short enough to stay true.
TTL should follow the volatility of the underlying knowledge rather than a convention. Documentation answers can live for days. Anything derived from operational data should expire in minutes, or not be cached.
The stronger pattern is event-based invalidation: when a source document changes, evict every cached answer derived from it. That requires storing which sources contributed to each cached response, which is a small amount of bookkeeping that removes most staleness risk.
Where do teams go wrong?
Tuning the threshold once
The right threshold depends on query distribution, and query distribution changes as your product does. A threshold validated at launch drifts out of calibration quietly, because nothing errors when it does.
Caching across tenants by default
Covered above and worth repeating, because it is the mistake with consequences beyond quality. Partition first, share only deliberately.
Ignoring the cold start
An empty cache has a zero hit rate, so benchmarks run immediately after deployment show no benefit and teams conclude it does not work. Measure after the cache has warmed on realistic traffic.
What do experienced teams do differently?
They cache at the retrieval layer first and treat response caching as a separate, later decision.
Retrieval caching captures a large share of the latency saving while leaving generation fresh, so a near-miss produces a slightly suboptimal context rather than a wrong answer. The risk profile is completely different for a modest reduction in benefit.
They also store the original cached query alongside the answer and surface it internally. When someone reports a strange response, seeing which earlier question produced it usually diagnoses the problem in seconds.
A short glossary
- Semantic cache
- A cache keyed by embedding similarity rather than exact string match, allowing differently-worded equivalent queries to hit.
- Similarity threshold
- The minimum distance score at which a stored entry is considered close enough to serve.
- False hit
- A cache hit returning an answer that does not correctly answer the new query.
- Event-based invalidation
- Evicting cached entries when a source document changes, rather than waiting for a timer to expire.
- Cache partitioning
- Separating cache namespaces by tenant, user or permission scope so entries cannot leak across boundaries.
The decision summary
| If your situation is | Do this |
|---|---|
| You are just starting | Cache embeddings only. Free, safe, immediate. |
| Retrieval latency dominates | Add retrieval-result caching. Low risk, most of the benefit. |
| Generation cost dominates and queries repeat | Add response caching with a tight threshold and weekly graded sampling. |
| Answers are personalised or permission-scoped | Partition per tenant, or do not cache responses at all. |
| Answers depend on live data | Skip response caching; use event-based invalidation if you must. |
| Users expect variety | Do not cache. Identical answers read as a broken product. |
Key takeaways
- Semantic caching keys on meaning, so differently-worded equivalent queries hit where string caching never would.
- The similarity threshold is the whole engineering problem, and its error costs are asymmetric.
- A false miss costs one generation; a false hit costs a confidently wrong answer with nothing in the logs.
- Negations and small qualifiers sit close in embedding space and have opposite answers.
- Embedding caching is free of correctness risk; response caching carries all of it.
- Permission-scoped responses must never share a cache, or a performance feature becomes a data leak.
- Report hit rate and false hit rate together, because either number alone is misleading.
Frequently asked questions
What similarity threshold should I use for a semantic cache?
There is no portable number, because it depends on your embedding model and query distribution. Start deliberately tight, log the similarity score on every hit, grade a sample weekly, and loosen only while the graded sample stays clean. Tightening after harm is far more costly than starting conservative.
Is semantic caching safe for personalised responses?
Not without partitioning. If the answer depends on who is asking, the cache key must include user or tenant identity. Sharing a cache across permission boundaries can serve one user content assembled from another user’s documents, which is a security incident rather than a quality issue.
What hit rate should I expect?
It depends entirely on how repetitive your traffic is. Support and documentation assistants see substantial repetition; open-ended creative tools see almost none. Measure after the cache has warmed on real traffic, since benchmarks run on a cold cache always show no benefit.
How do I stop the cache returning stale answers?
Prefer event-based invalidation over time-based expiry. Store which source documents contributed to each cached response, and evict those entries when a source changes. Where that is impractical, set the time-to-live from the volatility of the underlying knowledge rather than a default.
Should I cache the response or the retrieved documents?
Retrieved documents first. Retrieval caching captures much of the latency saving while the model still generates fresh, so a near-miss yields slightly suboptimal context instead of a wrong answer. Response caching saves more and carries all of the correctness risk.
Why do similar questions have different answers?
Because embedding similarity measures surface meaning, not logical equivalence. Negations, dates and small qualifiers barely move a vector while completely changing the correct answer. “Cancel before 30 days” and “cancel after 30 days” are a classic pair that a loose threshold will happily conflate.
Does semantic caching work for conversational agents?
Poorly at the response level, because answers depend on conversation history that differs between sessions. The same question at different points in a dialogue can require different answers. Embedding and retrieval caching still apply and are safe.
How do I detect a false hit in production?
Sample. Take roughly 100 cache hits each week and check whether the returned answer genuinely answers the new query. False hits raise no errors and appear nowhere in your metrics, so periodic human grading is the only reliable detection method available.
References
- Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB: Massive Text Embedding Benchmark. arXiv:2210.07316. Context on how embedding behaviour varies across tasks, which underlies threshold sensitivity.
- Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401.
- Johnson, J., Douze, M., and Jégou, H. Billion-scale similarity search with GPUs. arXiv:1702.08734. Background on the nearest-neighbour search underpinning cache lookup.
