Semantic Caching: Similar Is Not Safe to Reuse
A note on reusing answers across differently worded questions, and the one failure that makes it dangerous: two questions can look alike and need opposite answers.
Where this helps, and where it stops
- What it is
- A short note on the specific risk of reusing answers across differently worded questions.
- What it does not guarantee
- An implementation guide, a threshold recommendation, or a treatment of provider-side prompt caching, which is a different mechanism.
- When the distinction matters
- Considering a cache in front of a model to cut cost or latency.
An ordinary cache reuses an answer when the request is identical. That rarely helps in front of a model, because people ask the same thing in endlessly different words, and every rewording is a fresh miss.
A semantic cache loosens the match. Instead of requiring identical text, it reuses a stored answer when a new question is judged close enough to an earlier one. The saving is real. So is the new failure mode.
The failure that matters
Two questions can be worded almost identically and require opposite answers.
How do I delete this database?
Is it safe to delete this database?
These are close on any measure of wording. One wants instructions. The other wants a judgement, quite possibly the word no. A cache tuned to treat them as the same question will answer the second with the first, immediately and confidently.
Similarity measures how alike two questions look. It does not measure whether the same answer is still correct, still current, or still safe. That gap is the whole risk, and tuning does not close it. A threshold controls which pairs are admitted. It does not establish that reusing the old answer is safe. Negations, conditions and scope qualifiers are the small words that move an answer most while moving the wording least, which is the wrong way round for a measure of wording.
The second problem is time
A stored answer is a snapshot. When the material behind it changes, the cache keeps serving the old one until something removes it.
So a cache needs a way to be invalidated on purpose, tied to the thing the answers depend on. If answers are grounded in a document store, an update to those documents has to reach the cache. Expiry alone is a blunt instrument: it guarantees the answer is recent, not that it is right.
There is a sharper version of this. Whatever gets written into the cache will be served to other people later, so a wrong or manipulated answer stored once is reused without the model being consulted again.
Not the same as provider caching
Providers offer their own caching, which reuses computation over a repeated prefix of the input to reduce cost or latency. That is a mechanism inside the request path, and it does not decide that your new question deserves an old answer.
The cache described here is your application choosing to skip the model entirely. Only the second one can serve the wrong answer to a different question. Worth keeping the two apart, because they are often discussed under the same word.
If you use one
Scope it to questions where reuse is clearly safe, such as stable reference material, and keep it away from anything conditional, permission-shaped or personal. Decide what invalidates an entry before switching it on. Log what was served from the cache, because otherwise a wrong reuse is invisible; it looks like the model simply answered.
For a longer treatment of cache boundaries and policy, see the semantic cache policy guide.