What is semantic caching?
How an AI product can skip the model and reuse an earlier answer when a new question means the same thing.
Every call to an LLM costs money and takes time, and many questions are repeats of earlier ones in different words. Semantic caching reuses the earlier answer.
Save each question with its answer. When a new question means nearly the same as a saved one, send back the saved answer and don’t call the model.
A support team keeps saved replies. “Where’s my parcel?” and “When will my order arrive?” get the same one, though they share almost no words. A semantic cache does that matching automatically.
Why “semantic”
A normal cache only matches text that is exactly the same. A semantic cache matches on meaning.
- How do I reset my password?Both caches match
- I forgot my password, what now?Only the semantic cache matches
- How do I change my email?Neither matches
How it works
- 1Embed the question — turn it into a list of numbers that captures its meaning
- 2Search the cache — find the saved question whose numbers are closest
- 3Compare with a threshold — is it close enough to count as the same question?
- 4Hit — return the saved answer. The model is never called
- 5Miss — call the model as usual, then save the new question and answer
- 01Question
- 02Embedding
- 03Close match
- 04Saved answer
The first two steps use embeddings, the same idea that powers the search step in RAG.
The hard part: how close is close enough
“How do I cancel my order?” and “How do I cancel my subscription?” look very alike as embeddings, but they need different answers. Set the threshold too loose and people get a confident answer to a question they did not ask. Set it too strict and the cache almost never matches. There is no number that works everywhere. Teams tune it on their own real questions.
When not to use it
- Personal answers. “Where is my order?” has a different answer for every customer. A saved answer must never be shared between users.
- Things that change. Prices, stock and order status go out of date, so saved answers need an expiry.
- Long conversations. If the right answer depends on what was said earlier, matching on the last message alone is not enough.
A common misunderstanding
“It’s the same as prompt caching.” Prompt caching is a feature some model providers offer. When the start of a prompt repeats, such as a long system prompt, the provider reuses its work on that part, which makes the call cheaper and faster. The model still runs and writes a fresh answer. Semantic caching skips the model altogether.
An airline’s assistant gets “can I bring my dog?”, “pets on board?” and “travelling with a cat” over and over. A semantic cache recognises them as the same question and returns the saved answer instead of paying the model each time.
Related buzzwords
Embeddings · RAG · LLM · Token