Customer support
Password resets, invoice copies and shipping times are asked again and again. Each variant after the first is served from cache.
AI and LLMLLM applicationsPerformance and Scale plans
Semantic Response Caching
Cache LLM answers by meaning, not exact text, so paraphrased questions hit the cache and skip the model call.
- lookup before every model call
- 1
- similarity metrics
- 3
- threshold override
- Per call
Step 1 of 6
Create a namespace
Set the embedding size and the similarity that counts as a hit.
Example sessionplaying
# Create a namespace
hotkv> SCACHE.CREATE support_bot 1536 THRESHOLD 0.92
OK
An animated walkthrough with real HotKV commands and example replies. Click any step to jump to it.
The problem
A support assistant receives the same question phrased a hundred different ways. An exact-match cache misses nearly all of them, so every paraphrase pays for a full model call and waits for the full response.
How HotKV handles it
A semantic cache stores each answer next to the embedding of its question and looks new questions up by similarity. If a new prompt is close enough to one you have answered, the stored answer comes back and the model is never called. The threshold decides how close is close enough.
Step by step
What happens, command by command
- 01
Create a namespace
Set the embedding size and the similarity that counts as a hit.
SCACHE.CREATE support_bot 1536 THRESHOLD 0.92 - 02
A miss, so call the model once
The first time a question is asked there is nothing similar in the cache, so the model answers it.
SCACHE.GET support_bot 0.013 -0.21 0.88 ... - 03
Remember the answer
Store the question, the answer and the question's embedding, for one day.
SCACHE.SET support_bot "How do I reset my password?" "Open Settings > Security." 0.013 -0.21 0.88 ... TTL 86400 - 04
A paraphrase hits
"forgot my login pass" lands close to the stored question. The answer and its similarity come back, and the model is skipped.
SCACHE.GET support_bot 0.011 -0.19 0.87 ... - 05
Stricter for one lookup
A lookup can ask for a closer match than the namespace default, for answers where a near miss is not good enough.
SCACHE.GET support_bot 0.2 0.4 0.1 ... THRESHOLD 0.99 - 06
Measure the savings
Statistics per namespace show how often the cache answered instead of the model.
SCACHE.STATS support_bot
Where it fits
Real-world scenarios
Internal knowledge bots
Employees ask about holidays, expenses and VPN access in their own words. One answer serves all of them until it expires.
Developer tools
"How do I paginate?" and "how to page through results" get the same documented answer without a second model call.
Built in
What you get
Match by meaning
Similar questions share one cached answer, however they are worded.
SCACHE.GETSCACHE.SETYou control the strictness
Set the threshold per namespace and override it on a single lookup.
SCACHE.THRESHOLDSavings you can measure
Per-namespace statistics show hits, misses and entries.
SCACHE.STATSCosine, L2 or inner product
Pick the metric that matches your embedding model when you create the namespace.
SCACHE.CREATECommon questions
What does a cache hit cost?
One embedding call and a cache lookup. The model call and its generation time are skipped.
How do I avoid wrong answers from a loose match?
Raise the threshold. A higher threshold needs a closer match, so fewer questions hit and each hit is more certain.
Which plans include it?
The semantic cache is part of the Performance and Scale plans.
Related use cases
Try it on your own workload
HotKV speaks the Redis protocol, so you can point an existing client at it and run these same commands.