Built for the AI stack
Most teams bolt a vector database, a cache, a search engine, and a pile of glue code onto their LLM app. HotKV builds those primitives into one in-memory engine, so retrieval, memory, and cost control run next to your data.
Pick a question. Watch where it is answered, and how much of the stack it never has to touch.
- 00Questionpasses
- LLMBUDGETLLM Budgetpasses
- PCACHEPrompt Cacheanswered
- SCACHESemantic Cache
- RAGRAG Pipeline
- 05LLM
Answered from the prompt cache. The model is never called.
Times are the measured medians from the token savings benchmark below. The semantic cache time was not measured, so none is shown.
Nine AI engines, one server
Each one is a family of commands next to your data. Open any of them to see it in a real use case.
Why run AI on HotKV
The same multi-threaded core that led Redis and Valkey on every command we benchmarked also runs your retrieval, memory, and guardrails.
One engine, not five services
Vector search, semantic cache, RAG, agent memory, and a feature store all live in the same process. No separate vector DB, cache, and search cluster to deploy, secure, and keep in sync.
Sub-millisecond cache hits
Every AI primitive runs on the same multi-threaded core that led Redis and Valkey on all 12 commands in our benchmarks. In our LLM test, exact cache hits came back in about 0.05 ms, against 1.4 to 2.9 s for a model call.
Cost control in the data layer
Semantic and prompt caching cut LLM calls, embedding dedup cuts embedding calls, and per-key budgets cap spend before it happens. The savings are measured, not estimated.
Multi-tenant by design
Namespaces keep each tenant's vectors, memory, and budgets apart, and the Scale plan adds a tenant registry and an audit log for shared AI infrastructure across teams and customers.
Pay for fewer tokens
We sent the same 3,000 customer support questions to four current Claude models twice: once straight to the model, once with HotKV in front.
Tokens processed per model
What this means at scale
- Anthropic claude-fable-5-1: $303,100 a month without HotKV, $91,800 with it, about $211,300 saved every month.
- Anthropic claude-opus-5-5: $146,000 a month without HotKV, $45,000 with it, about $101,000 saved every month.
- Anthropic claude-sonnet-5-5: $55,100 a month without HotKV, $16,600 with it, about $38,500 saved every month.
- Anthropic claude-haiku-5-5: $2,700 a month without HotKV, $800 with it, about $1,900 saved every month.
At 1 million requests a day, extrapolated from the measured cost per request and assuming the same traffic pattern. The four Claude models ran with short answers and Fable 5.1 and Opus 5.5 at their lowest reasoning effort, so longer or reasoning-heavy answers would make each avoided call worth more.
Full numbersTokens, cost, answer quality and the similarity threshold trade-offShowHide
We ran Claude Fable 5.1, Opus 5.5, Sonnet 5.5 and Haiku 5.5 on the same 3,000 requests each, with and without HotKV. The provider's own prompt caching was on in every run, and tokens and costs are taken from each response's usage, thinking tokens included.
| Model | Tokens saved | Cost without | Cost with | Lower by | Model calls | Median response |
|---|---|---|---|---|---|---|
| Claude Fable 5.1 | 70% | $30.31 | $9.18 | 70% | 3,000 to 896 | 2.7 s to 127 ms |
| Claude Opus 5.5 | 70% | $14.60 | $4.50 | 69% | 3,000 to 898 | 2.9 s to 126 ms |
| Claude Sonnet 5.5 | 70% | $5.51 | $1.66 | 70% | 3,000 to 897 | 1.8 s to 128 ms |
| Claude Haiku 5.5 | 70% | $0.27 | $0.08 | 70% | 3,000 to 897 | 1.4 s to 125 ms |
How good are the cached answers?
For every reworded question served from cache, a different model compared the cached answer with the fresh answer the same model gave to that exact question, in random order. No model graded its own output. This is a language model judging, not human review, and its strictness differs between models, so compare cached with fresh only within one model.
| Model | Reworded questions judged | Cached answer fits | Fresh answer fits |
|---|---|---|---|
| Claude Fable 5.1 | 926 | 83.3% | 90.2% |
| Claude Opus 5.5 | 935 | 93.3% | 99.0% |
| Claude Sonnet 5.5 | 936 | 77.9% | 86.1% |
| Claude Haiku 5.5 | 936 | 67.9% | 78.6% |
The similarity threshold trades savings against answer quality
Each row is a separate run of Claude Sonnet 5.5 on the same 3,000 requests (cost without HotKV: $5.51).
| Threshold | Cost | Lower by | Cached answer fits | Fresh answer fits |
|---|---|---|---|---|
| 0.81 | $1.66 | 70% | 77.9% | 86.1% |
| 0.85 | $1.88 | 66% | 80.3% | 85.6% |
| 0.88 | $1.99 | 64% | 89.5% | 83.8% |
| 0.90 | $2.13 | 61% | 87.9% | 86.5% |
| 0.95 | $2.22 | 60% | 92.0% | 90.0% |
What these numbers do and do not show
- At the 0.81 threshold used for the results above, a cached answer fits the exact question 6 to 11 percentage points less often than a fresh one. Reworded questions are about 31% of requests, so across all responses that is roughly 2 to 3 points. In the threshold runs, cached answers from 0.88 upward were not worse than fresh ones, while still saving 61% to 64% of the cost.
- The "same intent" figure above does not mean "same correct answer". In a test of 119 deliberately close question pairs, where one detail such as deposit versus withdraw changes the correct answer, 62% of the changed questions would be served the cached answer at 0.81, 25% at 0.90 and 10% at 0.95. These pairs are built to be hard and show the worst case, not how often it happens in real traffic. If wrong answers are costly for your use case, raise the threshold.
- One public dataset of short banking questions, one embedding model, and an automated judge. The threshold effect was measured on one model. The other three were run at 0.81 only. Your repetition rate and questions will give different numbers.
3,000 customer-support questions drawn from the Banking77 test split with a fixed seed (intent popularity follows a Zipf distribution, s=1.0; questions are sampled uniformly within each intent, with repeats). Each question was sent to the model with the same support policy as the system prompt. Baseline: every request calls the model, with the provider's own prompt caching enabled. HotKV: exact-match lookup (GET on a SHA-256 key of the normalized question), then a semantic lookup (SCACHE.GET with an embedding of the question), and the model is called only on a miss. Token counts are taken from each API response's usage block; costs use list prices and include the embedding calls. The similarity threshold was chosen on the separate Banking77 train split as the lowest value giving at least 98% same-intent precision on semantic hits.
Embedding model: OpenAI text-embedding-3-small. Semantic match threshold: 0.81. Dataset: Banking77 (test split) by PolyAI, licensed CC BY 4.0. 3,000 requests across 77 intents.
Measured October 11, 2026. Savings depend on how often your traffic repeats itself. A workload with no repeated or similar questions saves nothing.
Ship AI features, not infrastructure
The AI command set comes with every Performance and Scale instance on HotKV Cloud. Request early access to start building retrieval, memory, and cost controls.