Prompt caching vs semantic caching: what each one saves
Prompt caching makes a repeated prefix cheaper. Semantic caching removes the model call. How each works, what we measured and which to use when.
Tag
Semantic caching
Engineering.Measured and explained.
Exact and semantic answer caching with HotKV, including similarity thresholds, cache hit rates and how we evaluate answer quality.
2 articles in Semantic caching
Articles
Ideas. Experiments. Updates.
HotKV cut Claude token use by 70% in our Banking77 benchmark
Four Claude models, 3,000 support requests each: HotKV caching cut model calls, tokens and API cost by about 70%. Results and answer-quality trade-offs.
Follow your curiosity