Prompt caching vs semantic caching: what each one saves
Prompt caching makes a repeated prefix cheaper. Semantic caching removes the model call. How each works, what we measured and which to use when.
Tag
Cost optimization
Engineering.Measured and explained.
Measured API costs and token savings with HotKV, including pricing assumptions, workload sensitivity and the limits of extrapolation.
2 articles in Cost optimization
Articles
Ideas. Experiments. Updates.
HotKV cut Claude token use by 70% in our Banking77 benchmark
Four Claude models, 3,000 support requests each: HotKV caching cut model calls, tokens and API cost by about 70%. Results and answer-quality trade-offs.
Follow your curiosity