AI and LLMLLM applications

Caching LLM Gateway

Point your OpenAI or Anthropic SDK at one URL and repeated requests are answered from HotKV instead of being paid for again. No code changes.

line of code to change: 1
static binary: ~6 MB
open source: Apache-2.0

Step 1 of 5

Start the gateway

One binary or container in front of the provider. The caller's own API key is forwarded, so the gateway needs none.

Example sessionplaying

# Start the gateway

$ hotkv-gateway --upstream https://api.openai.com --semantic --embedding-model text-embedding-3-small

(gateway listening on 127.0.0.1:8080, cache in HotKV)

An animated walkthrough with real HotKV commands and example replies. Click any step to jump to it.

The problem

Most LLM traffic repeats itself. The same system prompt and question arrive again and again from different users, retries and test runs, and every one of them is billed. Adding a cache inside every service, in every language you use, is slow and easy to get wrong.

How HotKV handles it

hotkv-gateway sits between your application and the LLM API. Change the SDK's base URL and identical requests are served from HotKV, streaming responses are replayed as streams, rephrased questions can be served by meaning, and identical requests in flight share one upstream call. If HotKV is slow or down, requests go straight to the provider.

Step by step

What happens, command by command

01
Start the gateway
One binary or container in front of the provider. The caller's own API key is forwarded, so the gateway needs none.
hotkv-gateway --upstream https://api.openai.com --semantic --embedding-model text-embedding-3-small
02
First request, a miss
Nothing cached yet. The request goes upstream once, and the completed 200 response is stored.
curl -si localhost:8080/v1/chat/completions -H "Authorization: Bearer $OPENAI_API_KEY" -H "Content-Type: application/json" -d @question.json | grep -i x-hotkv-cache
03
The same request, a hit
An identical request, even with keys in a different order, is answered from HotKV and never reaches the provider.
curl -si localhost:8080/v1/chat/completions -H "Authorization: Bearer $OPENAI_API_KEY" -H "Content-Type: application/json" -d @question.json | grep -i x-hotkv-cache
04
A rephrased question
With the semantic cache on, a differently worded single-turn question can be served by the answer to a similar one.
curl -si localhost:8080/v1/chat/completions -H "Authorization: Bearer $OPENAI_API_KEY" -H "Content-Type: application/json" -d @rephrased.json | grep -i x-hotkv-cache
05
A burst of identical requests
Three identical requests arrive while the first is still in flight. They share one upstream call.
for i in 1 2 3; do curl -si localhost:8080/v1/chat/completions -H "Authorization: Bearer $OPENAI_API_KEY" -d @new.json & done

Where it fits

Real-world scenarios

Evaluation and CI runs

Test suites and prompt evaluations send the same requests on every run. Behind the gateway, only changed prompts cost anything.

Polyglot platforms

A Python backend, a Node.js worker and a Go service all get the same cache by changing one base URL, with no library in each language.

Self-hosted models

In front of vLLM or Ollama the gateway saves GPU time instead of API fees, because repeated prompts never reach the model server.

Built in

What you get

Exact and semantic cache

Identical chat, completion and embeddings requests are cached, and single-turn chat questions can also match by meaning (--semantic).

Streaming replay

Streamed responses are captured and replayed as a stream, so clients behave exactly as before.

Safe by default

Each API key has its own cache. Errors and filtered answers are never stored, and requests can skip the cache with a header.

Never in the way

If HotKV is slow or unavailable, requests go straight to the provider. The gateway also reports its numbers at /_hotkv/metrics.

The only change in your code

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1")   # the only change

# For Anthropic: start the gateway with --upstream https://api.anthropic.com
import anthropic
client = anthropic.Anthropic(base_url="http://localhost:8080")

Reference:hotkv-gateway on GitHub PCACHE.STATS

Common questions

Does the gateway need a HotKV license?

No. It works on any HotKV server. With a license that includes the prompt cache, entries live in a PCACHE namespace and HotKV tracks tokens saved. The semantic cache always needs a license.

Can two customers see each other's answers?

No. The cache key includes the caller's credentials, so each API key has its own cache, unless you keep a single upstream key in the gateway or start it with --scope global.

Which providers does it work with?

OpenAI, Azure OpenAI, OpenRouter, vLLM, Ollama, LiteLLM and anything else that speaks the OpenAI API, plus Anthropic's Messages API with the exact cache.