Vector search: HotKV vs Redis 8 vector sets, from 1 to 100 clients

Vector search sits underneath many AI features: retrieval for RAG, semantic caching, agent memory. Redis 8 ships vector sets, a beta data type, built into the server, so it is the obvious comparison. We loaded the same OpenAI embeddings into HotKV and into Redis 8.10.2, ran the same 1,000 queries against each with 1 to 100 concurrent client connections, and measured how accurate the answers were, how many queries per second each engine served, and how long each query took.

In short. On 999,000 vectors, at a minimum recall of 95%, HotKV served 1.4 times the queries per second of Redis with one client and 7.9 times with 100 clients (51,146 against 6,510). At 99% recall the ratios were 1.8 and 5.4. HotKV's median latency was lower at every client count. Redis's throughput levelled off by 32 clients, while HotKV's kept growing to 100. HotKV loaded the vectors about 5 times faster, and its recall was higher at every search depth. This is one dataset on one machine, the HotKV build is an unreleased development build, and we did not measure Redis's Query Engine or its 8-bit mode, so read it as a measurement, not a ranking.

What we tested

Data

The first 999,000 rows of dbpedia-entities-openai-1M by Kumar Shivendu and Nirant Kasliwal (MIT license), and, for a smaller index, the first 100,000 rows. The rows are DBpedia entities embedded with OpenAI text-embedding-ada-002, 1,536 dimensions. The last 1,000 rows are the queries, so no row appears in both sets. The answer key is the exact top 10 nearest neighbours of every query, computed by brute force in 64-bit floating point independently of either engine, so both are graded against the same ground truth.

Engines and settings

• Engines. HotKV commit c8dafd37 (VSIM.CREATEINDEX, VSIM.SET, VSIM.SEARCH), an unreleased development build compiled with default release flags, not -march=native. Redis 8.10.2 built from its release tag with a stock make (VADD, VSIM). Both build HNSW graphs.
• Index settings. Cosine similarity, HNSW with M=16 and ef_construction=200 on both, set explicitly. They are also each engine's own defaults.
• Precision. Full-precision float32 vectors on both. Redis defaults to 8-bit quantization, so it ran with NOQUANT.
• Machine. One Azure Standard_D160alds_v7 (AMD EPYC 9V45, 160 vCPUs). Both servers were pinned to the same 128 vCPUs, the load generators to the other 32, all over loopback. HotKV ran 128 threads and Redis its default I/O threads. We tried 1, 4, 8 and 16 and the setting made no consistent difference.
• Load generator. One process per client connection, each keeping one request outstanding, with the requests built before the timed part. The busiest load-generator process used 8% of one CPU in any run, so the client never set the limit.

Procedure

For search depths (ef) of 40, 80, 150 and 300, and 500 on the larger index, we ran the 1,000 queries repeatedly with 1, 2, 4, 8, 16, 32, 50 and 100 clients: one second of warm-up, then four seconds counted, three times each, with the order of the two engines alternating. We report the median. Recall was measured separately, over one connection. HotKV was loaded over 32 connections in every run. The Redis indexes were built once, over 32 connections with VADD ... CAS, saved, and loaded for the measurements. The scripts and every trial are in the benchmark repository.

Results

Matched recall targets

The two engines do not reach the same recall at the same ef, so we compare throughput at shared minimum recall targets: for each target, the fastest measured search depth that reaches it. The depths and recalls reached are in the tables, because they differ a little. This uses measured depths, not interpolation. Queries per second on 999,000 vectors:

Median latency is HotKV / Redis. The same pattern held for a 90% target: 90,003 against 7,973 queries per second at 100 clients (11.3 times), both at ef 40.

Every search depth

Queries per second on 999,000 vectors. On 100,000 vectors the pattern is the same. At 95% recall HotKV served 1.5 times Redis's queries per second with one client (1,816 against 1,214) and 8.9 times with 100 (63,247 against 7,119).

How much of this is noise

Each cell is the median of three trials. Half of the cells had a spread between their lowest and highest trial of 1.3% of the median or less, nine in ten 7% or less, and the widest was 26% (Redis, 100,000 vectors, ef 300, 50 clients). The report lists every trial.

Recall was measured once per index build. Three independent builds of each engine on 100,000 vectors differed by up to 0.013 at the same depth, so HotKV's higher recall is clear up to ef 300 and within build-to-build variation at ef 500 on the smaller index.

Loading and memory

Loaded over 32 connections each, HotKV took 188 seconds for the 999,000 vectors (5,325 per second) and Redis 935 seconds (1,069 per second), including the time the load generator spent formatting each vector. On 100,000 vectors it was 15 seconds against 57. Redis loaded faster with CAS than without in a check on 20,000 vectors (2,279 against 1,488 vectors per second), so we used it. The Redis load ran in an earlier step with Redis pinned to four vCPUs while other checks ran on the machine; HotKV loaded alone on 128 vCPUs. Treat the Redis loading time as a lower bound on its speed, not a like-for-like comparison.

After loading 999,000 vectors the HotKV process held 6.77 GB and the Redis process 7.01 GB.

What limits Redis 8.10.2 here

Redis vector sets run each search on its own thread, but the HNSW code has a fixed number of thread slots (32, set when Redis is compiled), and vectors are added on the main thread. In our runs, from 32 clients up, Redis kept between 6 and 30 of the machine's vCPUs busy and its throughput had levelled off. HotKV kept about 92 vCPUs busy at 100 clients on ef 80 and its throughput was still growing. That is how each engine behaves on this machine, not something the benchmark configured.

Limitations

These are HotKV Ltd's measurements of the versions and configuration above, not a claim about every Redis deployment.

The test setup

• A development build. Commit c8dafd37 is a development build, not a release.
• One dataset, one machine. 1,536-dimension OpenAI embeddings, top 10 only, float32, cosine. Other dimensions, data or hardware can give different ratios.
• The client shares the machine. The load generators ran on separate vCPUs but shared the memory system and the last-level cache with the servers.
• Persistence off. Neither server wrote snapshots or an append-only file.

What was not measured

• Redis's 8-bit mode. Redis stores 8-bit quantized vectors by default, which uses less memory. We compared float32 against float32.
• The Redis Query Engine. Redis also offers vector search through FT.CREATE and FT.SEARCH. We tested vector sets only, so the results say nothing about the Query Engine.
• Redis loading and reload. The Redis indexes were built in an earlier run with Redis pinned to four vCPUs, saved, and reloaded for the measurements, so the Redis load times are a lower bound and are not comparable like for like with HotKV's. Searching a reloaded index was about 10 percent slower than a freshly built one in a check on 20,000 vectors, so Redis's query figures may be low by about that much.

Try it yourself

Vector search is built into the same server as HotKV's prompt cache, and the same vector index backs its semantic cache and agent memory. Vector search is included on every HotKV Cloud plan. The prompt cache, semantic cache and agent memory come with the Performance plan and above. Start on HotKV Cloud, or talk to sales to benchmark it against your own embeddings.