Order pipelines
Orders flow to billing, shipping and analytics, each reading the same stream at its own pace.
Core dataBackground work
Event Streaming with Consumer Groups
Append-only event streams with consumer groups, acknowledgements and automatic recovery of messages a crashed consumer left behind.
- append-only log
- Durable
- consumer groups per stream
- N
- lost messages on a crash
- 0
Step 1 of 5
Publish an event
The order service appends an event and gets its id back.
Example sessionplaying
# Publish an event
hotkv> XADD orders * type created order 1182 total 49.00
"1790985600123-0"
An animated walkthrough with real HotKV commands and example replies. Click any step to jump to it.
The problem
Services that react to events need more than a queue. Several independent consumers want the same events, each consumer group needs to share work across instances, and when an instance dies with messages in hand someone has to notice and take them over.
How HotKV handles it
A stream is an append-only log. Each consumer group reads it independently, delivers each message to one member of the group, and keeps it pending until the member acknowledges it. XAUTOCLAIM hands messages that have been pending too long to a healthy consumer.
Step by step
What happens, command by command
- 01
Publish an event
The order service appends an event and gets its id back.
XADD orders * type created order 1182 total 49.00 - 02
Two groups, one stream
Billing and shipping each have their own group and each receives the event.
XGROUP CREATE orders billing 0 MKSTREAM - 03
Consume
A billing instance reads new messages for its group, blocking until something arrives.
XREADGROUP GROUP billing billing-1 COUNT 10 BLOCK 5000 STREAMS orders > - 04
Acknowledge
Once the invoice is created, billing acknowledges and the message leaves its pending list.
XACK orders billing 1790985600123-0 - 05
Recover from a crash
A shipping instance died holding messages. After 60 seconds idle they are claimed by a healthy one.
XAUTOCLAIM orders shipping shipping-2 60000 0-0 COUNT 25
Where it fits
Real-world scenarios
Audit and activity feeds
User actions are appended to a stream and replayed by any service that needs the history.
IoT ingestion
Device messages are buffered in streams and processed by a group of workers that scales with load.
Built in
What you get
Append-only log
Entries get time-ordered ids and can be trimmed by length or age.
XADDXTRIMConsumer groups
Each group gets every message, and shares the work across its members.
XGROUPXREADGROUPAcknowledgements
Messages stay pending until acknowledged.
XACKXPENDINGAutomatic recovery
Messages idle too long are claimed by another consumer.
XAUTOCLAIMXCLAIMCommon questions
Can I replay old events?
Yes. XRANGE reads any part of the stream by id, and a new consumer group can start from the beginning.
How do I stop a stream growing forever?
Trim it with XTRIM or with the MAXLEN or MINID option of XADD.
Can XADD avoid duplicate events?
XADD supports idempotent producer ids (IDMP), so a retried publish does not create a second entry. See the XADD reference.
Related use cases
Try it on your own workload
HotKV speaks the Redis protocol, so you can point an existing client at it and run these same commands.