Redis Streams Fundamentals

Redis Streams are append-only logs that maintain ordered message sequences. Each stream uses a unique identifier for isolation between users or sessions.

Core operations: XADD appends messages, XREAD consumes them sequentially, XRANGE reads historical data. Blocking reads wait for new data with timeout support.

Delta Processing Pattern

Deltas are incremental message fragments that arrive piece by piece. Instead of resending complete messages, deltas contain only new or changed portions, reducing bandwidth and memory usage.

Delta structure includes ID, content fragment, timestamp, and metadata. Multiple producers write concurrently while Redis maintains message ordering.

Processing flow: generate deltas → push to Redis streams via XADD → consume sequentially.

Concurrent Processing with Go

Go routines handle stream processing independently of HTTP request lifecycles. Streaming continues even if the original request terminates.

Consumer pattern uses infinite loop: read from stream, process messages, sleep 100ms if empty. This balances responsiveness with resource efficiency.

for {
    msgs := redis.XRead(stream, lastID)
    if len(msgs) == 0 {
        time.Sleep(100 * time.Millisecond)
        continue
    }
    processDelta(msgs)
    updateLastID(msgs)
}

Channels enable safe communication between goroutines for producer-consumer patterns.

Memory Management

TTL automatically expires inactive streams to prevent memory leaks. Monitor activity and extend TTL for active streams while letting inactive ones expire.

For high-volume scenarios, use XTRIM to limit stream length or implement retention policies.

Real-Time Communication

Server-Sent Events work with Redis streams for real-time updates. Server maintains long-lived HTTP connections and pushes deltas as they arrive.

func streamHandler(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "text/event-stream")

    for {
        deltas := readFromRedisStream()
        for _, delta := range deltas {
            fmt.Fprintf(w, "data: %s\n\n", delta)
            w.(http.Flusher).Flush()
        }
    }
}

Browser EventSource API provides automatic reconnection handling.

Error Handling

Use connection pooling and automatic reconnection for Redis reliability. Circuit breaker patterns handle Redis unavailability.

Consumer groups provide at-least-once delivery with acknowledgments. Failed processing triggers retries or dead letter queues.

Monitor: stream length, consumer lag, processing time. Use XINFO STREAM for statistics and slow log analysis for debugging.

Redis streams learnings

Comments

More from this blog

Become a cracked product engineer today

Bipedal, humanoid, and the words for creature shapes in games

Lerp and smoothstep, what they actually do

IK and FK, what they actually are

How to make your textures fast on the GPU

Redis Streams Fundamentals

Delta Processing Pattern

Concurrent Processing with Go

Memory Management

Real-Time Communication

Error Handling

Command Palette

Comments

More from this blog

Redis Streams Fundamentals

Delta Processing Pattern

Concurrent Processing with Go

Memory Management

Real-Time Communication

Error Handling