Design a Distributed Message Queue (Kafka-style)

Prompt

Design a durable, scalable message queue (think Kafka / Kinesis): producers publish, consumers read, messages survive broker crashes, throughput scales horizontally, and consumers can replay. What are the ordering and delivery guarantees?

1. The core abstraction — an append-only log

A topic is an append-only log split into partitions. Each message gets a monotonically increasing offset within its partition. Consumers track their own offset and read forward; nothing is deleted on read (unlike SQS) — messages persist for a retention period, which is what makes replay possible.

topic "clicks", partition 0:  [m0][m1][m2][m3][m4...]   ← producers append at the tail
                                              ▲
                          consumer A offset ──┘  (reads forward, commits its offset)

2. Partitions = parallelism + ordering

The partition is the unit of both scale and order:

• Parallelism: N partitions ⇒ up to N consumers in a group read in parallel. To scale throughput, add partitions.
• Ordering — only within a partition. There is no global order across partitions. To keep related messages ordered (all events for one user), key them so they hash to the same partition. This is the most common ordering gotcha.

3. Consumer groups

A consumer group splits a topic's partitions across its members — each partition is consumed by exactly one consumer in the group, so work is divided without duplication. Add consumers to scale (up to the partition count); if one dies, its partitions rebalance to the others. Different groups read the same topic independently (each at its own offset) — that's how one stream fans out to many subscribers.

4. Replication & durability

Each partition is replicated to several brokers: one leader handles reads/writes, the followers replicate it. The set of replicas caught up to the leader is the ISR (in-sync replicas). A write is acknowledged once enough replicas have it (acks=all ⇒ all ISR), so the data survives a broker loss; if the leader dies, an ISR member is promoted. The acks setting is the durability vs latency dial.

5. Delivery semantics

• At-most-once: commit offset before processing — fast, can lose messages.
• At-least-once (default): process, then commit — a crash in between causes redelivery, so consumers must be idempotent (see background jobs).
• Exactly-once: achievable within the system via an idempotent producer (dedupes retries) + transactional writes (atomic produce + offset commit). End-to-end, it still relies on idempotent side effects.

Brokers are typically pull-based (consumers fetch at their own pace), which gives natural backpressure — a slow consumer just falls behind (rising consumer lag) instead of being overwhelmed.

Design drills