Design a Message Queue
Design a distributed message queue with topics, partitions, consumer groups, ordering guarantees, and at-least-once delivery.
Quick answer: Design a Message Queue
Design a Message Queue is covered here as a practical system design interview problem. Design a distributed message queue with topics, partitions, consumer groups, ordering guarantees, and at-least-once delivery. Use it to practice requirements, APIs, storage choices, scaling bottlenecks, data consistency, observability, and failure handling for a medium interview round.
Focus on the problem being solved, the main design choice, the failure mode, and the tradeoff you would defend in front of a senior engineer. This page is part of the DosiBridge system design knowledge graph, so it links related concepts, interview problems, tradeoffs, and references instead of leaving the topic isolated.
Problem Statement
Design a distributed message queue (like Kafka) that enables decoupled, asynchronous communication between producer and consumer services. The system must support topics with partitions for parallel consumption, consumer groups for load distribution, configurable delivery guarantees (at-least-once, at-most-once, exactly-once), and message ordering within partitions.
Requirements
Functional
- Producers publish messages to named topics; messages are routed to partitions by key (or round-robin if no key)
- Consumer groups: each partition is consumed by exactly one consumer in a group; rebalanced on consumer join/leave
- Message ordering guaranteed within a partition (not across partitions)
- Configurable retention: time-based (e.g., 7 days) or size-based (e.g., 1 TB per partition)
Non-Functional
- Throughput: 1M messages/second per topic, 10M aggregate across all topics
- Latency: <10ms end-to-end (producer publish to consumer receive) at p99
- Durability: No message loss -- messages replicated to 3 brokers before acknowledgment
- Scale: 10,000+ topics, 100,000+ partitions across a cluster
Core Architecture
-
Broker Cluster -- Each broker manages a set of partitions. A partition is an ordered, append-only log stored on disk. Writes are sequential (disk-friendly), achieving 500 MB/s per broker. Each partition has a leader broker (handles reads/writes) and 2 follower replicas (sync writes for durability).
-
Partition Manager -- Assigns partitions to brokers using a balanced placement algorithm. When a broker fails, reassigns its leader partitions to in-sync replicas (ISR) within seconds. Uses ZooKeeper (or KRaft in newer designs) for leader election and metadata management.
-
Consumer Group Coordinator -- Tracks consumer membership per group. When consumers join or leave, triggers a rebalance: reassigns partitions to consumers using a sticky assignment strategy (minimizes partition movement). Each consumer commits its offset (position in the log) periodically to track progress.
- Producer Client -- Batches messages (configurable by size or time), compresses batches (lz4/snappy), and sends to the leader broker for each partition. Supports configurable acks: acks=0 (fire-and-forget), acks=1 (leader only), acks=all (all ISR replicas). Retries on failure with idempotent producer IDs to prevent duplicates.
Database Choice
No external database for message storage -- messages are stored in the brokers' local filesystem as append-only segment files. Each segment is a sequential log file (e.g., 1 GB) with an index file mapping offsets to file positions. This exploits sequential disk I/O and OS page cache for near-memory-speed reads. ZooKeeper/KRaft for cluster metadata (broker registry, topic configs, partition assignments, consumer group offsets). Consumer offsets are stored in a special internal topic (__consumer_offsets).
Key API Endpoints
POST /api/v1/topics/\{topic\}/messages
-> Body: \{ key: "user-123", value: "\{order_id: 456\}", headers: \{\} \}
-> Returns: \{ partition: 7, offset: 10482 \}
GET /api/v1/topics/\{topic\}/partitions/\{partition\}/messages?offset=10480&limit=100
-> Returns: \{ messages: [\{ offset: 10480, key: "...", value: "...", timestamp: ... \}] \}
POST /api/v1/consumer-groups/\{group\}/offsets
-> Body: \{ offsets: [\{ topic: "orders", partition: 7, offset: 10483 \}] \}
Scaling Insight
The append-only log with sequential I/O is what makes message queues so fast. Random disk writes achieve ~100 IOPS; sequential writes achieve 500+ MB/s (equivalent to millions of small messages per second). By writing all messages sequentially to segment files and using the OS page cache for reads, a single broker can sustain 1 GB/s throughput on commodity hardware. Adding more partitions (and brokers) scales linearly.
Key Tradeoffs
| Decision | Option A | Option B | Chosen |
|---|---|---|---|
| Delivery guarantee | At-most-once (fast) | At-least-once (safe) | At-least-once default -- consumers must be idempotent; lost messages are worse than duplicates |
| Storage | In-memory only (fast, volatile) | Disk-backed log (durable) | Disk-backed -- sequential I/O is nearly as fast, provides retention and replay capability |
| Metadata coordination | ZooKeeper (external) | KRaft (built-in Raft) | KRaft -- eliminates operational dependency, simpler deployment |
Original DosiBridge notes for Design a Message Queue
This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Design a Message Queue is to connect the concept to an asynchronous workflow where replay, ordering, and poison messages shape the architecture. That framing changes the design discussion from definitions to operating choices.
What makes this topic distinct
- Core Entity: For Design a Message Queue, decide the core entity before choosing tools; otherwise the design looks clean but fails when message changes.
- Read/Write Path: For Design a Message Queue, decide the read/write path before choosing tools; otherwise the design looks clean but fails when queue changes.
- Contention Point: For Design a Message Queue, decide the contention point before choosing tools; otherwise the design looks clean but fails when design changes.
- Abuse Path: For Design a Message Queue, decide the abuse path before choosing tools; otherwise the design looks clean but fails when message changes.
- Recovery Workflow: For Design a Message Queue, decide the recovery workflow before choosing tools; otherwise the design looks clean but fails when queue changes.
Design a Message Queue correctness boundary
For this problem, separate the user-visible action from background work. The correctness boundary is the smallest state transition that must not be duplicated, lost, or reordered. Naming that boundary makes the solution more original than a stock API/database/cache template.
Design a Message Queue scaling bottleneck
The first bottleneck should be described in the language of this system: a hot timeline, a contested seat, a busy wallet ledger, a media-processing queue, or a search index update delay. That bottleneck decides which component deserves the deepest interview discussion.
Production review question
If Design a Message Queue suddenly receives 10x traffic, which entity becomes contested first, and what queue, lock, shard, or cache protects that entity without hiding correctness bugs?
Practical metric to track
Track message-design health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Design a Message Queue, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.