Discord Message Storage: From MongoDB to ScyllaDB

Company Context

Discord serves over 150 million monthly active users exchanging billions of messages daily. Messages must be stored durably, served with low latency, and queried efficiently — primarily by channel, with the most recent messages displayed first. The message storage system is one of Discord's most critical infrastructure components.

The Problem at Scale

Discord's first message store was MongoDB, which worked well initially but could not handle the growing data volume. They migrated to Cassandra, choosing it for its write throughput and linear scalability. Cassandra worked for years, but as Discord grew to trillions of stored messages, operational pain increased. Cassandra's JVM-based architecture caused unpredictable GC pauses that spiked tail latencies. Compaction on large datasets consumed enormous I/O. Maintenance operations like repair and anti-entropy became prohibitively expensive. Read latencies on cold data were unacceptable for Discord's real-time experience.

Architecture Solution

Discord migrated from Cassandra to ScyllaDB, a C++ reimplementation of Cassandra's protocol that eliminates JVM garbage collection and uses a shard-per-core architecture for predictable performance.

The data model partitions messages by (channel_id, bucket), where a bucket is a time window (roughly 10 days). This ensures each partition stays bounded in size — a critical design for LSM-tree databases where large partitions cause compaction problems. The primary key within each partition is the message snowflake ID (a time-based unique identifier), which naturally sorts messages chronologically.

A data services layer sits between the application servers and ScyllaDB. This layer handles request coalescing (if 100 users open the same channel simultaneously, only one database query is issued), consistent routing, and write-behind caching. This service eliminated hot-partition thundering herd problems that previously overwhelmed the database during popular events.

The migration itself was performed with zero downtime using a dual-write strategy: writes went to both Cassandra and ScyllaDB, then reads were gradually shifted over after data validation confirmed consistency.

Key Techniques Used

• Partition key design: channel_id + time bucket keeps partitions bounded
• Snowflake IDs: Time-sortable unique identifiers eliminate the need for a separate sort key
• Request coalescing: Data services layer deduplicates identical concurrent queries
• Shard-per-core architecture: ScyllaDB assigns each CPU core its own data shard, eliminating lock contention
• Dual-write migration: Zero-downtime migration by writing to both databases simultaneously
• Write-behind caching: Reduces read load on the database for recently accessed channels

Lessons for System Design Interviews

This case study is a strong reference for "design a chat system" questions. Demonstrate awareness that partition key design is the most impactful decision in wide-column databases. Show that you understand why time-bucketing prevents unbounded partition growth. Mention the data services layer as an example of application-level caching and request deduplication, which is often more effective than database-level optimization.

Lessons for Production

Database technology choices are not permanent — Discord migrated twice as their scale changed. The data services layer was arguably more impactful than the database migration itself, showing that a well-designed application layer can compensate for database limitations. Time-bucketed partition keys are a proven pattern for time-series-like data in wide-column stores.

Original DosiBridge notes for Discord Message Storage: From MongoDB to ScyllaDB

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Discord Message Storage: From MongoDB to ScyllaDB is to connect the concept to a fan-out system where freshness, delivery order, and hot users create the real scaling pressure. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

• Practice Loop: For Discord Message Storage: From MongoDB to ScyllaDB, decide the practice loop before choosing tools; otherwise the design looks clean but fails when discord changes.
• Implementation Exercise: For Discord Message Storage: From MongoDB to ScyllaDB, decide the implementation exercise before choosing tools; otherwise the design looks clean but fails when messages changes.
• Review Checkpoint: For Discord Message Storage: From MongoDB to ScyllaDB, decide the review checkpoint before choosing tools; otherwise the design looks clean but fails when discord changes.
• Failure Drill: For Discord Message Storage: From MongoDB to ScyllaDB, decide the failure drill before choosing tools; otherwise the design looks clean but fails when messages changes.
• Comparison Target: For Discord Message Storage: From MongoDB to ScyllaDB, decide the comparison target before choosing tools; otherwise the design looks clean but fails when discord changes.

Production review question

If a team adopts Discord Message Storage: From MongoDB to ScyllaDB, what observable signal proves it helped instead of only making the architecture diagram more fashionable?

Practical metric to track

Track discord-messages health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Discord Message Storage: From MongoDB to ScyllaDB, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.

Sources

• How Discord Stores Trillions of Messages — Discord Engineering, 2023