Change Data Capture: CDC Patterns for Real-Time Data Sync

The Problem Change Data Capture (CDC) Solves

CDC enables real-time data synchronization between databases, caches, search indexes, and analytics systems without tight coupling. It is how modern systems keep Elasticsearch in sync with PostgreSQL, or update a cache when the database changes.

How It Works Under the Hood

Change Data Capture (CDC) is a pattern that tracks changes (inserts, updates, deletes) in a database and streams them as events to other systems. Instead of polling the database for changes, CDC reads the database's transaction log (WAL/binlog) and publishes events in real-time.

Debezium connects to PostgreSQL's WAL replication slot. When a row is inserted, updated, or deleted, Debezium captures the change and publishes it to a Kafka topic (e.g., dbserver1.public.orders). Downstream consumers (Elasticsearch, cache, analytics) subscribe to the topic and update their own stores.

This creates a real-time data pipeline: Database change → WAL → CDC connector → Kafka → Consumers.

The Mental Model

• Log-based CDC: Reads the database's write-ahead log (PostgreSQL WAL, MySQL binlog). Most reliable, no impact on database performance.
• Trigger-based CDC: Database triggers fire on changes and write to a change table. Simpler but adds load to the database.
• Polling-based CDC: Periodically query the database for recent changes (WHERE updated_at > last_poll). Simple but not real-time.
• Debezium: The most popular open-source CDC tool. Reads MySQL/PostgreSQL/MongoDB logs and streams to Kafka.
• Event sourcing vs CDC: Event sourcing is designing your system around events from the start. CDC retrofits event streaming onto an existing database.

Real Systems That Depend on This

LinkedIn pioneered CDC at scale, using Databus to stream database changes to search indexes and caches.

Uber uses Debezium + Kafka for real-time data synchronization across hundreds of microservices.

Airbnb uses CDC to keep their search index (Elasticsearch) in sync with their primary database (MySQL).

Where This Shows Up in Interviews

1. What is CDC and when would you use it?
2. How does log-based CDC work?
3. What are the advantages of CDC over polling?
4. How do you handle schema changes with CDC?

Tradeoffs

• Complexity: CDC adds a data pipeline to manage (Debezium, Kafka, consumers).
• Latency: Log-based CDC has sub-second latency; polling can have minutes of delay.
• Schema evolution: Database schema changes can break CDC consumers if not handled carefully.

Watch Out For

1. Using polling when log-based CDC is available — misses changes between polls
2. Not handling schema evolution — a column rename breaks all consumers
3. Not monitoring CDC lag — downstream systems can fall behind

Go Deeper

• Pub/Sub — start here if this is new to you
• Message Queues
• Event-Driven vs Request-Driven
• Data Replication

Original DosiBridge notes for Change Data Capture: CDC Patterns for Real-Time Data Sync

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Change Data Capture: CDC Patterns for Real-Time Data Sync is to connect the concept to a production service where growth exposes the first weak assumption in the original design. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

• SLO: For Change Data Capture: CDC Patterns for Real-Time Data Sync, decide the SLO before choosing tools; otherwise the design looks clean but fails when change changes.
• Capacity Headroom: For Change Data Capture: CDC Patterns for Real-Time Data Sync, decide the capacity headroom before choosing tools; otherwise the design looks clean but fails when data changes.
• Failure Mode: For Change Data Capture: CDC Patterns for Real-Time Data Sync, decide the failure mode before choosing tools; otherwise the design looks clean but fails when capture changes.
• Degradation Path: For Change Data Capture: CDC Patterns for Real-Time Data Sync, decide the degradation path before choosing tools; otherwise the design looks clean but fails when change changes.
• Operator Signal: For Change Data Capture: CDC Patterns for Real-Time Data Sync, decide the operator signal before choosing tools; otherwise the design looks clean but fails when data changes.

Production review question

If a team adopts Change Data Capture: CDC Patterns for Real-Time Data Sync, what observable signal proves it helped instead of only making the architecture diagram more fashionable?

Practical metric to track

Track change-capture health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Change Data Capture: CDC Patterns for Real-Time Data Sync, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.