intermediate9 min readUpdated 2026-06-08

Data Pipelines: Batch, Streaming, Reliability and Backpressure

Design data pipelines for system design: batch vs streaming, queues, checkpoints, retries, backpressure, data quality, and operational tradeoffs.

Quick answer: Data Pipelines: Batch, Streaming, Reliability and Backpressure

Data Pipelines: Batch, Streaming, Reliability and Backpressure is covered here as a practical system design concept. Design data pipelines for system design: batch vs streaming, queues, checkpoints, retries, backpressure, data quality, and operational tradeoffs. Use it to understand the Databases idea, where it appears in production systems, and how to discuss it clearly in interviews.

Focus on the problem being solved, the main design choice, the failure mode, and the tradeoff you would defend in front of a senior engineer. This page is part of the DosiBridge system design knowledge graph, so it links related concepts, interview problems, tradeoffs, and references instead of leaving the topic isolated.

Aspect	Details
What it is	An automated system that moves data from sources to destinations, combining extraction, transformation, routing, and delivery in batch or streaming mode
When to use	Any scenario requiring data movement between systems — analytics, ML feature pipelines, cross-service data synchronization, event processing, or regulatory data feeds
When NOT to use	Simple point-to-point integrations where a direct API call suffices, or when data volumes are small enough that manual exports work fine
Real-world example	LinkedIn's data pipeline processes over 7 trillion messages per day through Apache Kafka, feeding real-time analytics, search indexing, ML models, and compliance systems simultaneously
Interview tip	Distinguish between batch and streaming pipelines, explain when each is appropriate, and describe how the Lambda architecture combines both for comprehensive data processing
Common mistake	Building only batch pipelines when the business needs real-time data — or building everything as streaming when simple daily batch jobs would suffice and be much simpler to maintain
Key tradeoff	Streaming pipelines provide low-latency data but are significantly more complex to build, test, debug, and operate compared to batch pipelines

Why This Matters

Data pipelines are the circulatory system of modern data infrastructure. Every organization beyond a certain scale needs to move data between operational databases, analytics warehouses, ML feature stores, search indexes, caches, and third-party systems. Without well-designed pipelines, data becomes siloed, stale, and inconsistent. Understanding pipeline architectures — batch vs streaming, exactly-once vs at-least-once delivery, backpressure handling, and schema evolution — is essential for any engineer building data-intensive systems. In system design interviews, pipelines are often the hidden component that makes the entire system work.

System architecture diagram for Data Pipelines showing how services, databases, and caches connect — System architecture for Data Pipelines

The Building Blocks

Batch Processing: Processes data in scheduled intervals (hourly, daily) — simpler to build, test, and debug, but introduces latency. Tools: Apache Spark, dbt, Airflow, AWS Glue.
Stream Processing: Processes data continuously as events arrive with sub-second latency. Tools: Apache Kafka Streams, Apache Flink, Spark Structured Streaming, AWS Kinesis.
Exactly-Once Semantics: Guaranteeing each record is processed exactly once (not duplicated, not lost) across failures — the hardest problem in pipeline engineering, solved via idempotent writes and transactional processing.
Backpressure: When a downstream stage can't keep up with the upstream producer's rate, backpressure mechanisms slow the producer or buffer excess data to prevent data loss or out-of-memory crashes.
Schema Registry: A centralized store for data schemas (e.g., Confluent Schema Registry) that enforces compatibility between producers and consumers, preventing breaking changes from corrupting downstream pipelines.

Under the Hood

A data pipeline consists of sources, processing stages, and sinks connected by transport mechanisms. In batch pipelines, an orchestrator (Airflow, Dagster) triggers extraction jobs on a schedule. Each job reads from a source, processes data in memory or on distributed compute (Spark), and writes to a sink. The orchestrator manages dependencies, retries, and monitoring. Batch pipelines are transactional — a job either succeeds completely or fails and can be re-run.

Step-by-step diagram showing how Data Pipelines processes a request from start to finish — How Data Pipelines works step by step

Streaming pipelines use a message broker (Kafka, Pulsar) as the backbone. Producers write events to topics, and consumer groups read from topic partitions in parallel. Each consumer maintains an offset tracking its position in the partition. Processing frameworks like Flink or Kafka Streams provide windowing (tumbling, sliding, session windows), state management (keyed state for aggregations), and checkpointing (snapshots of consumer offsets and state for failure recovery).

The Lambda architecture combines both: a batch layer reprocesses all historical data periodically for accuracy, while a speed layer processes recent events in real-time for low latency. A serving layer merges both views. The Kappa architecture simplifies this by treating everything as a stream — the batch layer is just replaying the stream from the beginning. Modern platforms increasingly adopt Kappa, using Kafka's long retention and Flink's exactly-once processing to handle both real-time and reprocessing workloads with a single pipeline.

How Companies Actually Do This

LinkedIn Processes over 7 trillion messages per day through Apache Kafka (which LinkedIn created), feeding data into Hadoop for analytics, Elasticsearch for search, Pinot for real-time dashboards, and ML models for feed ranking.

Comparison table for Data Pipelines contrasting approaches, tradeoffs, and when to use each — Comparing key aspects of Data Pipelines

Netflix Operates both batch pipelines (Spark on AWS EMR for content analytics) and streaming pipelines (Flink for real-time personalization), processing billions of events daily to power recommendations and monitor streaming quality.

Uber Their Michelangelo ML platform uses data pipelines that combine batch feature computation (Spark) with real-time feature streaming (Kafka + Flink) to serve ML features for ride pricing, ETA prediction, and fraud detection.

Common Pitfalls

Not handling late-arriving data — in streaming pipelines, events can arrive out of order; without watermarks and late-data policies, aggregations produce incorrect results or silently drop records
Coupling producers and consumers — when pipeline stages share schemas tightly, any source change breaks downstream consumers; use a schema registry with backward/forward compatibility rules
Ignoring pipeline observability — without metrics on throughput, lag, error rates, and processing time per stage, you can't detect silent data loss, growing backlogs, or performance degradation until users report bad data

Data flow diagram for Data Pipelines showing how requests and responses move through the system — Data flow through Data Pipelines

Interview Questions Worth Practicing

Design a data pipeline that feeds both a real-time dashboard and a daily analytics report from the same event stream.
How would you handle exactly-once processing in a Kafka-based pipeline that writes to a database?
Explain the Lambda and Kappa architectures — what problems do they solve, and when would you choose one over the other?

The Tradeoffs

Batch vs Streaming: Batch is simpler and cheaper but adds hours of latency; streaming is real-time but significantly more complex to build, test, and operate
Exactly-Once vs At-Least-Once: Exactly-once semantics prevent duplicates but require transactional processing with higher overhead; at-least-once with idempotent sinks is simpler and often sufficient
Flexibility vs Governance: Schema-less pipelines are easy to evolve but risk data corruption; schema registries enforce compatibility but add operational overhead and slow down changes

Component diagram for Data Pipelines showing each building block and its responsibility — Key components of Data Pipelines

How to Explain This in an Interview

Here is how I would explain Data Pipelines in a system design interview:

Frame data pipelines as the plumbing that moves data between systems. Distinguish batch (scheduled, higher latency, simpler) from streaming (continuous, real-time, complex). For batch, mention orchestrators like Airflow that manage DAGs of extract-transform-load tasks with retries and dependency management. For streaming, explain the Kafka-based pattern: producers write events to topics, consumers process them in parallel using partition-based parallelism, and frameworks like Flink handle windowing, state management, and exactly-once delivery via checkpointing. Discuss the Lambda architecture (batch + speed layers merged in a serving layer) vs Kappa (everything is a stream, batch = replay from beginning). Key operational concerns: exactly-once delivery (idempotent sinks + transactional processing), schema evolution (use a schema registry), backpressure (prevent fast producers from overwhelming slow consumers), and observability (lag monitoring, throughput metrics, data quality checks).

Interview preparation checklist for Data Pipelines with key points to mention and mistakes to avoid — Interview tips for Data Pipelines

Original DosiBridge notes for Data Pipelines: Batch, Streaming, Reliability and Backpressure

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Data Pipelines: Batch, Streaming, Reliability and Backpressure is to connect the concept to a production service where growth exposes the first weak assumption in the original design. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

Query Shape: For Data Pipelines: Batch, Streaming, Reliability and Backpressure, decide the query shape before choosing tools; otherwise the design looks clean but fails when data changes.
Write Amplification: For Data Pipelines: Batch, Streaming, Reliability and Backpressure, decide the write amplification before choosing tools; otherwise the design looks clean but fails when pipelines changes.
Replication Lag: For Data Pipelines: Batch, Streaming, Reliability and Backpressure, decide the replication lag before choosing tools; otherwise the design looks clean but fails when data changes.
Migration Safety: For Data Pipelines: Batch, Streaming, Reliability and Backpressure, decide the migration safety before choosing tools; otherwise the design looks clean but fails when pipelines changes.
Data Ownership: For Data Pipelines: Batch, Streaming, Reliability and Backpressure, decide the data ownership before choosing tools; otherwise the design looks clean but fails when data changes.

Production review question

If a team adopts Data Pipelines: Batch, Streaming, Reliability and Backpressure, what observable signal proves it helped instead of only making the architecture diagram more fashionable?

Practical metric to track

Track data-pipelines health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Data Pipelines: Batch, Streaming, Reliability and Backpressure, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.