Heartbeats in Distributed Systems

The Problem Heartbeats in Distributed Systems Solves

Failure detection is the foundation of fault tolerance. Without heartbeats, you cannot know when a server has crashed, and failover cannot begin. Every distributed system uses some form of heartbeat.

How It Works Under the Hood

Heartbeats are periodic signals sent between nodes in a distributed system to indicate they are alive and functioning. If a node stops sending heartbeats, other nodes detect the failure and take corrective action (failover, rebalancing, alerting).

Node A sends a heartbeat message to the coordinator every 5 seconds. The coordinator expects a heartbeat within 5 seconds. If 3 consecutive heartbeats are missed (15 seconds), the coordinator declares Node A as failed and initiates failover. Node A's work is redistributed to healthy nodes.

The challenge: distinguishing between a crashed node and a slow network. Too aggressive timeout = false positives (healthy node marked as dead). Too conservative = slow failure detection.

The Mental Model

• Heartbeat interval: How often heartbeats are sent (typically 1-5 seconds).
• Timeout: How many missed heartbeats trigger failure detection (typically 3 misses = 15 seconds).
• Push vs Pull: Push — each node sends periodic messages. Pull — a monitor polls each node.
• Phi Accrual failure detector: Instead of binary alive/dead, computes a suspicion level based on heartbeat timing distribution. Used by Cassandra.
• Gossip-based failure detection: Nodes gossip about each other's health. More resilient than a central monitor.

Real Systems That Depend on This

Apache ZooKeeper uses session heartbeats — clients must send heartbeats within the session timeout or the session expires.

Kubernetes uses node heartbeats — kubelets send heartbeats to the control plane every 10 seconds.

Cassandra uses a Phi Accrual failure detector with gossip protocol for decentralized failure detection.

Where This Shows Up in Interviews

1. How do you detect node failures in a distributed system?
2. What happens if a heartbeat is missed due to network congestion?
3. What is the tradeoff between heartbeat interval and failure detection speed?

Tradeoffs

• Speed vs Accuracy: Short intervals detect failures fast but increase network traffic and false positives.
• Centralized vs Decentralized: A central monitor is simple but is itself a SPOF. Gossip-based detection is resilient but slower.
• Network overhead: With 1000 nodes sending heartbeats every second, that is 1000 messages/second.

Watch Out For

1. Setting timeout too low — network jitter causes false failure detection
2. Relying on a single heartbeat monitor — the monitor is a SPOF
3. Not distinguishing between node failure and network partition

Go Deeper

• fault-tolerance — start here if this is new to you
• Service Discovery
• gossip-protocol
• consensus-algorithms
• failover

Original DosiBridge notes for Heartbeats in Distributed Systems

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Heartbeats in Distributed Systems is to connect the concept to a production service where growth exposes the first weak assumption in the original design. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

• Membership: For Heartbeats in Distributed Systems, decide the membership before choosing tools; otherwise the design looks clean but fails when heartbeats changes.
• Clock Drift: For Heartbeats in Distributed Systems, decide the clock drift before choosing tools; otherwise the design looks clean but fails when heartbeats changes.
• Retry Semantics: For Heartbeats in Distributed Systems, decide the retry semantics before choosing tools; otherwise the design looks clean but fails when heartbeats changes.
• Partition Behavior: For Heartbeats in Distributed Systems, decide the partition behavior before choosing tools; otherwise the design looks clean but fails when heartbeats changes.
• Reconciliation: For Heartbeats in Distributed Systems, decide the reconciliation before choosing tools; otherwise the design looks clean but fails when heartbeats changes.

Distinct angle: heartbeat signals are failure suspicion, not proof

A heartbeat only says a peer was recently reachable; it does not prove the peer is healthy or that the network is correct. Production systems treat missing heartbeats as suspicion and combine them with timeouts, quorum rules, and graceful removal. This nuance prevents split-brain decisions when a slow process or network partition looks like a dead node.

Production review question

If a team adopts Heartbeats in Distributed Systems, what observable signal proves it helped instead of only making the architecture diagram more fashionable?

Practical metric to track

Track heartbeats health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Heartbeats in Distributed Systems, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.