The Problem

In distributed systems, servers crash. Networks fail. How do you know if a server is still alive or has failed?

Solution: The Heartbeat Protocol.

• The Beat: Every T seconds (e.g., 1s), Server A sends a "Heartbeat" signal to a monitor.
• The Timeout: If the monitor doesn't receive a signal for a threshold (e.g., 3 consecutive beats), it assumes Server A is dead.

Implementation Strategies

Push Model

Server actively sends heartbeats to monitor.

import time
import threading

class HeartbeatSender:
    def __init__(self, server_id, interval=1.0):
        self.server_id = server_id
        self.interval = interval
        self.running = False
    
    def start(self):
        self.running = True
        threading.Thread(target=self._send_loop, daemon=True).start()
    
    def _send_loop(self):
        while self.running:
            print(f"[{self.server_id}] Heartbeat sent")
            time.sleep(self.interval)

Pull Model

Monitor actively pings servers.

class HeartbeatMonitor {
    async checkServer(serverUrl) {
        try {
            const response = await fetch(`${serverUrl}/health`, {
                signal: AbortSignal.timeout(5000)
            });
            return response.ok;
        } catch {
            return false; // Server is down
        }
    }
}

Advanced: Adaptive Timeout

import statistics

class AdaptiveMonitor:
    def __init__(self):
        self.latencies = []
    
    def calculate_timeout(self):
        if len(self.latencies) < 5:
            return 3.0
        
        mean = statistics.mean(self.latencies)
        stddev = statistics.stdev(self.latencies)
        return mean + (3 * stddev)  # 99.7% confidence

Real-World Examples

Kubernetes

• Kubelet sends heartbeat every 10s
• Master checks every 5s
• Grace period: 40s before marking node NotReady
• Pod eviction after 5 minutes

Redis Sentinel

• Pings master every 1s
• 5s timeout → Subjectively Down
• Quorum agreement → Objectively Down
• Triggers automatic failover

Cassandra

• Uses Phi Accrual Failure Detector
• Gossip protocol every 1s
• Phi threshold: 8 (suspicion level)
• Adaptive to network conditions

Common Patterns

Flapping Detection

Require N consecutive failures before marking dead:

class FlappingDetector:
    def __init__(self, required_failures=3):
        self.consecutive_failures = 0
    
    def heartbeat_missed(self):
        self.consecutive_failures += 1
        return self.consecutive_failures >= self.required_failures

Choosing the Timeout: The Math That Matters

The single most important design decision in a heartbeat system is the timeout multiplier — how many missed intervals you tolerate before declaring death.

Consider a heartbeat interval of 1 second. Real networks are not clean: a garbage-collection pause on the sender, a retransmitted TCP segment, or a brief spike in CPU steal time on a cloud VM can all delay a single heartbeat by hundreds of milliseconds. If you declare a node dead after one missed beat, you will trigger failovers constantly — and failovers are expensive. A database promotion can mean seconds of write unavailability, cache warm-up storms, and connection churn across every client.

A practical framework:

The rule of thumb used by most production systems: detection timeout ≥ 3 × heartbeat interval, and the action taken on detection (eviction, failover) should have an additional, longer grace period. Kubernetes embodies this layering: a node is marked NotReady after 40 seconds, but pods are not evicted for another 5 minutes — because rescheduling every pod on a node is far more disruptive than tolerating a slow node for a few minutes.

Heartbeats and Split-Brain

A heartbeat only proves that a path between two machines is alive. It cannot distinguish "the server died" from "the network between us broke." This is the root of the split-brain problem: if a primary database stops receiving heartbeat acknowledgments from its monitor, is the primary dead, or is the monitor partitioned away?

Acting on a single observer's opinion is how you end up with two primaries accepting writes simultaneously. Production systems solve this in one of three ways:

1. Quorum agreement — Redis Sentinel requires a majority of sentinels to agree a master is down (ODOWN) before failover begins. A single sentinel's opinion (SDOWN) triggers nothing.
2. Leases — instead of "I'm alive" messages, the node holds a time-bound lock (a lease) granted by a coordination service like ZooKeeper or etcd. When the lease expires, the node must stop acting as leader even if it is healthy, because it can no longer prove it holds the role. This converts failure detection from a guess into a guarantee, at the cost of requiring loosely synchronized clocks.
3. Fencing tokens — every time leadership changes, a monotonically increasing token is issued. Downstream systems reject writes carrying an old token, so even a "zombie" former leader that believes it is still alive cannot corrupt data.

Where Heartbeats Live in Real Architectures

Heartbeats appear at every layer of the stack, often under different names:

• Load balancers (health checks): an API gateway or L7 balancer polls /health endpoints every few seconds and removes failing backends from rotation. This is a pull-model heartbeat.
• Consensus protocols: in Raft, the leader's AppendEntries messages double as heartbeats. If followers miss them for an election timeout (typically 150–300 ms), they start a new election. The heartbeat is the authority signal.
• Cluster membership: Cassandra and Consul piggyback liveness information on a gossip protocol, so each node maintains a probabilistic view of the whole cluster without a central monitor — no single point of failure in the failure detector itself.
• Message consumers: Kafka consumers send heartbeats to the group coordinator; miss the session.timeout.ms window and your partitions are rebalanced to another consumer. Many production incidents trace back to a slow message handler starving the heartbeat thread — which is why Kafka separates max.poll.interval.ms (processing liveness) from the heartbeat session (process liveness).

Common Pitfalls in Production

1. Sharing a thread between work and heartbeats. If the event loop that processes requests also sends heartbeats, heavy load makes a busy node look dead — and killing busy nodes under peak load is a self-inflicted cascading failure. Always send heartbeats from a dedicated thread or sidecar.
2. Checking liveness but not readiness. A process can be alive but useless (e.g., disconnected from its database). Kubernetes distinguishes liveness probes (restart me) from readiness probes (stop routing to me). Conflating them causes restart loops during downstream outages.
3. Synchronized heartbeat storms. If 10,000 agents all report at the top of each second, the monitor sees a spike-and-silence pattern. Add per-node jitter (±10–20% of the interval) to smooth load.
4. Ignoring the failure detector's own failure. Who monitors the monitor? Centralized monitors need their own redundancy — or use decentralized detection (gossip, quorum) so no single observer's opinion is authoritative.

Key Metrics

• Heartbeat success rate
• Average latency and latency variance (feeds adaptive timeouts)
• Timeout frequency
• False positive rate (nodes marked dead that were actually alive)
• Time-to-detection for confirmed failures

Interview Tips

• Timeout tradeoff: Short = fast detection, more false positives. Long = fewer false positives, slower detection.
• Adaptive timeouts: Use mean + 3*stddev for dynamic networks (like Cassandra)
• Network partitions: Require quorum to avoid split-brain scenarios
• Push vs Pull: Push scales better for many servers; Pull better for dashboards
• Real examples: Kubernetes uses 40s grace period, Cassandra uses Phi Accrual Detector

Related Concepts

• Leader Election
• Raft Consensus
• Gossip Protocol
• Circuit Breaker