Preventing Cascading Failures

If Service A calls Service B, and Service B is down, Service A shouldn't keep waiting and timing out. It should "trip the circuit" and fail fast.

The 3 States (Finite State Machine)

The Circuit Breaker is a state machine with three possible states:

1. Closed (Normal Operation)

• Behavior: Requests flow through to the service normally.
• Counting: We count failures (e.g., 500 errors, timeouts).
• Tripping: If failures > threshold within a time window, trip to Open.

2. Open (Tripped)

• Behavior: Requests are blocked immediately. The breaker throws a CircuitOpenException.
• Why: This prevents the "Thundering Herd" problem where 1000s of requests hammer a struggling database, preventing it from recovering.
• Reset: After reset_timeout seconds, transition to Half-Open.

3. Half-Open (The Canary)

• Behavior: We allow 1 request to pass through to test the waters.
• Success: The external service is back! Reset counts and go to Closed.
• Failure: It's still broken. Go back to Open and double the wait time (Exponential Backoff).

Implementation (Python)

Here is a thread-safe implementation of a basic Circuit Breaker.

import time
import threading

class CircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_timeout=10):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.state = "CLOSED"
        self.failures = 0
        self.last_failure_time = 0
        self.lock = threading.Lock()

    def call(self, func, *args, **kwargs):
        with self.lock:
            if self.state == "OPEN":
                # Check if it's time to try again (Half-Open)
                if time.time() - self.last_failure_time > self.recovery_timeout:
                    self.state = "HALF-OPEN"
                else:
                    raise Exception("Circuit is OPEN")

        try:
            result = func(*args, **kwargs)
        except Exception as e:
            # If we fail in Half-Open, go back to Open immediately
            self.failures += 1
            self.last_failure_time = time.time()
            if self.failures >= self.failure_threshold or self.state == "HALF-OPEN":
                self.state = "OPEN"
            raise e

        # Success! Reset everything.
        with self.lock:
            self.state = "CLOSED"
            self.failures = 0
        
        return result

Advanced Logic

1. Thundering Herd Problem

When a system recovers, if 10,000 users retry at the exact same millisecond, the system crashes again. Solution: Add Jitter (randomness) to the retry interval.

2. Bulkhead Pattern

Often used with Circuit Breakers. Isolate different parts of the system so failures don't cascade.

• Example: Connection pool for Service A is separate from Service B. If A is slow, it consumes its own pool but doesn't starve B.

Tuning the Breaker: Where the Real Engineering Lives

The state machine is the easy part; choosing the numbers is where teams get burned.

Count-Based vs. Rate-Based Tripping

A fixed failure count ("trip after 5 failures") behaves very differently at different traffic levels. At 10,000 req/s, 5 failures is statistical noise; at 2 req/s, it's total outage. Mature implementations trip on error rate over a rolling window with a minimum volume:

Trip when: error rate > 50% and at least 20 requests were observed in the last 10 seconds.

The minimum-volume clause prevents a quiet service from flipping states based on two unlucky requests. This is exactly how Netflix's Hystrix (and its successor, Resilience4j) model it.

What Counts as a Failure?

Not every error should move the needle:

• Timeouts and 5xx: yes — these indicate the dependency is unhealthy.
• 4xx client errors: no — a 404 or 400 means your request was wrong; the service is working fine. Counting these means one buggy caller can trip the breaker for everyone.
• Slow calls: the sneakiest category. A dependency answering successfully in 9.8 seconds against a 10-second timeout is effectively down — it's consuming your thread pool at 50x the normal rate. Resilience4j lets you count "slow calls" (e.g., > 2s) toward the trip threshold, which catches brown-outs that error-rate metrics miss.

Half-Open Concurrency

Allowing exactly one probe request is safest, but slow to recover a high-traffic service. Allowing many probes risks re-triggering the collapse. Common compromise: permit a small fixed number (5–10) of concurrent probes, require all of them to succeed, and ramp traffic gradually after closing rather than opening the floodgates at once.

Fallback Design: What Happens When the Circuit Is Open?

A breaker without a fallback strategy just converts slow failures into fast ones — better, but users still see errors. Design a fallback per dependency, in this order of preference:

1. Stale data: serve the last cached response ("product recommendations as of 10 minutes ago"). Users rarely notice.
2. Default/degraded response: an empty recommendations row, a generic thumbnail, a "popular items" list computed offline.
3. Queue for later: for writes, accept the request into a message queue and process when the dependency recovers ("your review will appear shortly").
4. Fail visibly: only for truly critical paths (payments) — and even then, distinguish "definitely failed" from "unknown state" in the user message.

The important discipline: fallbacks must be cheap and dependency-free. A fallback that itself calls another service creates a failure chain — the outage tour continues.

Circuit Breakers in the Modern Stack

You rarely hand-roll this in production anymore:

• Service meshes (Istio/Envoy) implement outlier detection at the proxy layer: hosts returning consecutive 5xx are ejected from the load-balancing pool for a penalty window — a per-host circuit breaker with no application code.
• Libraries: Resilience4j (Java), Polly (.NET), gobreaker (Go), opossum (Node.js) provide the pattern with metrics hooks.
• API gateways apply breakers per upstream, protecting the whole edge from one sick backend. See API Gateway Pattern.

One caution as breakers proliferate: layered breakers interact. If the mesh, the client library, and the gateway each have their own thresholds, recovery becomes hard to reason about — three timers must align before traffic flows. Pick one primary enforcement layer and use the others as backstops with looser settings.

Why use it?

• Resource Protection: Don't waste threads/connections waiting for a dead service.
• User Experience: Return a fallback (cached data or "Service Unavailable") instantly instead of a 30s spinner.
• Recovery Room: A struggling dependency gets breathing space to restart, drain queues, and warm caches — instead of being hammered back into the ground the moment it shows a pulse.

Related Concepts

• Backpressure
• Rate Limiting
• Heartbeat Protocol
• Microservices Architecture