Circuit Breakers and Bulkheads

A circuit breaker wraps calls to a dependency and stops calling it when it's clearly failing, returning an error or a fallback immediately instead. Retries handle transient failures, but retrying a service that is completely down only wastes threads and adds latency. The circuit breaker detects these persistent failures and stops making doomed requests, which keeps one broken dependency from tying up every thread in its callers and turning one failure into a system-wide outage.

The physical analogy

The electrical circuit breaker in your home protects it from overload. When an appliance draws too much current, the breaker trips and cuts the power, so the wiring and other devices aren't damaged. Once the problem is fixed, you reset the breaker and power returns.

A software circuit breaker works the same way. When a service fails repeatedly, the breaker "trips" and stops sending it requests, protecting the callers from wasting resources on calls that will fail. After a cool-down it checks whether the service has recovered, and closes again if it has.

The problem: cascading failure

Suppose a recommendations service starts taking 30 seconds to respond. The product page calls it on every request. Each waiting call holds a thread; within seconds every thread in the product service is stuck waiting, and product pages fail even though only recommendations are broken. Retries make it worse by sending even more traffic to the struggling service.

Three states

1. Closed (normal): every request goes through. The breaker watches each one and tracks failures, errors and timeouts alike, over a rolling window. If 10 of the last 100 requests failed, the failure rate is 10%. As long as the rate stays under the threshold, the circuit stays closed.
2. Open (tripped): the failure rate crosses the threshold. A service that normally fails 5% of requests suddenly fails 60%, so the breaker opens. Now it rejects every request immediately, without calling the service, so no threads or network capacity are wasted on calls that will almost certainly fail, and the struggling service gets breathing room to recover.
3. Half-open (testing): after the cool-down period, the breaker lets a small number of test requests through. If they succeed, the service has recovered: the circuit closes and normal traffic resumes. If they fail, the service is still down and the circuit opens again for another cool-down.

Configuration

Implementation

Don't write your own: libraries such as Resilience4j (Java, the successor to Netflix Hystrix), Polly (.NET), opossum (Node.js) and pybreaker (Python) implement it. In Python with pybreaker:

import requests
from pybreaker import CircuitBreaker, CircuitBreakerError

# Open after 5 consecutive failures; try again after 30 seconds.
breaker = CircuitBreaker(fail_max=5, reset_timeout=30)

@breaker
def call_inventory_service(product_id):
    response = requests.get(f"http://inventory/stock/{product_id}", timeout=2)
    response.raise_for_status()
    return response.json()

def get_stock(product_id):
    try:
        return call_inventory_service(product_id)
    except CircuitBreakerError:
        # The circuit is open: don't call the service at all, use a fallback.
        return {"stock": None, "source": "fallback", "message": "Stock temporarily unavailable"}
    except requests.RequestException:
        # The call failed and was counted by the breaker.
        return {"stock": None, "source": "fallback", "message": "Stock temporarily unavailable"}

Note the fallback returns "unknown" rather than 0: telling customers an item is out of stock when you simply don't know would lose sales. See fallbacks for choosing safe defaults.

Monitoring circuit state

Record every state change as a metric and a log line, count how often each circuit opens, and put it on a dashboard so operators can see which dependencies are in trouble. A circuit that opens often means either the dependency is unreliable or your threshold is too aggressive. A circuit that stays open for a long time means a service is completely down and needs attention. Alert on both. See observability.

Working with fallbacks

An open circuit fails requests immediately, which is only useful if the caller has something better to do than show an error. Return a fallback instead:

• Serve cached or default data: if the recommendations circuit opens, show popular items; if the user-profile circuit opens, return cached profile data.
• Hide the feature (no recommendations carousel) while the rest of the page works.
• Queue the work to retry later (send the email when the provider recovers).
• Return a clear, fast error for things that truly can't proceed (payment authorization).

Together they make systems resilient: the breaker detects persistent failure quickly and the fallback gives an acceptable alternative, so users experience reduced functionality instead of a complete failure.

Bulkheads

A ship's bulkheads keep one flooded compartment from sinking the ship. In software, a bulkhead gives each dependency its own limited pool of threads, connections or concurrent calls. If recommendations hang, they can exhaust only their own 20-connection pool; the 200 threads serving the rest of the page are untouched.

These patterns work together: every remote call should have a timeout, run in a bounded pool, behind a circuit breaker, with limited retries and a fallback.

Tuning tips

• Count timeouts as failures; slowness is the most common way dependencies fail.
• Use a minimum request volume before evaluating the error rate.
• Keep breaker state per dependency (and per host or endpoint), so one bad replica doesn't open the breaker for all.
• Skip retries when the circuit is open: retrying a known-dead service only adds latency.