Circuit Breakers & Bulkheads

The previous page ended on a warning: when a dependency is down, retrying into it can be what keeps it down. There’s a deeper version of that insight. When a downstream service is failing, the correct thing to do is often stop calling it entirely — not retry, not even try once. Every call you make to a dead service wastes a thread waiting for a timeout, holds a connection, and adds load to something that’s already on fire. The healthiest response to a sick dependency is to leave it alone until it recovers.

That’s the circuit breaker. Its companion, the bulkhead, attacks the same cascade from a different angle: instead of stopping bad calls, it contains them so they can only damage a corner of your system. Both are named after physical safety devices, and the metaphors are exact.

The circuit breaker: fail fast, give the patient room to heal

An electrical circuit breaker trips and cuts the current when it detects a fault, protecting the wiring from melting. A software circuit breaker does the same for service calls: it sits between you and a dependency, watches the success/failure rate, and when failures cross a threshold it trips — for a while, calls to that dependency fail instantly without even being attempted.

The three states

                  failures exceed threshold
        ┌──────────────────────────────────────────┐
        │                                            ▼
   ┌─────────┐                                  ┌─────────┐
   │ CLOSED  │                                  │  OPEN   │
   │ calls   │                                  │ calls   │
   │ pass    │◄──── trial call succeeds ───┐    │ fail    │
   │ through │                              │   │ instantly│
   └─────────┘                              │   └─────────┘
        ▲                              ┌─────────┐   │
        │                              │HALF-OPEN│◄──┘ after a
        └── trial fails, back to OPEN ─│ let ONE │  cooldown timer
                                       │ trial   │
                                       │ through │
                                       └─────────┘

• Closed — normal operation. Calls flow through; the breaker just counts outcomes. If the failure rate stays low, it does nothing. (Confusingly, “closed” means working — it follows the electrical metaphor: a closed circuit conducts.)
• Open — tripped. The dependency is judged unhealthy, so every call fails immediately without touching the network. No threads wait on timeouts; no load is added to the struggling service. A cooldown timer runs.
• Half-open — after the cooldown, the breaker cautiously lets one (or a few) trial calls through. If they succeed, the dependency has recovered → back to closed. If they fail, it’s still sick → back to open for another cooldown.

Why this is so valuable

Two big wins. First, it breaks the cascade: a dead dependency stops consuming your threads and connections, so your service stays healthy and responsive (it can return a fast error or a fallback) instead of hanging and dragging its callers down. Second, it lets the sick service recover by removing the firehose of doomed requests — the open state is a deliberate gift of breathing room.

What does a circuit breaker buy us, and what does it cost? It buys fast failure and cascade containment — you stop pouring effort into a hole. It costs you availability during false trips: a breaker that trips too eagerly will cut off a dependency that was only briefly slow, turning a minor degradation into a hard “service unavailable.” Tuning the threshold, the window, and the cooldown is a real trade-off, and an over-sensitive breaker is its own kind of outage.

The breaker is most powerful paired with a fallback

Failing fast is good; failing fast with a graceful fallback is far better. When the breaker is open, instead of erroring you can serve stale cached data, a default value, or a degraded response. The breaker decides when to give up; the fallback decides what to do instead. This is the bridge to Graceful Degradation & Load Shedding.

The bulkhead: isolate so one leak can’t sink the ship

A ship’s hull is divided into watertight bulkhead compartments. Puncture one and that compartment floods — but the others stay dry and the ship stays afloat. Without bulkheads, a single breach floods the whole hull. The software pattern is identical: partition your resources so that exhausting one pool can’t starve the others.

The classic failure this prevents: your service calls dependencies A, B, and C, all sharing one thread pool of 100 threads. Dependency C gets slow. Calls to C hang and pile up, and within seconds all 100 threads are stuck waiting on C. Now calls to the perfectly healthy A and B can’t get a thread either. C’s problem has taken down A and B through shared resource exhaustion — a cascade through a shared pool.

The bulkhead fix: give each dependency its own isolated pool.

WITHOUT BULKHEADS                  WITH BULKHEADS
┌─────────────────────┐           ┌──────┐ ┌──────┐ ┌──────┐
│  shared pool (100)  │           │ A:30 │ │ B:30 │ │ C:40 │
│  C floods all 100   │           │  ok  │ │  ok  │ │FLOOD │
│  → A, B starve too  │           └──────┘ └──────┘ └──────┘
└─────────────────────┘            C drowns alone; A & B serve on

Now when C floods, it can consume at most its own 40 threads. A and B keep their pools and keep serving. The blast radius of C’s failure is contained to C.

What bulkheads cost

What does this buy us, and what does it cost? It buys fault isolation — one bad dependency can no longer starve the rest. It costs you utilization and flexibility: partitioned pools can’t share spare capacity, so under normal load you may have idle threads in A’s pool while B’s pool is momentarily saturated, where a shared pool would have lent them over. You trade some efficiency for the guarantee that a local failure stays local. Bulkheads also exist at coarser grains — separate connection pools, separate processes, separate clusters, even separate accounts — each a stronger (and more expensive) wall.

How they work together

Circuit breakers and bulkheads are complementary, not competing. The bulkhead ensures a failing dependency can only consume its slice of resources — it caps the blast radius spatially. The circuit breaker ensures that once that slice is clearly failing, you stop spending it on doomed calls and let the dependency recover — it caps the damage temporally. Together with timeouts and bounded retries, they form the standard toolkit for keeping a single sick component from becoming a system-wide outage. The recurring thread holds: each wall you build buys isolation and survival, and charges you in efficiency, idle capacity, and the risk of a misconfigured threshold cutting off something that was actually fine.

It actually happened — Netflix Hystrix (2012)

When Netflix moved onto AWS and split its monolith into hundreds of microservices, it hit exactly the cascade this page warns about: one slow dependency exhausting a shared thread pool and dragging healthy services down with it. Their answer, Hystrix, open-sourced in 2012, baked both patterns above into a single library — a circuit breaker around every remote call and per-dependency thread-pool isolation (which they explicitly called bulkheading), so a single sick downstream could consume only its own slice of threads. Hystrix is most of why these patterns became mainstream across the industry. The interesting epilogue is the cost side of the trade: Netflix moved Hystrix into maintenance mode in late 2018, having found that a fixed thread pool per dependency wastes capacity and that adaptive concurrency limits — which learn the safe limit from live latency instead of a hand-tuned number — protect about as well with less waste. Even the canonical implementation eventually paid the utilization bill that bulkheads charge.

The architect’s lens

Two patterns, one job — keeping a sick component from sinking the system. Run each through the five questions:

• Why do they exist? Because a dead dependency doesn’t just fail your calls — it holds your threads and connections while they wait on timeouts, so its sickness spreads upstream through shared resource exhaustion. The breaker stops you spending effort on doomed calls; the bulkhead caps how much you can spend.
• What problem do they solve? The cascade. The breaker caps damage temporally (once a dependency is clearly failing, fail instantly and let it heal); the bulkhead caps it spatially (one slow dependency can drown only its own pool, not the shared 100 threads that would starve healthy A and B too).
• What are the trade-offs? The breaker trades availability during false trips — too eager a threshold turns a brief slowdown into a hard “service unavailable.” The bulkhead trades utilization — partitioned pools can’t lend spare capacity, so you run with idle threads in one pool while another saturates.
• When should I avoid them? When a dependency is in-process or effectively can’t fail independently (nothing to isolate); when traffic is so low the breaker never gathers a statistically meaningful failure rate; or, for fixed-pool bulkheads, when adaptive concurrency limits would protect as well with less waste — the lesson Netflix took when it retired Hystrix.
• What breaks if I remove them? Without the breaker, your service keeps hammering a downed dependency, hangs on every timeout, and drags its own callers down. Without the bulkhead, one slow dependency exhausts the shared pool and every dependency — healthy or not — becomes unreachable.

Check your understanding

1. Why is “stop calling a failing dependency entirely” often healthier than retrying it? What two problems does that solve at once?
2. Walk through the three circuit-breaker states and the transitions between them. Why is the half-open state necessary rather than going straight from open to closed?
3. What is the cost of a circuit breaker tuned too aggressively, and how does it become its own outage?
4. Describe the shared-thread-pool cascade a bulkhead prevents, step by step.
5. Bulkheads improve isolation but hurt one specific metric. Which one, and why is that the price of containment?

Show answers

1. A call to a dead service still wastes a thread waiting for a timeout, holds a connection, and adds load to something already on fire. Stopping entirely solves both at once: it keeps your resources free (so you stay healthy and responsive) and removes the firehose of doomed requests, giving the sick dependency room to recover.
2. Closed = normal, calls flow and the breaker just counts outcomes. When failures cross the threshold it trips to open, where every call fails instantly without touching the network while a cooldown timer runs. After the cooldown it goes half-open and lets one trial call through — success returns it to closed, failure sends it back to open. Half-open is necessary because going straight to closed would dump full traffic onto a still-sick service; the single trial probes recovery without re-triggering the cascade.
3. A breaker tuned too aggressively trips on a dependency that was only briefly slow, turning a minor degradation into a hard “service unavailable.” It buys fast failure but costs availability during false trips — an over-sensitive breaker is its own kind of outage.
4. Service calls A, B, and C through one shared pool of 100 threads. C gets slow; calls to C hang and pile up; within seconds all 100 threads are stuck waiting on C. Now calls to the perfectly healthy A and B can’t get a thread either, so C’s problem has taken down A and B through shared resource exhaustion.
5. Utilization (and flexibility). Partitioned pools can’t lend spare capacity to each other, so you may have idle threads in A’s pool while B’s is saturated. That wasted efficiency is exactly what does this buy us, and what does it cost? in miniature: you trade some utilization for the guarantee that a local failure stays local.