Redundancy & Failover

A single point of failure (SPOF) is any one component whose death takes the whole system with it: the lone database, the one load balancer, the single power feed, the one engineer who knows how the deploy works. The entire game of availability is finding SPOFs and removing them — and the only known way to remove one is redundancy: have more than one of the thing, so the survivors carry on when one dies.

That sounds obvious, and the principle is. The subtlety — and the place real outages come from — is in how the spare takes over. This page works from “why redundancy at all” up to the surprisingly treacherous mechanics of failover.

Why redundancy works: the arithmetic of independence

Suppose one server is up 99% of the time — it’s down about 3.65 days a year. Put two independent servers behind a load balancer and require only one to be up. The system is down only when both are down at once. If the failures are truly independent, that’s 0.01 × 0.01 = 0.0001 — 99.99% availability, down from 3.65 days to about 53 minutes a year. Two cheap unreliable things composed into one reliable thing. This is the core magic, and it’s why redundancy is the foundation of every nines target in Availability, SLAs & the Nines.

The word “independent” is doing enormous work

The arithmetic only holds if failures don’t correlate. Two servers in the same rack share a power strip, a top-of-rack switch, and a cooling zone — when that fails, both die together and your “99.99%” evaporates. Real redundancy means spreading copies across failure domains: different racks, different power, different availability zones, different regions. The most expensive reliability bugs are correlated failures hiding behind a redundant-looking diagram.

The redundancy spectrum: passive to active

Redundancy is not one design — it’s a spectrum trading cost against recovery speed.

COLD STANDBY      WARM STANDBY        HOT / ACTIVE-PASSIVE   ACTIVE-ACTIVE
spare is off      spare running,      spare fully synced,    all nodes serve
must boot &       data lagging,       idle, ready instantly  traffic at once
restore           promote in seconds
└── cheapest ────────────────────────────────────────── most expensive ──┘
└── slowest recovery ──────────────────────────────── fastest recovery ──┘

Active-passive

One node serves all traffic; a standby waits. On failure, traffic fails over to the standby, which gets promoted to primary. This is the default for stateful systems — most relational databases run a primary with one or more replicas, and promote a replica when the primary dies.

What does this buy us, and what does it cost? It buys simplicity: there’s exactly one writer, so you never have to reconcile conflicting writes — consistency is easy. It costs you idle capacity (you’re paying for a standby that does no useful work) and a recovery gap: there’s a window between “primary died” and “standby is serving” where you’re down.

Active-active

Every node serves live traffic simultaneously; load is shared. Lose one and the others absorb its share — there’s no promotion step because there’s no special primary.

What does this buy us, and what does it cost? It buys no idle capacity (every machine earns its keep) and near-instant failure absorption (the survivors were already serving). It costs you the hard problem of coordination: with multiple active writers you must handle conflicting concurrent writes, which drags you into consistency trade-offs, distributed locking, or conflict resolution. Active-active is straightforward for stateless services and genuinely difficult for stateful ones.

Stateless is the cheat code

Active-active is easy when nodes hold no durable state — any node can serve any request because there’s nothing to keep in sync. This is why architectures push state down into a few carefully-managed stateful tiers (databases, caches) and keep the application tier stateless and horizontally cloned. The fewer things that hold state, the fewer things that are hard to make redundant.

Failover: the dangerous part

Here’s the counterintuitive truth: the failover mechanism is itself a common cause of outages. The redundancy is there to make you safer, but the switch-over is a high-risk maneuver, for several reasons.

Detecting failure is ambiguous. How do you know the primary is dead and not just slow, or briefly unreachable? You use health checks and timeouts — but set them too aggressive and a momentary blip triggers an unnecessary failover; set them too lax and you stay down longer than needed. There’s no perfect threshold.

Split-brain. The nightmare scenario: the primary isn’t actually dead — the network between it and the standby failed. The standby, seeing no primary, promotes itself. Now you have two primaries, both accepting writes, diverging. When the network heals, you have two conflicting versions of the truth and no clean way to merge them. Systems prevent this with quorum (a majority must agree before promotion) or fencing (forcibly disabling the old primary — “STONITH”: shoot the other node in the head).

        ┌──────────┐   network partition   ┌──────────┐
        │ PRIMARY  │  ╳ ────────────────╳  │ STANDBY  │
        │ (alive!) │                        │ promotes │
        │ accepts  │                        │ itself,  │
        │ writes   │                        │ accepts  │
        └──────────┘                        │ writes   │
              └──── both think they're it ──┘
                   = SPLIT BRAIN, diverging data

The standby was never tested. A standby that has sat idle for a year may have drifted config, an expired certificate, or insufficient capacity to actually take the full load. Failover that’s never rehearsed often fails when it’s finally needed. This is why mature teams practice failover drills and even deliberately kill primaries in production (chaos engineering) — the only trustworthy failover is one you’ve watched work.

The pragmatic synthesis

Redundancy buys availability by composing independent copies; failover is how you cash that in — but it’s a live maneuver with sharp edges. Spread copies across real failure domains, prefer stateless active-active where you can, accept the consistency cost of active-passive where you must, and rehearse the switch-over so the spare you’re paying for actually works when the weather turns.

By the numbers — the shared component sets the ceiling

The 99% → 99.99% jump assumes independence. Now make the two servers share one rack whose power and top-of-rack switch are up 99.9% of the time. The pair is up only when the rack is up and at least one server is up:

A = A_rack × (1 − (1 − A_server)²)
  = 0.999  × (1 − 0.01²)
  = 0.999  × 0.9999  ≈ 0.9989   →  ~99.9%

Redundancy lifted the server pair from 99% to 99.99% — but the shared rack drags the whole thing back down to its own 99.9%. A redundant pair can never be more available than the least-redundant thing they share. That single line is the entire argument for spreading copies across independent failure domains, and the precise arithmetic behind the caution above.

It actually happened — the AWS S3 us-east-1 outage (2017)

On 28 February 2017, an Amazon engineer following an established playbook to debug the S3 billing system mistyped a command and removed far more capacity than intended. The deletion swept up servers running two core S3 subsystems in us-east-1, and bringing them back required a full restart of those subsystems — which had not been restarted in years and had grown so large that the restart and its safety checks took roughly four hours. Because a huge slice of the internet (and many other AWS services) quietly treated us-east-1 S3 as one hidden dependency, the blast radius was enormous; AWS’s own status dashboard couldn’t even update, because it too relied on S3. Two lessons land squarely on this page: a single fat-fingered command is a correlated failure that no amount of in-region replication survives, and the recovery path — the restart — was itself the slow, dangerous, never-rehearsed-at-this-scale maneuver. Afterward AWS made the capacity-removal tool drain more slowly and refuse to take a subsystem below its safe minimum.

The architect’s lens

Redundancy is the one known cure for a single point of failure — but the cure has sharp edges. The five questions:

• Why does it exist? Because any lone component is a SPOF whose death takes the whole system with it, and the only way to remove a SPOF is to have more than one of the thing. Two independent 99% servers compose into a 99.99% pair — two cheap unreliable things made into one reliable one.
• What problem does it solve? Survival of single-component failure. A spare carries on when one copy dies — and the design choice (cold → warm → active-passive → active-active) trades cost and idle capacity against how fast the survivor takes over.
• What are the trade-offs? Active-passive buys easy consistency (one writer) but costs idle capacity and a recovery gap; active-active buys full utilization and instant absorption but costs the hard problem of conflicting concurrent writes. And the whole arithmetic rests on independence — copies sharing a rack, power, or AZ fail together and the shared component sets the availability ceiling.
• When should I avoid it? When failures are correlated anyway (same failure domain), the second copy buys far less than the math promises — spend instead on spreading domains. For stateful systems, avoid active-active unless you’ve genuinely solved conflict resolution; the coordination cost can exceed the availability gain.
• What breaks if I remove it? You’re back to a SPOF: one server, rack, or region failing takes everything down. And note the redundancy itself isn’t enough — an unrehearsed failover (drifted config, expired cert, split-brain on a partition) often fails exactly when needed, so the switch-over needs drills and fencing/quorum to be trustworthy.

Check your understanding

1. Two servers are each up 99% of the time. Why is the pair not automatically 99.99% available, and what condition must hold for that math to work?
2. Contrast active-passive and active-active on two axes: idle capacity and consistency difficulty. Which suits a stateless web tier, and why?
3. What is split-brain, what causes it, and name two mechanisms that prevent it.
4. Why is the failover itself a frequent source of outages, even when the redundancy is correctly provisioned?
5. Your “redundant” database pair lives in the same rack. Explain precisely why that undermines the availability arithmetic.

Show answers

1. The 0.01 × 0.01 = 0.0001 math only holds if the two failures are independent. The pair is not automatically 99.99% available because real failures often correlate — shared power, switch, cooling, or region. The condition that must hold is that copies sit in separate failure domains (different racks, power, AZs, regions) so they don’t die together.
2. Active-passive has idle capacity (you pay for a standby doing no useful work) but easy consistency (exactly one writer, no conflicts to reconcile). Active-active has no idle capacity and near-instant failure absorption but hard consistency (multiple writers means conflicting concurrent writes). A stateless web tier suits active-active, because there’s no durable state to keep in sync — any node can serve any request.
3. Split-brain is when both nodes accept writes and diverge. It’s caused by a network partition: the primary isn’t dead, but the standby can’t see it, so the standby promotes itself — now there are two primaries writing conflicting data with no clean merge. Two mechanisms prevent it: quorum (a majority must agree before promotion) and fencing (forcibly disabling the old primary — STONITH).
4. Failover is a high-risk live maneuver: detecting failure is ambiguous (dead vs. just slow — no perfect health-check threshold), a partition can trigger split-brain, and the standby was never tested — drifted config, an expired cert, or insufficient capacity means the spare you paid for fails exactly when needed. This is why mature teams run failover drills and chaos engineering.
5. Two servers in the same rack share a power strip, a top-of-rack switch, and a cooling zone — correlated failure domains. When any of those fails, both die together, so their failures are not independent and the 0.01 × 0.01 arithmetic collapses; the “99.99%” on the diagram evaporates into the availability of the single shared rack.