Revision · Reliability & Resilience

This part started from one honest assumption — everything fails, eventually, and usually at the worst time — and built outward from a single redundant component to surviving the loss of a whole region. The throughline is a cascade: one slow dependency ties up threads, exhausts pools, and takes down the system, so every technique here exists to break that chain at some point.

What this part covered

• Failure is the weather, not an edge case — a 10,000-machine fleet loses a node every few hours, so the goal is not preventing failure but staying useful while parts are broken.
• Redundancy & failover — remove single points of failure along the active-passive to active-active spectrum, remembering the failover itself is often the most dangerous moment.
• Timeouts, retries & backoff — you must give up at the right time, retry only what’s idempotent, and use exponential backoff with jitter so a blip doesn’t become a self-inflicted retry storm.
• Circuit breakers & bulkheads — the breaker fails fast to give a sick dependency room to heal, while the bulkhead isolates resource pools so one leak can’t sink the whole ship.
• Rate limiting — cap incoming work with token bucket, leaky bucket, or fixed/sliding windows, deciding where to enforce it and trading raw protection against fairness to users.
• Graceful degradation & load shedding — serve a worse-but-working experience instead of an outage, and deliberately drop the right low-value work to keep the core alive under overload.
• Disaster recovery (RPO/RTO) — turn “don’t lose data” into a budget with two numbers, pick a strategy along the cost curve, and remember backups are necessary but not sufficient.
• The recurring trade question — no reliability technique is pure upside; each buys survival and charges hardware, latency, weaker consistency, or operational complexity.

The takeaway

Reliability engineering is the art of placing circuit breakers on the cascade diagram — building the machinery that does nothing useful until something breaks, then contains the damage. Once you can look at any architecture and ask “when this breaks, what happens next?”, you’re ready for the next question: when it’s live and misbehaving, how do you see what’s wrong — the subject of Observability.