Disaster Recovery (RPO/RTO)

Redundancy (covered earlier) handles a server or a rack failing. Disaster recovery (DR) handles the catastrophe: an entire datacenter or cloud region goes dark — fire, flood, a fat-fingered region-wide deletion, a provider outage. The question stops being “is it up?” and becomes “how much data can we afford to lose, and how fast must we be back?” Those two numbers — RPO and RTO — turn a vague fear into an engineering budget.

The two numbers that define a DR plan

   ┌─────────── disaster strikes ───────────┐
   │                                         │
   ▼                                         ▼
...──[ last good backup ]───────[ X ]───────[ outage ]────────[ service restored ]──...
        └────── RPO ──────┘                  └──────── RTO ────────┘
        how much DATA you lose               how much TIME you're down

• RPO — Recovery Point Objective: the maximum acceptable data loss, measured in time. “RPO = 5 minutes” means after a disaster you may lose up to the last 5 minutes of writes. RPO is set by your backup/replication frequency.
• RTO — Recovery Time Objective: the maximum acceptable downtime. “RTO = 1 hour” means you must be serving again within an hour. RTO is set by how fast your failover/restore runs.

How to remember which is which

RPO = the Past (how far back must we recover — how much data is gone). RTO = the Time to get back. Lower numbers for either are more expensive; demanding both near-zero is the most expensive posture of all.

The DR strategies, cheapest to costliest

Each strategy is a point on the cost-vs-RPO/RTO curve. What does this buy us, and what does it cost? is literally the design question here.

Strategy	How it works	RPO / RTO	Cost
Backup & restore	Periodic backups to another region; spin up infra after disaster	Hours / hours	$
Pilot light	Core data replicated continuously; minimal infra idle, scale up on failover	Minutes / tens of min	$$
Warm standby	A scaled-down copy of the system always running in another region	Seconds-min / minutes	$$$
Multi-site active-active	Full system live in 2+ regions serving traffic simultaneously	~0 / ~0	$$$$

The jump from warm standby to active-active is where costs explode — you’re now running (and paying for) your entire stack twice and solving cross-region data consistency (see Replication and CAP & PACELC).

Why backups are necessary but not sufficient

Replication protects against hardware failure but faithfully replicates mistakes: a bad migration or a DELETE with no WHERE clause propagates to every replica instantly. Backups — point-in-time copies you can roll back to — are your only defense against logical corruption and human error.

A backup you’ve never restored is not a backup

The single most common DR failure is discovering, mid-disaster, that the backups are corrupt, incomplete, or take 12 hours to restore when your RTO was 1 hour. Test restores regularly (game days / DR drills). An untested DR plan is a hope, not a plan. Also: keep at least one backup copy isolated (different account/region/credentials) so a compromise or accidental deletion can’t take the backups with it.

Picking your targets

Don’t default to “zero.” Derive RPO/RTO from the cost of downtime and data loss versus the cost of the DR tier:

A payments ledger:   RPO ≈ 0   (losing a transaction is unacceptable) → active-active / sync replication
An analytics dashboard: RPO = 24h, RTO = 1 day (a day-old rebuild is fine) → nightly backup & restore

Different parts of one system often warrant different tiers — protect the ledger fiercely, let the recommendation cache rebuild lazily.

By the numbers — why RTO is harder than it looks

Restoring from backup isn’t instant — it’s bounded by how fast you can move the data. Suppose your primary database holds 2 TB and you can pull and write the restore at a sustained 200 MB/s:

2 TB ÷ 200 MB/s = 2,000,000 MB ÷ 200 MB/s = 10,000 s ≈ 2.8 hours

And that is just the copy — before index rebuilds, replaying the last increment of writes, DNS cutover, and validation. A casual “we’ll restore from backup” plan quietly implies an RTO measured in hours. Anything tighter forces you up the cost curve to warm standby or active-active, where the replica is already built and waiting — which is exactly the price jump in the table above.

It actually happened — GitLab’s database deletion (2017)

On 31 January 2017, a GitLab.com engineer fighting a replication problem ran rm -rf against what they believed was a stale database directory — and wiped the live primary instead, hundreds of gigabytes gone in seconds. Then came the part this page keeps warning about: of the five separate backup and replication mechanisms they thought they had, none had produced a usable recent copy — pg_dump was silently failing, the snapshotting wasn’t doing what they assumed, and so on. They recovered from a roughly six-hour-old snapshot that happened to exist on a staging server, losing about six hours of issues, comments, and accounts. The RPO they actually achieved had nothing to do with the policy on paper and everything to do with which backup happened to work. It is the danger box above in the flesh: a backup you have never restored is not a backup.

The thread

What does this buy us, and what does it cost? Every step toward zero RPO/RTO buys resilience against ever-larger disasters and costs real money and complexity (duplicate infrastructure, cross-region consistency). DR planning is the discipline of pricing the disaster honestly and buying exactly as much protection as each piece of data deserves — no more, no less.

Check your understanding

1. Define RPO and RTO precisely, and say which one each of “backup frequency” and “failover speed” determines.
2. Why does replication not protect you from a bad migration, and what does?
3. Order the four DR strategies by cost and explain what jumps so sharply at active-active.
4. Why is an untested backup dangerous, and what’s the fix?
5. Give two systems that justify very different RPO targets, and explain why.

Show answers

1. RPO (Recovery Point Objective) is the maximum acceptable data loss, measured in time — “RPO = 5 min” means you may lose the last 5 minutes of writes. RTO (Recovery Time Objective) is the maximum acceptable downtime — “RTO = 1 hour” means you must be serving again within an hour. Backup/replication frequency determines RPO; failover/restore speed determines RTO.
2. Replication faithfully replicates mistakes: a bad migration or a DELETE with no WHERE propagates to every replica instantly, so the corruption is everywhere. Point-in-time backups you can roll back to are your only defense against logical corruption and human error.
3. Cheapest to costliest: backup & restore (hours/hours, $) → pilot light (minutes/tens-of-min, $$) → warm standby (seconds-min/minutes, $$$) → multi-site active-active (~0/~0, $$$$). Costs explode at active-active because you’re running and paying for your entire stack twice and solving cross-region data consistency.
4. The most common DR failure is discovering mid-disaster that backups are corrupt, incomplete, or take 12 hours when your RTO was 1 hour — an untested DR plan is a hope, not a plan. The fix is to test restores regularly (game days / DR drills) and keep at least one backup copy isolated in a separate account/region.
5. A payments ledger justifies RPO ≈ 0 (active-active / sync replication) because losing a single transaction is unacceptable. An analytics dashboard justifies RPO = 24h (nightly backup & restore) because a day-old rebuild is perfectly fine. Different data deserves different protection — price the disaster honestly and buy exactly as much as each piece warrants.