Deployment Strategies

Every outage has a leading cause that dwarfs hardware failure: someone deployed something. A deployment strategy is how you ship change while keeping the blast radius small and the undo button fast. The goal isn’t to never break things — it’s to break them for few users, briefly, and to recover instantly.

The naïve way and why it hurts

“Stop the old version, start the new one” (a recreate deploy) means downtime during the swap and, worse, an all-or-nothing bet: if the new build is bad, 100% of users hit it at once and your only recovery is another full deploy. Everything below exists to shrink that bet.

Rolling deployment

Replace instances gradually — a few at a time — so the service stays up throughout.

[v1][v1][v1][v1]  →  [v2][v1][v1][v1]  →  [v2][v2][v1][v1]  →  [v2][v2][v2][v2]
            (old and new run side by side during the roll)

• Buys: no downtime, no extra fleet cost.
• Costs: v1 and v2 run simultaneously (your code and DB schema must tolerate that — see the migration note below), and rollback means rolling back, which is slow.

Blue-green deployment

Run two full environments. “Blue” serves production; “green” gets the new version. Test green, then flip the load balancer to it instantly.

   LB ──► Blue (v1)  [live]        Deploy v2 to Green, test it
   LB ──► Green (v2) [live]   ◄── flip the LB; Blue stays warm as instant rollback

• Buys: instant cutover, instant rollback (flip back to blue), test in a prod-identical env.
• Costs: you pay for double the infrastructure during the deploy, and shared state (the database) still has to be compatible across both.

Canary deployment

Route a small slice of real traffic (1% → 5% → 25% → 100%) to the new version while watching metrics. If error rate or latency climbs, abort before most users are affected.

  99% ──► v1
   1% ──► v2   ◄── watch p99, error rate, business metrics
              healthy? widen to 5%, 25%, 100%.  bad? route back to 0%.

• Buys: the smallest blast radius — real-traffic validation with most users protected.
• Costs: needs solid metrics and alerting to decide automatically, plus traffic-splitting infrastructure.

Canary is observability’s payoff

A canary is only as good as your ability to tell the new version is misbehaving. This is where Part 7 pays off: without per-version RED metrics and tracing, a canary is just a slow rollout you can’t evaluate. Automated canary analysis compares the canary’s metrics against the baseline and promotes or rolls back on its own.

Feature flags: decouple deploy from release

The most important idea here: deploying code and releasing a feature don’t have to be the same event. Ship the code dark behind a flag, then turn it on for 1% of users — or off instantly if it misbehaves — without a redeploy.

deploy (code present, flag OFF)  ──►  flip flag ON for 1% ──► 100%
                                       (kill switch = flip OFF, no deploy)

This turns “rollback” from a deploy into a config change measured in seconds, and lets you separate the engineering risk (the deploy) from the product decision (the release).

The hidden constraint: backward-compatible changes

Rolling, blue-green, and canary all run old and new code at once against one shared database. That means schema changes must be backward-compatible: add columns before you read them, never rename or drop in a single step. The safe pattern is expand → migrate → contract across multiple deploys. A “simple” non-compatible migration is how teams turn a zero-downtime strategy into an outage.

It actually happened — Knight Capital (2012)

On 1 August 2012, the trading firm Knight Capital deployed new order-routing code to its production servers — but the rollout missed one of the eight machines. Worse, the deploy repurposed an old feature flag that, on that one stale server, switched on long-dead code called “Power Peg” instead of the new behaviour. The fleet was now inconsistent: seven servers running the new logic, one running a zombie from years past. In roughly 45 minutes that single server fired millions of unintended orders into the market, and Knight lost around $440 million — enough to destroy the firm, which was acquired soon after. Two failures this page warns about, in one incident: a deploy that wasn’t applied uniformly (a canary, or any deploy that verified every node, would have caught the odd server out), and a feature flag reused to mean something new rather than created fresh and deleted once its old code was dead.

The thread

What does this buy us, and what does it cost? Each strategy trades infrastructure cost and complexity for a smaller blast radius and faster recovery. Recreate is cheap and dangerous; canary with feature flags is the gold standard but demands real observability and disciplined, compatible schema changes. Pick the cheapest strategy that makes your rollback fast enough for the damage a bad deploy could do.

The architect’s lens

Before you pick a deploy strategy (or reach for feature flags), answer the five questions:

• Why does it exist? Because the leading cause of outages isn’t hardware — it’s someone deployed something, and a naive recreate deploy is an all-or-nothing bet that puts a bad build in front of 100% of users at once with no fast undo.
• What problem does it solve? Shrinking blast radius and recovery time. Rolling keeps the service up with no extra fleet; blue-green gives instant cutover and instant rollback; canary validates on a 1%→5%→25% slice so most users are protected; feature flags turn rollback into a config flip measured in seconds.
• What are the trade-offs? Each strategy buys a smaller blast radius with more cost/complexity: blue-green pays for double the infrastructure during the deploy; canary demands solid per-version metrics and alerting plus traffic-splitting plumbing. All of them run old and new code at once, so schema changes must be backward-compatible.
• When should I avoid it? Recreate is fine when brief downtime is acceptable and the system is trivial. Skip canary if you lack the observability to evaluate it — it’s just a slow rollout you can’t read. And never ship a non-compatible migration in one step; use expand → migrate → contract.
• What breaks if I remove it? Without a gradual strategy, every deploy is a coin flip that can hit all users instantly with a slow, full-redeploy recovery. Knight Capital (2012) is the cautionary tale: a non-uniform deploy plus a reused feature flag turned one stale server into a $440M, 45-minute loss.

Check your understanding

1. Why is a “recreate” deploy risky beyond just causing downtime?
2. Contrast the cost and rollback speed of rolling vs blue-green.
3. What makes canary the smallest-blast-radius option, and what capability is it entirely dependent on?
4. How do feature flags separate “deploy” from “release,” and why does that speed up rollback?
5. Why must schema changes be backward-compatible for these strategies, and what is the expand→migrate→contract pattern?

Show answers

1. Beyond the downtime during the swap, a recreate deploy is an all-or-nothing bet: if the new build is bad, 100% of users hit it at once, and your only recovery is another full deploy. There’s no small blast radius and no fast undo.
2. Rolling costs nothing extra (no spare fleet) but rollback is slow — you have to roll back gradually. Blue-green costs double the infrastructure during the deploy, but gives instant rollback — flip the load balancer back to the still-warm blue environment.
3. Canary routes only a small slice of real traffic (1% → 5% → 25% → 100%) to the new version while watching metrics, so most users are protected and you abort before they’re affected — the smallest blast radius. It’s entirely dependent on observability: solid per-version RED metrics and alerting, without which a canary is just a slow rollout you can’t evaluate.
4. Deploying code and releasing a feature don’t have to be the same event — ship the code dark behind an OFF flag, then flip it ON for 1% of users without a redeploy. This turns rollback from a deploy into a config change measured in seconds (the kill switch is just flipping the flag back OFF), separating engineering risk from the product decision.
5. Rolling, blue-green, and canary all run old and new code at once against one shared database, so the schema must work for both versions simultaneously — a non-compatible migration turns a zero-downtime strategy into an outage. Expand → migrate → contract: first expand (add the new column/structure, backward-compatibly), then migrate (move data and shift reads/writes over across deploys), then contract (drop the old structure once nothing uses it) — never rename or drop in a single step.