Alerting & On-Call

Telemetry is useless if no one looks at it. Alerting is the bridge from “the system knows something is wrong” to “a human is doing something about it.” But an alert has a cost most teams underestimate: it spends a human’s attention, and worse, their trust. An alert that fires for nothing trains people to ignore alerts — including the one that mattered. This page is about firing the pager only when it should, and making sure that when it does, the person who wakes up can actually fix the thing.

The cardinal rule: alert on symptoms, not causes

This is the single most important idea in operational alerting. A cause is something internal an engineer worries about — “CPU is at 95%,” “the cache hit rate dropped,” “a replica fell behind.” A symptom is something a user actually feels — “checkout is returning errors,” “page loads take 8 seconds,” “no orders have completed in 5 minutes.”

Alert on symptoms. Here’s why the distinction is not pedantic:

• A cause without a symptom is not an emergency. CPU at 95% with users perfectly happy is not worth waking someone. If you alert on it, you’ve spent a human’s night on a non-problem — and taught them the pager lies.
• A symptom catches causes you never predicted. You cannot enumerate every way checkout can break, but you can detect that checkout is broken. Symptom alerts catch the unknown-unknowns that cause-based alerts miss entirely.

   CAUSE-based alert                  SYMPTOM-based alert
   "CPU > 90%"                        "checkout error rate > 2%"
   "cache hit rate < 80%"             "p99 latency > 1s"
   "disk 85% full"                    "successful orders/min == 0"
   ─────────────────────             ─────────────────────────────
   may fire with no user impact      fires exactly when users hurt
   misses unpredicted failures       catches the unknown-unknowns
   → belongs on a DASHBOARD          → belongs on the PAGER

The RED metrics — rate, errors, duration — are symptom metrics by construction, which is why they make the best paging signals. Causes still belong on dashboards: once a symptom pages you, the cause dashboards are how you diagnose it. The rule is about what wakes someone up, not about what you measure.

The page/ticket/log triage

Every potential alert sorts into one of three urgencies. Page (wake a human now): a user-facing symptom that’s actively bad and needs a human. Ticket (handle in business hours): degraded but not on fire — disk filling slowly. Log/dashboard (no notification): informational. Most things teams page on should be tickets. Demoting alerts is most of the work of a healthy on-call.

Error budgets: alerting tied to SLOs

How bad does a symptom have to be before it’s worth a 3 a.m. page? “Any errors at all” is too sensitive — every real system has a steady trickle of failures. The principled answer comes from SLOs and error budgets, which connect back to availability and the nines.

• An SLO (Service Level Objective) is a target: “99.9% of checkout requests succeed over 30 days.”
• The error budget is the inverse — the allowed failure. 99.9% over 30 days permits roughly 43 minutes of downtime (or its equivalent in errors) per month. That budget is a resource you are allowed to spend.

This reframes alerting entirely. You don’t page on “an error happened”; you page on the rate at which you’re burning the budget. A slow burn (you’ll exhaust the month’s budget in 3 days) is a ticket. A fast burn (you’ll exhaust it in an hour) is a page. Burn-rate alerting makes the pager proportional to actual danger:

   budget remaining ████████████░░░░░░░░  60%
                    │
   slow burn  → "you'll run out in 5 days"   → ticket, investigate tomorrow
   fast burn  → "you'll run out in 45 min"   → PAGE, act now

The error budget also resolves the eternal dev-vs-ops fight: if budget remains, ship features fast; if the budget is blown, freeze risky deploys and spend effort on reliability. The number turns a values argument into a data-driven decision.

Alert fatigue: the failure mode that kills alerting

The deadliest problem in on-call is not too few alerts — it’s too many. Alert fatigue is what happens when the pager cries wolf:

• A noisy alert fires 40 times a week, 39 of them false. The on-call learns to dismiss it reflexively — and dismisses the 40th, the real one, too.
• A flapping alert (toggling on/off on a borderline threshold) pages repeatedly for the same issue.
• One root cause fires twenty downstream alerts at once — an alert storm — burying the actual signal.

The cost is paid in human reliability: a fatigued on-call misses real incidents, burns out, and quits. Fighting fatigue is continuous work: delete alerts that never lead to action, raise thresholds that fire too eagerly, add hysteresis so alerts don’t flap, group/deduplicate related alerts into one notification, and route by severity so a ticket never buzzes a phone at 3 a.m. A good rule: every page should be actionable, novel, and urgent. If it’s none of those, it shouldn’t be a page.

An alert with no action is just noise

Before adding any alert, answer: “what will the on-call do when this fires?” If the honest answer is “look at it and go back to sleep,” it is not an alert — it is a dashboard panel. The existence of a human response is what separates a signal from noise.

Runbooks: making the page actionable

It’s 3 a.m., the pager fires, and the responder is half-asleep and not the person who wrote this service. What turns that page into a fix is a runbook — a short, specific document linked directly from the alert that says: what this alert means, how to confirm it’s real, the first diagnostic steps, known causes and their fixes, and how to escalate.

A good runbook is the difference between a 5-minute mitigation and a 2-hour fumble. It encodes the hard-won knowledge of past incidents so the next responder doesn’t rediscover it under pressure. The discipline that closes the loop is the blameless postmortem: after every incident, write down what happened, why (without blaming people — blame the system and the process), and what concrete change prevents a recurrence. Those changes flow back into better alerts, better runbooks, and often a safer deployment strategy — which is the next page.

By the numbers — what a burn rate actually means

The 99.9%/30-day SLO above gives a budget of about 43 minutes. A burn rate is how fast you’re spending it relative to “on track”: burn rate 1 spends the whole month’s budget in exactly 30 days. Google’s SRE practice turns this into concrete page thresholds:

burn rate  1    →  budget lasts 30 days    →  you're fine
burn rate  6    →  5% of budget in 6 hours →  page (slower fast-burn)
burn rate 14.4  →  2% of budget in 1 hour  →  page (fast burn)

A 14.4× burn means about 1.44% of requests are failing right now; left unchecked it would torch the entire month’s budget in roughly two days, so it pages immediately — even though only ~52 seconds of the 43-minute budget has been spent so far. That’s the whole point of burn-rate alerting: it fires on trajectory, catching a fast burn long before the budget is gone while ignoring the slow trickle a static “any errors” alert would scream about.

The architect’s lens

Alerting is the discipline that connects telemetry to a human at 3 a.m. Run it through the five questions:

• Why does it exist? Because telemetry no one looks at is useless — something has to convert “the system knows it’s wrong” into “a human is acting.” But that conversion spends a scarce, fragile resource: attention and trust.
• What problem does it solve? Catching user-affecting failure fast without drowning the responder. Symptom alerts (the RED metrics — checkout error rate, p99) catch the unknown-unknowns a fixed list of cause alerts would miss; burn-rate alerting tied to an error budget makes the page proportional to danger (14.4× burn → fires now; slow trickle → ticket).
• What are the trade-offs? Every alert costs human attention, so the dial is sensitivity vs. fatigue. Too eager and you cry wolf — 39 false pages train the on-call to dismiss the 40th real one. The cure (tuning, hysteresis, dedup, runbooks) is continuous, unglamorous work.
• When should I avoid it? Don’t page on a cause with no user symptom (CPU at 95% with happy users → dashboard, not pager); don’t page on anything non-actionable (“look at it and go back to sleep” → it’s a dashboard panel); demote most would-be pages to tickets.
• What breaks if I remove it? Incidents are discovered by angry users or a revenue dip instead of the system. Conversely, remove the discipline (not the alerts) and fatigue sets in — a buried real page is missed and the on-call burns out, which is the failure mode that actually kills alerting.

Check your understanding

1. State the symptom-vs-cause rule. Why does a symptom alert catch failures you never predicted, and why does a cause without a symptom not deserve a page?
2. Given a 99.9% monthly SLO, roughly how much downtime is the error budget? How does burn rate decide whether something is a page or a ticket?
3. How does an error budget settle the argument between shipping features fast and protecting reliability?
4. What is alert fatigue, what does it actually cost, and name three concrete tactics to reduce it.
5. What does a runbook contain, and why does linking one from every page matter most at 3 a.m.?

Show answers

1. Alert on symptoms (things a user feels — “checkout is returning errors”) not causes (things engineers worry about — “CPU at 95%”). A symptom catches unknown-unknowns: you can’t enumerate every way checkout breaks, but you can detect that it’s broken, so symptom alerts catch failures you never predicted. A cause without a symptom (CPU at 95% with happy users) isn’t an emergency — paging on it spends a human’s night on a non-problem and teaches them the pager lies; it belongs on a dashboard.
2. A 99.9% monthly SLO permits roughly 43 minutes of downtime (or equivalent errors) per month — that’s the error budget. You don’t page on “an error happened”; you page on the rate at which you’re burning the budget: a slow burn (“you’ll run out in 5 days”) is a ticket, a fast burn (“you’ll run out in 45 min”) is a page. Burn-rate alerting makes the pager proportional to actual danger.
3. It turns a values argument into a data-driven decision: if budget remains, ship features fast; if the budget is blown, freeze risky deploys and spend effort on reliability. The number — not opinion — decides.
4. Alert fatigue is what happens when the pager cries wolf — a noisy alert fires 40 times, 39 false, so the on-call reflexively dismisses the 40th real one too. It’s paid in human reliability: a fatigued on-call misses real incidents, burns out, and quits. Three tactics: delete alerts that never lead to action, raise thresholds / add hysteresis so alerts don’t flap, and group/deduplicate related alerts and route by severity. The rule: every page should be actionable, novel, and urgent.
5. A runbook says what the alert means, how to confirm it’s real, the first diagnostic steps, known causes and fixes, and how to escalate. It matters most at 3 a.m. because the responder is half-asleep and not the person who wrote the service — the runbook encodes hard-won knowledge from past incidents so the next responder doesn’t rediscover it under pressure, turning a 2-hour fumble into a 5-minute mitigation.