Metrics

A log tells you about one event. A metric tells you about all of them at once: “we served 12,000 requests this minute, 0.3% of them failed, and the slowest 1% took over 800 ms.” Metrics are the cheap pillar of observability — they get cheap precisely by throwing detail away and keeping only numbers aggregated over time. That trade is what makes them perfect for dashboards and alerts and useless for explaining any single request. This page builds the metric types from scratch, then gives you two ready-made frameworks for deciding which metrics to collect.

Why not just count logs?

You could answer “how many requests failed?” by querying logs. But aggregating billions of log lines on every dashboard refresh is slow and expensive. A metric pre-aggregates: instead of storing a line per event, you store a running number per time window. The system increments a counter in memory and periodically writes one compact data point — “errors at 14:03 = 37.” A dashboard reading that is reading a handful of numbers, not scanning a haystack.

What does this buy us? Cheap storage and instant queries over long time ranges. What does it cost? You lose the detail — a metric can tell you the error rate spiked, never which user or why. That’s why metrics point you at a problem and logs and traces explain it. Metrics are the smoke alarm; the other pillars are the investigation.

The three metric types

Almost every metric is one of three shapes:

   COUNTER          GAUGE              HISTOGRAM
   only goes up     up and down        distribution of values
   (or resets)      a snapshot         bucketed
   ───────────      ──────────         ──────────────
   total requests   current memory     request latency
   total errors     queue depth        response sizes
   bytes sent       active connections  → p50, p90, p99

• Counter — a monotonically increasing total: requests served, errors, bytes sent. You never read the raw value; you read its rate of change (“requests per second”). Counters survive resets cleanly because you only care about deltas.
• Gauge — a value that goes up and down, sampled now: memory in use, queue depth, active connections, temperature. A gauge is a snapshot, not a total.
• Histogram — the most important and most subtle. It buckets observations to capture a distribution, so you can ask for percentiles. This is the only way to see tail latency. An average latency of 100 ms can hide a p99 of 3 seconds — and the average is a lie precisely for the users who are suffering. Never alert on averages; alert on percentiles.

Why averages lie

Imagine 99 requests at 50 ms and 1 request at 5,000 ms. The average is ~100 ms — a number no single request actually experienced, and one that completely hides the user stuck waiting 5 seconds. Percentiles (p50, p99) describe what real users feel. A histogram is what lets you compute them server-side without storing every individual timing.

The RED method: instrument every service

Knowing the metric types doesn’t tell you which metrics to collect. The RED method gives a dead-simple default for any request-driven service. For every service, track three things:

   R — Rate       requests per second
   E — Errors     failed requests per second (or error %)
   D — Duration   latency distribution (p50/p90/p99 via histogram)

These three answer “is this service healthy from the caller’s point of view?” — which is exactly the symptom you want to alert on (more in Alerting & On-Call). RED is request-centric: it describes the experience of whoever is calling the service.

The USE method: instrument every resource

RED watches the work; the USE method watches the machine doing the work. For every resource — CPU, memory, disk, network, a connection pool — track:

   U — Utilization   % of time the resource was busy
   S — Saturation    how much work is queued/waiting (the backlog)
   E — Errors        error events from the resource

USE is resource-centric and catches a different failure: a service whose RED metrics still look okay but whose database connection pool is 100% utilized with a growing wait queue — saturation climbing before duration visibly degrades. Saturation is often the leading indicator; it goes red before users feel it. The two methods are complementary: RED tells you users are hurting, USE often tells you why, and often sooner.

Method	Watches	Best at
RED	request-driven services	”are callers being served well?” (symptoms)
USE	resources/hardware	”is the underlying machine choking?” (causes, early)

The cost: cardinality

Metrics are cheap — until you add labels. A label is a dimension you attach to a metric so you can slice it: http_requests_total{service="checkout", status="500"}. Labels are powerful: they let you break “total requests” down by service, region, status code. But each unique combination of label values creates a separate time series that must be stored and indexed. The number of combinations is the cardinality, and it multiplies:

   service (10) × status (15) × region (5)  = 750 series   ← fine
   ...add  user_id (1,000,000)              = 750,000,000 series   ← catastrophe

Putting a high-cardinality field — user_id, request_id, email, raw URL with IDs in it — into a metric label is the classic, budget-destroying mistake. It’s called a cardinality explosion, and it can make a metrics system slower and more expensive than the logs it was supposed to replace.

The rule: labels are for low-cardinality dimensions you’ll group by (status, region, endpoint template). Anything unbounded or per-request — the user, the exact ID, the full URL — belongs in a log or a trace, not a metric label. This is the precise mirror of logging’s cost: logs are expensive in volume, metrics are expensive in cardinality. Knowing which signal to reach for is what keeps both the system and the bill healthy.

Pull vs push, briefly

Two collection models exist. Pull (the collector scrapes each target’s /metrics endpoint on a schedule, e.g. Prometheus) makes target health trivially observable — a target that can’t be scraped is visibly down. Push (targets send metrics to a collector) suits short-lived jobs and serverless that may not be around to be scraped. Most large systems use a mix; the choice is mostly about how long your workloads live.

Under the hood — you can’t average a p99

Here’s a trap that bites teams the moment they run more than one instance: there is no way to combine per-server percentiles into a fleet-wide percentile. Averaging the p99 from each of ten servers is wrong; so is taking the maximum. A percentile simply isn’t additive — the only correct method is to merge the underlying histograms (sum the bucket counts across all instances) and read the percentile off the combined distribution. This is exactly why good metrics systems export histograms (bucket counts) rather than pre-computed percentiles: bucket counts add up cleanly across servers and time windows, percentiles do not. The catch is accuracy — a histogram only knows a value landed somewhere in a bucket, so a coarse bucket layout yields a coarse p99. Choose bucket boundaries around the latencies you actually care about, or the percentile you alert on is only as sharp as the buckets underneath it.

The architect’s lens

Metrics are the cheap pillar — but cheap only while you respect their one hard limit. The five questions:

• Why do they exist? Because aggregating billions of log lines on every dashboard refresh is slow and expensive. A metric pre-aggregates — store a running number per time window instead of a line per event — so a dashboard reads a handful of numbers, not a haystack.
• What problem do they solve? Cheap, instant, long-range answers to “how much / how many / how fast.” RED (rate, errors, duration) instruments every service for caller-facing symptoms; USE (utilization, saturation, errors) instruments every resource for causes, with saturation often a leading indicator that goes red before users feel it.
• What are the trade-offs? You buy cheapness by throwing detail away — a metric tells you the error rate spiked, never which user or why. And latency must be a histogram, not an average: 99 calls at 50 ms + 1 at 5,000 ms averages to a 100 ms no one experienced. Never alert on averages.
• When should I avoid it? Don’t put an unbounded, per-request field — user_id, request_id, email, a raw URL — into a label. Each unique combination is a separate time series, so high-cardinality labels cause a cardinality explosion that makes metrics pricier than the logs they replaced. Those fields belong in a log or trace.
• What breaks if I remove them? You lose the smoke alarm — the cheap, always-on signal that points you at a problem and feeds dashboards, RED/USE health, and burn-rate alerts. Logs and traces can reconstruct detail, but querying them for aggregate trends is exactly the slow, expensive thing metrics exist to avoid.

Check your understanding

1. Why is a metric cheaper than counting log lines, and what exactly do you give up for that cheapness?
2. Match each to its metric type: total errors, current queue depth, request latency distribution. Why must latency be a histogram and not a gauge?
3. Why is alerting on average latency dangerous? What should you alert on instead, and why?
4. Contrast RED and USE: what does each watch, and why is saturation often a leading indicator that RED misses?
5. What is a cardinality explosion, and which kinds of fields must never become metric labels? Where do those fields belong instead?

Show answers

1. A metric pre-aggregates — it stores a running number per time window instead of a line per event, so a dashboard reads a handful of numbers rather than scanning a haystack, giving cheap storage and instant queries over long ranges. What you give up is detail: a metric tells you the error rate spiked, never which user or why — that’s what logs and traces are for.
2. Total errors → counter (monotonically increasing; you read its rate of change). Current queue depth → gauge (a snapshot that goes up and down). Request latency distribution → histogram. Latency must be a histogram because that’s the only way to get percentiles — an average of 100 ms can hide a p99 of 3 seconds, and a gauge is just a single sampled value, not a distribution.
3. Averages lie: 99 requests at 50 ms and 1 at 5,000 ms average to ~100 ms — a number no single request experienced, hiding the user stuck waiting 5 seconds. Alert on percentiles (p50, p99) instead, because they describe what real users actually feel — especially the suffering ones in the tail.
4. RED watches the work — Rate, Errors, Duration of request-driven services (“are callers being served well?”, a symptom). USE watches the resource doing the work — Utilization, Saturation, Errors of CPU, memory, pools (“is the machine choking?”, a cause). Saturation (queued/waiting work) is often a leading indicator because a pool can be 100% utilized with a growing wait queue before duration visibly degrades — it goes red before users feel it.
5. A cardinality explosion happens because each unique combination of label values creates a separate stored time series, so adding a high-cardinality label multiplies series catastrophically (user_id with a million values → hundreds of millions of series). Never make unbounded / per-request fields labels — user_id, request_id, email, raw URLs with IDs. Those belong in a log or a trace; labels are only for low-cardinality dimensions you’ll group by (status, region, endpoint template).