Logging

A log is the oldest debugging tool that still works: a timestamped record of something that happened. “User 42 logged in.” “Payment failed: card declined.” “Cache miss for key cart:991.” Logs are the highest-fidelity pillar of observability — each line can carry as much context as you care to write — and for exactly that reason they are the most expensive to store and the easiest to turn into noise. This page builds logging up from the single print statement to a system that can reconstruct one request’s entire journey across a dozen machines.

The problem with a print

A lone developer debugging a script writes print("got here"). It works because there is one process, one machine, one human reading one terminal. Production breaks every one of those assumptions: hundreds of processes, across many machines, with no human watching any single terminal. The print statement that saved you locally is invisible at scale. Logging is the engineering of that print statement into something you can find, filter, and read later, across the fleet, when you don’t yet know what you’re looking for.

Structured logging: the single biggest upgrade

The instinct is to log human prose:

2026-06-22 14:03:11  Payment failed for user 42 on order 991 after 1200ms

A human reads that fine. A machine — the thing that actually has to search billions of these — cannot. To find “all failed payments over 1000ms for user 42,” it must parse free text with fragile regexes. Structured logging emits machine-readable key–value records instead, usually JSON:

{
  "ts": "2026-06-22T14:03:11Z",
  "level": "error",
  "msg": "payment failed",
  "user_id": 42,
  "order_id": 991,
  "duration_ms": 1200,
  "reason": "card_declined",
  "trace_id": "a1b2c3..."
}

Now “failed payments over 1000ms for user 42” is a precise query over fields, not a guess over prose. What does this buy us? Queryable, aggregatable, filterable logs. What does it cost? A little discipline at every log site and slightly larger lines on disk — a trade almost always worth making.

Log levels: a volume dial

Not every event deserves the same attention. Levels let you emit detail liberally but read selectively:

Level	Meaning	Typical use
ERROR	something failed and a user/operation was affected	exceptions, failed payments
WARN	something unexpected, but handled	retry succeeded, fell back to cache
INFO	normal, noteworthy events	request served, job completed
DEBUG	fine-grained detail for diagnosis	variable values, branch taken
TRACE	firehose	every loop iteration

In production you typically run at INFO, keeping the firehose closed. When you’re chasing a bug, you turn the dial up to DEBUG — ideally without redeploying, via dynamic config. The level is your primary control over the eternal tension between enough detail to debug and few enough lines to afford.

Never log secrets or PII carelessly

Logs are copied, shipped to third-party systems, and retained for months. A password, token, or full credit-card number in a log line is a breach waiting to be discovered. Redact at the log site. This is where logging meets secrets management and encryption.

Centralization: logs you can’t read are worthless

A log file on a server that just crashed — or that autoscaling just terminated — is gone. And even healthy, logs scattered across 200 machines are unsearchable by hand. So production logging ships every line off the box, immediately, to a central store:

   app server ─┐
   app server ─┤   ship       ┌────────────────┐   index &
   app server ─┼──────────►   │  log pipeline  │  ────────►  searchable store
   app server ─┤  (agent /    │ (buffer/parse) │             (query, dashboards,
   worker     ─┘   sidecar)   └────────────────┘             retention, alerts)

A small agent on each host (or a sidecar) tails the logs, buffers them, and forwards them to a pipeline that parses, enriches, and indexes them into a searchable backend. The buffer matters: when the central store hiccups, you want logs queued locally, not dropped. This is the same producer/consumer decoupling you saw with message queues — the agent is a producer, the store is a consumer, and the buffer absorbs the mismatch.

Correlation IDs: stitching one request together

Here is the problem centralization creates: now every machine’s logs are interleaved in one giant stream. A single user request touched the gateway, three services, and a database — and their log lines are scattered among millions of others. How do you pull back just this request’s story?

The answer is a correlation ID (a.k.a. request ID or trace ID): a unique token generated at the edge — typically at the reverse proxy or API gateway — and propagated through every downstream call, usually as an HTTP header. Every service stamps it onto every log line it emits for that request.

   edge generates  trace_id = a1b2c3
        │  header: X-Request-Id: a1b2c3
        ▼
   gateway  ──► service A ──► service B ──► database
     (logs)      (logs)       (logs)       (logs)
       all carrying trace_id = a1b2c3

Now one query — trace_id = "a1b2c3" — reassembles the entire request from logs written on five different machines, in order. This is the logging counterpart to a full distributed trace; the correlation ID is the thread both pillars pull on. Without it, centralized logs are a haystack; with it, every request is a labeled needle.

The cost: volume and noise

Logging’s defining trade-off is brutal at scale. A busy service can emit more bytes of logs than it processes in actual data. The costs compound:

• Storage and indexing. Indexed, searchable log storage is far pricier than raw object storage. Retention is a direct dial on the bill — 7 days vs 90 days can be a 10x difference.
• Noise. When everything is logged at INFO, the one line that mattered is buried under a million that didn’t. Excessive logging actively reduces observability.
• Performance. Synchronous logging on the hot path adds latency; high-volume logging can saturate disk or network I/O.

The mitigations are sampling (keep all errors, but only 1% of routine success lines), tiered retention (hot/searchable for days, cold/archived for months), and ruthless level discipline. The goal is never “log everything” — it’s “log what you’ll actually query,” because a log you never read is pure cost.

Logs vs metrics: pick the right tool

If you find yourself logging a line just to count how often something happens, you want a metric, not a log. Counting is what metrics are cheap at; logs are for the detail of individual events. Using logs as a counter is the single most common way teams overspend on observability.

By the numbers — a service can out-log its own data

Take a service handling 10,000 requests/second that writes a modest five log lines per request at ~1 KB each:

10,000 req/s × 5 lines × 1 KB  =  50 MB/s
50 MB/s × 86,400 s/day         ≈  4.3 TB/day  ≈  130 TB/month

Indexed, searchable log storage is the expensive kind — often far pricier per byte than the object storage holding the business data itself — so a single chatty service can run up a telemetry bill that dwarfs the cost of the compute it’s observing. That’s the whole reason this page preaches sampling (keep every error, ~1% of routine successes), tiered retention, and level discipline: at this volume, “log everything” isn’t thorough, it’s bankrupt.

It actually happened — passwords in the logs (2018)

In May 2018, Twitter disclosed that a bug had written user passwords to an internal log in plaintext — captured on the way in, before the hashing step that was supposed to protect them. There was no evidence anyone had misused them, but the company asked all of its roughly 330 million users to change their passwords anyway. In the same month, GitHub disclosed a near-identical bug. Neither was a break-in; both were ordinary logging quietly capturing a secret that should never touch disk in readable form. It is the danger box above made real: logs are copied, shipped to third parties, and retained for months, so a credential logged once is a credential exposed for as long as the log lives. Redact at the log site, before the value is ever written.

The architect’s lens

Logging is the highest-fidelity pillar and the easiest to overspend on. Run it through the five questions:

• Why does it exist? Because the lone-developer print("got here") is invisible at scale — production has hundreds of processes across many machines with no human watching any terminal. Logging is the engineering of that print into something you can find, filter, and read later, across the fleet.
• What problem does it solve? High-fidelity, per-event detail you can query. Structured (JSON) logging turns “failed payments over 1000 ms for user 42” from a fragile regex into a precise field query; correlation IDs stitch one request’s story back together from logs on five machines; centralization survives a crashed or autoscaled-away host.
• What are the trade-offs? Fidelity costs money and noise. A chatty service can out-log its own data — 10k req/s × 5 lines × 1 KB ≈ 130 TB/month of the expensive indexed kind — and at INFO the one line that mattered drowns under a million that didn’t, so over-logging actively reduces observability.
• When should I avoid it? Don’t log a line just to count something — that’s a cheap metric, and using logs as a counter is the most common way teams overspend. Never log secrets or PII (the Twitter/GitHub 2018 plaintext-password bugs); redact at the log site.
• What breaks if I remove it? You lose the only signal that explains what exactly happened in a single failing event — metrics show the rate spiked and traces show where, but logs carry the reason (reason: card_declined). Drop the correlation ID and centralized logs become an unsearchable haystack.

Check your understanding

1. Why does free-text logging break down at scale, and what specifically does structured (key–value) logging make possible?
2. What is a log level for, and how does it let you reconcile “enough detail to debug” with “few enough lines to afford”?
3. Why must logs be shipped off the host immediately, and what role does the local buffer play?
4. Explain how a correlation ID lets you reconstruct one request’s story from logs written on many machines. Where is it generated, and how does it travel?
5. Give two concrete reasons over-logging reduces observability, and name two mitigations.

Show answers

1. A machine — the thing that has to search billions of lines — can’t query free-text prose without fragile regexes. Structured (key–value, usually JSON) logging makes logs queryable, aggregatable, and filterable: “failed payments over 1000 ms for user 42” becomes a precise query over fields instead of a guess over text.
2. A level lets you emit detail liberally but read selectively. You run production at INFO (firehose closed) and turn the dial up to DEBUG when chasing a bug — ideally without redeploying, via dynamic config. The level is your primary control over the tension between enough detail to debug and few enough lines to afford.
3. A log file on a server that just crashed or was autoscaled away is gone, and logs scattered across 200 machines are unsearchable by hand — so every line is shipped off the box immediately to a central store. The local buffer absorbs the producer/consumer mismatch: when the central store hiccups, logs queue locally instead of being dropped.
4. A correlation ID (request/trace ID) is a unique token generated at the edge (the reverse proxy or API gateway) and propagated through every downstream call, usually as an HTTP header; every service stamps it onto every log line it emits for that request. Then one query — trace_id = "a1b2c3" — reassembles the entire request, in order, from logs written on five different machines.
5. Over-logging reduces observability because (a) noise — the one line that mattered is buried under a million that didn’t, and (b) cost/performance — storage and indexing balloon, and synchronous logging on the hot path adds latency or saturates I/O. Two mitigations: sampling (keep all errors, only ~1% of routine successes) and tiered retention (hot/searchable for days, cold/archived for months) — plus ruthless level discipline.