Caching

Caching is the highest-leverage performance technique in all of systems, and it rests on a single observation: the same answers are requested over and over. If computing or fetching an answer is expensive, store it the first time and hand out the copy for free thereafter. A cache is just a fast store of answers you’ve already produced, placed close to where they’re needed.

The leverage is enormous — a read that took 50 ms against a database can take well under a millisecond from memory — but caching carries one unavoidable cost that shapes every decision around it: a cache holds a copy, and copies go stale. The entire discipline of caching is managing the gap between the copy and the truth.

What does a cache buy us, and what does it cost? It buys latency and throughput (and relieves load on the slow thing behind it). It costs you staleness, memory, and a new source of bugs: serving data that is no longer correct.

Why it works: locality

Caches pay off because real access patterns are skewed, not uniform:

• Temporal locality — something accessed now is likely to be accessed again soon (a trending post, a logged-in user’s profile).
• Spatial / popularity locality — a small fraction of items get the vast majority of requests (the classic 80/20, often far steeper). Cache the hot few and you serve most traffic from cache.

If every item were equally and rarely accessed, a cache would help little. It’s the skew that makes caching a superpower.

The one metric: hit ratio

The number that decides whether a cache is worth anything is the cache hit ratio — the fraction of requests served from the cache rather than the slow backend:

   hit ratio = hits / (hits + misses)

   effective latency ≈ hit_ratio × cache_latency
                     + (1 − hit_ratio) × backend_latency

A small ratio change matters more than it looks. Going from 90% to 99% hits cuts the miss rate by 10× — and misses are where all the slow, expensive work happens. The miss path is also where danger lives: when many misses hit the backend at once (a stampede), the cache that was protecting your database becomes the funnel that overloads it. That failure mode has its own page — Cache Stampede & Thundering Herd.

Eviction: a cache is finite

A cache is small on purpose (fast memory is expensive), so it must constantly decide what to throw out to make room. The eviction policy is that decision:

Policy	Evicts	Good when
LRU (Least Recently Used)	the item untouched longest	recent use predicts future use (usually)
LFU (Least Frequently Used)	the item with fewest hits	popularity is stable over time
TTL (Time To Live)	anything older than a set age	data has a natural freshness window
FIFO	the oldest inserted	simple; rarely optimal

LRU is the workhorse — it captures temporal locality cheaply. LFU resists a burst of one-off requests evicting your genuinely popular items, but adapts poorly when popularity shifts. TTL is special: it’s not really about space, it’s about correctness — it’s how you cap staleness by saying “this copy may be wrong after N seconds, so stop trusting it.” Most real caches combine TTL (for freshness) with LRU/LFU (for space).

The cache hierarchy

Caching isn’t one box — it’s a series of them along the request path, each closer to the user and each catching what the one behind it would have served:

   Browser cache        ← on the user's device; zero network. Static assets, API responses.
        │ miss
   CDN edge cache       ← near the user (see CDNs page). Static + cacheable dynamic content.
        │ miss
   Reverse-proxy cache  ← at your front door (e.g. Varnish/nginx). Full-page or fragment cache.
        │ miss
   Application cache     ← in-memory near app servers (Redis/Memcached). Query results, sessions, objects.
        │ miss
   Database cache        ← the DB's own buffer pool / query cache.
        │ miss
   Disk / origin compute ← the slow source of truth. The thing every layer above is protecting.

The deeper a request falls, the more expensive it gets. Each layer’s job is to make sure as few requests as possible fall to the next one. A cache miss at the browser might still hit at the CDN; a CDN miss might still hit the app cache. The hierarchy multiplies your hit ratio.

Cache the right layer for the job

Per-user data (a logged-in dashboard) belongs in the application cache, keyed by user — it can’t be shared at the CDN. Shared, public data (a product image, a published article) belongs as far out as possible — ideally the CDN or browser — so it never touches your servers at all. Matching data to the right layer is most of the design.

Staleness: the cost you can never fully escape

The moment you store a copy, you accept that the truth might move and your copy won’t know. Every caching strategy is a different answer to how do we deal with that? — expire it after a TTL, update the cache when the source changes, or accept eventual convergence. These strategies (write-through, write-back, write-around, cache-aside) and their consistency implications are the subject of Caching Strategies, which builds directly on this page. Here, hold onto the principle:

There is no caching without staleness. Caching is the art of choosing how much staleness you can tolerate, for which data, and how you’ll bound it.

Some data tolerates seconds of staleness without anyone noticing (a view count). Some tolerates none (an account balance, a permission check) and either shouldn’t be cached or must be invalidated precisely. Knowing which is which — per field, not per system — is the skill.

The two hardest problems

There’s an old joke that the two hard problems in computing are cache invalidation, naming things, and off-by-one errors. The joke is funny because invalidation really is hard: knowing exactly when a cached copy became wrong, and reliably purging it everywhere it lives, across a whole hierarchy, is a genuinely deep problem — and the source of a large share of “impossible” production bugs.

By the numbers — the last few points of hit ratio do the heavy lifting

Plug real numbers into the page’s own formula — a 1 ms cache in front of a 50 ms backend — and watch what the hit ratio buys:

   95% hits:  0.95 × 1 ms + 0.05 × 50 ms = 0.95 + 2.50 = 3.45 ms
   99% hits:  0.99 × 1 ms + 0.01 × 50 ms = 0.99 + 0.50 = 1.49 ms

Pushing from 95% to 99% — a four-point gain that sounds marginal — more than halves effective latency, because every point you add is a point removed from the brutally expensive miss path. That’s the economic case for caching in one calculation: the value is concentrated at the very top of the ratio, which is also the hardest part to win. What does the extra effort buy? Disproportionately more than the number suggests.

The architect’s lens

Before you reach for a cache (or a cache layer like Redis), answer the five questions that turn an implementer into an architect — the why behind the how:

• Why does it exist? Because recomputing or re-fetching the same answer is wasteful, and the slow source of truth (a database, an upstream API, a render) is the scarcest, most expensive resource in the system. A cache exists to keep already-produced answers close to where they’re needed.
• What problem does it solve? Latency and load. It collapses a 50 ms backend read to a sub-1 ms memory read and shields that slow backend from traffic it could never survive — the same win seen from the user’s side and from the database’s side.
• What are the trade-offs? You trade freshness for speed. The instant you hold a copy you’ve accepted staleness, extra memory cost, and a whole new class of bugs (serving data that’s quietly wrong). You also take on invalidation — famously one of the hardest problems in computing.
• When should I avoid it? When the data can’t tolerate any staleness and you can’t invalidate precisely (an account balance, a permission check); when access is uniform so the hit ratio would be near zero; or when the backend is already fast and a cache only adds a second thing that can be wrong. Don’t reach for Redis when a correct, indexed Postgres query is fast enough.
• What breaks if I remove it? Every request falls through to the slow source of truth. Tail latency balloons, the database’s load multiplies by your former hit ratio, and a system tuned to run at 99% hits can tip straight into overload — which is exactly why a cache outage (Redis going down) is so dangerous: the protection vanishes all at once.

Check your understanding

1. Why does caching depend on locality? What would a perfectly uniform access pattern do to a cache’s value?
2. Why does improving hit ratio from 90% to 99% matter so much more than from 50% to 59%?
3. Contrast LRU and LFU. Why is TTL eviction fundamentally about correctness rather than space?
4. Walk a request down the cache hierarchy. Why does per-user data belong in the app cache while a public image belongs at the CDN?
5. Restate the core cost of caching in one sentence. Give one piece of data you’d never cache and say why.

Show answers

1. Caching pays off because real access is skewed, not uniform — temporal locality (something accessed now is likely accessed again soon) and popularity locality (a small fraction of items get most requests). A perfectly uniform access pattern would mean the cache almost never sees the same item twice, so its hit ratio — and its value — collapses to nearly nothing.
2. Because misses are where all the slow, expensive work happens, and hit ratio improvements cut the miss rate multiplicatively: 90%→99% takes misses from 10% to 1%, a 10× drop, whereas 50%→59% only trims misses from 50% to 41%, well under 2×. The high end is where the leverage is.
3. LRU evicts the item untouched longest (captures temporal locality cheaply); LFU evicts the least-frequently-hit item (resists one-off bursts but adapts poorly when popularity shifts). TTL is about correctness, not space — it caps staleness by declaring a copy untrustworthy after N seconds, regardless of how much room is left.
4. Browser → CDN edge → reverse-proxy → application cache → database cache → origin, each layer catching what the next would have served. Per-user data (a logged-in dashboard) can’t be shared across users, so it belongs in the application cache keyed by user; a public image is identical for everyone, so it belongs as far out as possible (the CDN) where it never touches your servers.
5. There is no caching without staleness — caching is choosing how much staleness you can tolerate, for which data, and how to bound it. You’d never cache (without precise invalidation) an account balance or a permission check, because serving a stale copy means showing wrong money or granting wrong access.