Revision · Advanced & Rare Concepts

This part catalogued the non-obvious ways a correct-looking design betrays you — failures that pass code review and the load test, then surface months later under real concurrency, scale, or partial failure.

What this part covered

• Consistent hashing — a hash ring lets you add or remove nodes without reshuffling nearly every key, and virtual nodes are non-negotiable for even load; it buys smooth scaling at the cost of operational complexity.
• Vector clocks & CRDTs — wall-clock timestamps can’t tell you what happened, so vector clocks count causality and CRDTs merge themselves — buying automatic convergence at the cost of metadata and constrained data types.
• The dual-write problem & outbox — you cannot atomically write to a database and a queue, so the transactional outbox (drained by polling or CDC) makes it one write, trading a polling loop and latency for atomicity.
• Exactly-once semantics — true exactly-once delivery is physically impossible; “effectively-once” is at-least-once delivery plus idempotent processing, which is the guarantee you actually engineer for.
• Hot partitions & the celebrity problem — one viral key collapses perfectly balanced shards onto a single node; caching, replicating, or splitting the hot key spreads the heat at the cost of extra complexity and staleness.
• Tail latency & p99 — the mean is a comforting lie, and fan-out makes the slowest 1% the norm; hedged requests, tiered timeouts, and partial results fight the tail while spending extra load.
• Backpressure & flow control — a fast producer kills a slow consumer, so bounded queues, load shedding, and end-to-end propagation push back instead of unboundedly buffering or silently dropping.
• Probabilistic data structures — Bloom filters, count-min sketch, and HyperLogLog hash into a small fixed structure, trading a sliver of accuracy for enormous memory savings — being approximately right at scale.
• Cache stampede & thundering herd — when a hot entry expires, ten thousand requests stampede the database at once; single-flight locking, stale-while-revalidate, and jittered TTLs tame the surge.

The takeaway

Three axes run through all of it — concurrency, scale, and delivery under failure — and the same villains recur across them: the network that drops messages also makes clocks unreliable; the skew that creates a hot partition also spikes p99. Every technique here carries a price tag, and senior intuition is just knowing it in advance. Next, Part 12 takes this trade-off lens to the mid-2020s frontier.