Replication

A single database server is a single point of failure and a single point of throughput. The disk dies, the box reboots, the read traffic outgrows one CPU — and your whole system goes with it. Replication is the answer: keep complete copies of your data on more than one machine. It is the most common, most important reliability technique in data systems.

But copying introduces the deepest tension in distributed data. Replication buys you availability and read scale, and it charges you in consistency. The instant there are two copies, they can disagree — and reconciling that disagreement is the entire subject of this page.

What replication buys (and the bill it sends)

Three concrete wins, one recurring cost:

• Availability — if one node dies, another has the data, so the system survives.
• Read scalability — spread read queries across many copies; ten replicas can serve roughly ten times the reads.
• Latency — put a copy near your users, so reads don’t cross an ocean.

The bill: the copies must be kept in sync, and they can’t all be perfectly in sync at the same instant. Every replication design is a different answer to who accepts writes, and how fast do the others find out?

Three topologies

Single-leader (primary/replica)

One node is the leader (primary). All writes go to it. The leader streams its changes to one or more followers (replicas), which apply them in order and serve reads.

        writes
          │
          ▼
     ┌─────────┐   replication stream   ┌──────────┐
     │  LEADER  │ ─────────────────────► │ FOLLOWER │ ── reads
     └─────────┘ ─────────────┐         └──────────┘
                               └───────► ┌──────────┐
                                          │ FOLLOWER │ ── reads
                                          └──────────┘

This is the workhorse — Postgres, MySQL, MongoDB, and most relational setups default to it. Writes have one clear ordering (no conflicts), reads scale across followers, and it’s simple to reason about. The catch: the leader is a write bottleneck, and if it dies you must failover (promote a follower), a delicate dance covered alongside Read Replicas & CQRS.

Multi-leader

Several nodes accept writes — typically one leader per data center — and they replicate to each other. This shines for multi-region setups (writes stay local and fast) and offline-capable clients. But two leaders can accept conflicting writes to the same row at the same time, and now you have a conflict that must be resolved (last-write-wins, application merge, CRDTs). Conflict resolution is genuinely hard, which is why multi-leader is a specialist tool, not a default.

Leaderless

No designated leader. The client (or a coordinator) writes to several nodes at once and reads from several too, using quorums to paper over stragglers. Dynamo-style systems — Cassandra, Riak, Voldemort — work this way. (Despite the name, AWS’s DynamoDB is not leaderless: each partition has a leader elected via consensus, a deliberate departure from the original Dynamo design.)

   Write to W nodes,  Read from R nodes,  total N replicas.
   If  W + R > N , every read overlaps at least one node that saw the latest write.
   e.g. N=3, W=2, R=2  →  2 + 2 > 3  ✓  (tunable consistency)

Leaderless trades the simplicity of one ordering for no single point of write failure and tunable consistency knobs (W and R), at the cost of needing repair mechanisms (read-repair, anti-entropy) to heal divergence.

Synchronous vs asynchronous

Orthogonal to topology is when the leader considers a write “done”:

SYNC:   leader waits for follower to confirm  →  durable on 2 nodes, but SLOW,
                                                  and stalls if the follower lags
ASYNC:  leader replies immediately, ships later → FAST, but a leader crash can
                                                  LOSE the not-yet-shipped writes

• Synchronous guarantees the follower has the data before acknowledging the client — no data loss on leader failure — but the write is only as fast as the slowest follower, and a stuck follower can stall all writes.
• Asynchronous acknowledges immediately and ships changes in the background — fast and resilient to slow followers — but a crash between ack and ship loses those writes.

Many systems run semi-synchronous: one follower synchronous (durability), the rest async (speed) — buying most of the safety for most of the performance.

Replication lag and the anomalies it causes

Async replication means followers are always a little behind — milliseconds usually, seconds or more when overloaded. This replication lag is invisible until a user trips over it.

Read-after-write: “where did my comment go?”

A user posts a comment (write → leader) and immediately reloads (read → a lagging follower that hasn’t received it yet). Their own comment is gone. This is the classic read-after-write consistency violation. Fixes: route a user’s reads to the leader for a short window after they write; track each client’s write position and only read from followers that have caught up to it; or read your own writes from the leader.

Two cousins:

• Monotonic reads — read once from a fresh follower (see comment), refresh and hit a stale follower (comment vanishes again). Time appears to run backward. Fix: pin each user to one follower so they never see older state than they already saw.
• Consistent prefix — with partitioned data, you might see an answer before its question. Fix: ensure causally-related writes land in the same partition or are ordered.

These anomalies are not bugs in the database; they are the price of asynchronous copying, and they sit on the spectrum formalized in Consistency Models.

It actually happened — GitLab.com’s lost six hours (2017)

On 31 January 2017, a GitLab engineer fighting replication lag on a misbehaving secondary ran a deletion against what they believed was the broken replica — and hit the primary instead, wiping roughly 300 GB of live production data in seconds. The disaster wasn’t the typo; it was what came next. The team discovered that five separate backup and replication mechanisms were all broken or had never worked: the database dump was silently failing on a version mismatch, the disk snapshots weren’t enabled for that host, the cloud backups were empty. They were rescued only by a roughly six-hour-old copy a developer had taken by chance for unrelated work — so GitLab recovered, but permanently lost about six hours of issues, merge requests, and comments. That six hours was its real recovery point that day. The lesson sits at the center of this page: a replica is not a backup, and a backup you have never restored from is only a rumor. Test the restore, not just the dump.

The thread

Replication is how data systems refuse to die with a single machine and how they serve far more reads than one box ever could. The moment you make a second copy, though, you’ve signed up for managing a gap between copies — synchronous closes the gap at the cost of speed, asynchronous keeps speed at the cost of a lag window where users can see stale or vanishing data. Single-leader keeps ordering simple; multi-leader and leaderless trade that simplicity for write availability and geography. There is no configuration that gives you all the reads, all the availability, and perfect consistency at once — pick which two you’ll pay for.

The architect’s lens

Replication is rarely optional in a serious system, but its topology and sync mode are real choices — run them through the five questions:

• Why does it exist? Because a single database server is a single point of failure and a single point of throughput. Replication keeps complete copies of your data on more than one machine.
• What problem does it solve? Three wins at once: availability (a node dies, another has the data), read scalability (ten replicas serve roughly ten times the reads), and latency (a copy near your users avoids the cross-ocean trip).
• What are the trade-offs? The copies can’t all be in sync at the same instant. Synchronous closes the gap but is only as fast as the slowest follower; asynchronous is fast but a crash loses unshipped writes and lag produces anomalies (read-after-write, monotonic reads). Single-leader keeps ordering simple; multi-leader and leaderless trade that for write availability and geography at the cost of conflict resolution and repair.
• When should I avoid a given mode? Avoid multi-leader unless you truly need multi-region or offline writes — conflict resolution is genuinely hard. Avoid fully synchronous when a lagging follower must not be allowed to stall all writes. And never treat a replica as a backup (GitLab, 2017).
• What breaks if I remove it? One node’s death takes the whole system down, reads can’t scale past a single CPU, and every read hits the one origin — there’s no nearby copy and no survivor when the disk fails.

Check your understanding

1. List the three things replication buys you and the one recurring cost it always charges.
2. Contrast single-leader, multi-leader, and leaderless replication. What problem does each one’s structure introduce?
3. In a leaderless quorum with N=5, what W and R guarantee that a read sees the latest write? Why?
4. Explain synchronous vs asynchronous replication, and what “semi-synchronous” buys.
5. Describe the read-after-write anomaly and one concrete way to prevent it.

Show answers

1. Replication buys availability (another node has the data if one dies), read scalability (spread reads across many copies), and latency (put a copy near users). The one recurring cost: the copies must be kept in sync and can’t all be perfectly in sync at the same instant.
2. Single-leader: one node takes all writes and streams to followers — simple ordering, but the leader is a write bottleneck and failover is delicate. Multi-leader: several nodes accept writes and replicate to each other — great for multi-region, but introduces write conflicts that must be resolved. Leaderless: clients write/read several nodes with quorums — no single write SPOF, but needs repair mechanisms to heal divergence.
3. With N=5 you need W + R > N, so any W and R summing to at least 6 (e.g. W=3, R=3) guarantees a read sees the latest write. Because if every read overlaps at least one node that accepted the latest write, the freshest value is always among the nodes you read.
4. Synchronous waits for a follower to confirm before acknowledging — no data loss on leader failure, but only as fast as the slowest follower and a stuck follower stalls writes. Asynchronous acknowledges immediately and ships later — fast and resilient to slow followers, but a crash between ack and ship loses those writes. Semi-synchronous makes one follower synchronous (durability) and the rest async (speed), buying most of the safety for most of the performance.
5. Read-after-write: a user writes (to the leader), immediately reads from a lagging follower that hasn’t received it yet, and their own write appears to be gone. One concrete fix: route that user’s reads to the leader for a short window after they write (others can also track the client’s write position and only read from followers caught up to it).