Leader Election & Coordination

Many distributed problems get dramatically simpler if exactly one node is “in charge” — one writer, one scheduler, one owner of a shard. But “pick one node” is itself a hard distributed problem: the moment you have a leader, you have to handle the leader dying, the leader being wrongly declared dead, and two nodes both believing they’re the leader. This page is about getting that right.

Why elect a leader at all?

Coordination is expensive. If every node can write, you need conflict resolution, consensus on every operation, or careful commutative data types. Funnel decisions through a single leader and a whole class of races disappears:

Leaderless: every node decides → conflicts, coordination on every write
Leader:     one node decides   → simple ordering, but a single point to replace

The trade-off is stark — what does this buy us, and what does it cost? A leader buys simplicity and a natural ordering of operations; it costs you a failover problem and a potential bottleneck. Most systems decide that trade is worth it, then spend their complexity budget on making failover safe.

Electing the leader

You can’t just let nodes shout “I’m the leader” — partitions and crashes make that chaotic. Real systems delegate election to a consensus protocol (see Consensus: Raft & Paxos). The pattern almost everyone uses in practice:

• Run a small, strongly-consistent coordination service — ZooKeeper, etcd, or Consul — itself backed by Raft/ZAB across an odd number of nodes (3 or 5) so it can tolerate failures via majority quorum.
• A would-be leader takes a lease: a lock with a time-to-live that it must keep renewing (“heartbeating”). If it stops renewing — because it crashed or got partitioned away — the lease expires and another node can claim leadership.

   etcd / ZooKeeper (Raft quorum: 3 or 5 nodes)
        │   holds the lease key + TTL
        ▼
   Node A renews lease every 2s ──► A is leader
   (A crashes) ──► lease expires after TTL ──► Node B claims it ──► B is leader

The danger: split-brain

The hard case isn’t a clean crash — it’s a node that’s slow or partitioned but still alive. Its lease expires, the cluster elects a new leader, and then the old leader wakes up still believing it’s in charge. Now two leaders are issuing writes. That’s split-brain, and it corrupts data.

Why a lock alone isn’t enough

A classic failure: leader A acquires a lock, then pauses (a long GC pause, a VM stall, a network blip). The lock times out; B acquires it and starts writing. Then A resumes — unaware it ever lost the lock — and also writes. Both think they hold the lock. The lock did its job and it still wasn’t enough, because A’s clock and B’s clock disagree about who’s current.

The fix: fencing tokens

The robust solution is to make every lock grant a monotonically increasing number — a fencing token — and have the protected resource itself reject anything stale.

A acquires lock → token 33 → (A stalls) ...
B acquires lock → token 34 → B writes with token 34  ✓ storage records "highest seen: 34"
A wakes, writes with token 33 → storage sees 33 < 34 → REJECTED

The storage layer only accepts writes whose token is ≥ the highest it has seen. Now a delayed ex-leader cannot clobber the new one, even if its software still thinks it’s the boss. The coordination service hands out ever-increasing tokens (e.g. ZooKeeper’s zxid/version), and the data store enforces them. This is the difference between a lock that advises and one that guarantees.

What coordination services give you

Beyond leader election, a coordination service is the Swiss-army tool for “we need the cluster to agree on a small, critical fact”:

Use	What it provides
Leader election	Whoever holds the lease key is leader; auto-expiry on failure
Service discovery	A consistent registry of “who is alive and where”
Distributed locks	Mutual exclusion across nodes (with fencing tokens)
Configuration	A single source of truth nodes can watch for changes
Membership	Detecting nodes joining/leaving via ephemeral keys + heartbeats

Keep the coordination service small and off the hot path

These systems are strongly consistent, which means they’re not built for high throughput — every write goes through consensus. Use them for rare, critical decisions (who’s leader, what’s the config), not for per-request traffic. Putting a distributed lock in your hot path is a common way to turn a fast system slow.

It actually happened — Elasticsearch’s split-brain foot-gun (pre-7.0)

For years, Elasticsearch left split-brain prevention as a manual setting: minimum_master_nodes, which operators were supposed to set to a quorum (nodes / 2 + 1). Leave it too low — easy to do, and the out-of-the-box behavior was unsafe for multi-node clusters — and a network partition could let both sides elect their own master. Each half then accepted writes independently, and on healing one half’s updates had to be discarded: classic split-brain data loss. Version 7.0 (2019) finally removed the foot-gun, replacing the manual knob with a built-in, quorum-based coordination layer that computes the majority itself and won’t let a minority elect a master. The lesson is this page’s thesis in miniature: correctness of “who’s in charge” must come from a quorum the system enforces, never from a number a human remembered to configure.

The thread

What does this buy us, and what does it cost? Leader election buys a single point of decision that makes ordering and ownership trivial; it costs a failover protocol, and — done naïvely — risks split-brain. The deeper lesson is that time and liveness can’t be trusted in a distributed system: you can never be sure a node is dead, only that you haven’t heard from it. Fencing tokens accept that uncertainty and make correctness depend on monotonic order rather than on guessing who’s alive.

The architect’s lens

Electing a leader trades one class of hard problems for another — run it through the five questions before you build on it:

• Why does it exist? Because many distributed problems get dramatically simpler with exactly one node in charge (one writer, one scheduler, one shard owner) — but “pick one node” is itself a hard problem, so election is delegated to a consensus-backed coordination service.
• What problem does it solve? It gives a single point of decision that makes ordering and ownership trivial, with a lease/TTL that auto-expires on failure so another node can take over without a human.
• What are the trade-offs? You buy simplicity and natural ordering at the cost of a failover protocol and a potential bottleneck. The deep hazard is split-brain — a slow or partitioned-but-alive old leader still writing after a new one is elected — because you can never be sure a node is dead, only that you haven’t heard from it.
• When should I avoid it? Keep the coordination service off the hot path: it’s strongly consistent (every write goes through consensus), built for rare critical decisions, not per-request traffic — a distributed lock in the request path is a classic way to turn a fast system slow.
• What breaks if I rely on a plain lock? A lock alone advises but doesn’t guarantee — a stalled ex-leader clobbers the new one. You need fencing tokens (monotonic numbers the storage layer itself enforces, rejecting stale writes) and quorum-enforced election, so correctness of “who’s in charge” comes from the system, never from a number a human remembered to configure (Elasticsearch’s pre-7.0 foot-gun).

Check your understanding

1. What class of problems does electing a single leader simplify, and what new problem does it create?
2. Why do real systems lean on a service like etcd/ZooKeeper instead of nodes electing among themselves directly?
3. Describe split-brain and the exact sequence (including a GC pause) that produces it.
4. How does a fencing token prevent a stalled ex-leader from corrupting data, and which component must enforce it?
5. Why should a coordination service be kept off the request hot path?

Show answers

1. Electing one leader simplifies the whole class of races that arise when every node can write — conflict resolution, consensus on every operation, commutative data types — by funneling decisions through a single decider that gives natural ordering and ownership. The new problem it creates is failover: you must now handle the leader dying, being wrongly declared dead, and a potential throughput bottleneck.
2. Nodes electing among themselves directly is chaotic under partitions and crashes — you can’t just let nodes shout “I’m the leader.” Real systems delegate to a small, strongly-consistent coordination service like etcd/ZooKeeper, itself backed by Raft/ZAB consensus across an odd number of nodes, which holds the lease and resolves the election correctly via majority quorum.
3. Split-brain is when two nodes both believe they’re leader and both issue writes, corrupting data. The sequence: leader A acquires the lock/lease, then pauses (a long GC pause, VM stall, or network blip); the lease times out and B acquires it and starts writing; then A resumes unaware it ever lost the lock and also writes — two leaders, because A’s and B’s clocks disagree about who’s current.
4. Each lock grant carries a monotonically increasing fencing token (e.g. ZooKeeper’s zxid). The protected resource itself must enforce it, accepting only writes whose token is ≥ the highest it has seen — so when stalled ex-leader A wakes and writes with token 33 after B wrote with 34, storage rejects 33 as stale. The data store, not the lock, is what guarantees correctness.
5. A coordination service is strongly consistent, so every write goes through consensus — it’s built for rare, critical decisions (who’s leader, what’s the config), not high throughput. Putting a distributed lock in the per-request hot path is a common way to turn a fast system slow.