Leader Election

Leader election picks one node out of several to perform a job that must happen exactly once at a time: running a scheduler, being the primary database, owning a partition, or coordinating a cluster. The hard part isn't choosing a leader; it's making sure there is never more than one acting as leader, even when networks partition and processes pause.

Why you need a single leader

• Singleton jobs: a cron scheduler that must fire each job once, a billing run, a compaction coordinator.
• Write ordering: a single primary database or partition leader orders writes without conflicts. See replication.
• Coordination: one node assigns shards or tasks to workers.
• Efficiency: avoid every node doing the same expensive work.

Leases: the standard approach

A leader holds a lease: a time-limited claim recorded in a strongly consistent store (etcd, ZooKeeper, Consul, or a database row). It must renew the lease before it expires; if it dies or is partitioned away, the lease lapses and another node acquires it.

• ZooKeeper: each candidate creates an ephemeral sequential node; the lowest number is leader; others watch the node just before theirs, avoiding a herd of notifications.
• etcd: an election API built on leases and revisions; Kubernetes controllers use Lease objects for the same purpose.
• Database row: UPDATE leader SET holder = 'B', expires = now() + 10s WHERE expires < now(): simple, but only as reliable as the database.
• Consensus protocols like Raft include election internally. See consensus.

Split brain and fencing

Leases expire by time, but a leader can be paused (a long garbage-collection pause, a VM migration, an overloaded host) longer than its lease. It wakes up believing it's still leader, while a new leader has already been elected. Now two nodes act as leader: split brain. Both may write to the database or both may fire the scheduled job.

The fix is a fencing token: every time the lease is granted, the store hands out a monotonically increasing number (the term or revision). The leader includes it with every action, and the resources it touches reject any token older than one they've already seen.

Without fencing, a lease is only a hint. With it, safety no longer depends on clocks or pause times. See distributed locks.

Tuning failover

• Lease length trades failover speed against false failovers: 10–15 seconds is common; very short leases cause flapping during network blips.
• Renew well before expiry (every third of the lease) and step down voluntarily if renewal fails.
• Graceful handover: on shutdown, release the lease so a standby takes over immediately rather than after a timeout.
• Avoid a single leader where you can: partitioning work so each node leads its own shards (as Kafka does per partition) spreads load and limits the blast radius.