Distributed Locks and Fencing Tokens

A distributed lock ensures that only one process, across many machines, works on a shared resource at a time: processing an order, renewing a certificate, running a migration. Unlike a lock inside one process, it must survive crashes, network partitions and pauses, which makes "exactly one holder" much harder than it sounds.

Two reasons to lock

• Efficiency: avoid doing the same work twice (two workers generating the same report). If the lock occasionally fails, you waste some work. A simple lock is fine.
• Correctness: two holders would corrupt data or break an invariant (double-spending, two writers to one file). Here the lock must be safe even when processes pause, which requires fencing.

Lease-based locks

Distributed locks are leases: they expire automatically, so a crashed holder can't block everyone forever.

# Acquire: set only if absent, with a 30-second expiry and a unique token
SET lock:invoice:981  "worker-7:uuid-4f2c"  NX  PX 30000

# Release: delete only if we still own it (atomic check-and-delete in a Lua script)
if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("DEL", KEYS[1])
end
return 0

• Unique owner value: so a process never deletes a lock someone else now holds (after its own expired).
• Expiry longer than the work, with renewal (a "watchdog" extending the lease) for long tasks.
• Retry with backoff and jitter when the lock is taken, rather than spinning.

Why expiry alone isn't safe

Any lease-based lock has the same flaw: the holder can pause after acquiring the lock (a garbage-collection pause, a CPU-starved container, a slow disk) for longer than the lease. The lock expires, another process acquires it, and when the first wakes up, two processes believe they hold the lock. Checking "do I still hold it?" right before writing doesn't help, because the pause can happen between the check and the write.

A fencing token is a number that increases every time the lock is granted. The protected resource stores the highest token it has accepted and rejects lower ones. This requires the resource to participate (a database WHERE version < ? check, a storage API that accepts tokens). ZooKeeper's zxid and etcd's revision numbers serve as fencing tokens.

Choosing a lock service

Often you don't need a lock

• Database constraints and conditional writes: a unique constraint or UPDATE … WHERE status = 'pending' is atomic and safe without any lock. See isolation levels.
• Optimistic concurrency: version numbers detect conflicts at write time.
• Partition ownership: assign each entity to one worker (Kafka partitions keyed by ID), so no two workers ever touch the same entity.
• Idempotency: if doing the work twice is harmless, a lock only saves effort. See idempotency.