Distributed Locking

The Core Idea

A distributed lock is a mechanism that ensures only one process across a distributed system can access a shared resource at a time. Unlike local locks (mutex), distributed locks must work across multiple servers that communicate over a network.

Step-by-Step Walkthrough

Using Redis: SET lock_key unique_value NX PX 30000 (set only if not exists, expire in 30 seconds). If SET succeeds, the client holds the lock. When done, the client deletes the key (only if the value matches — prevents releasing someone else's lock). If the client crashes, the lock expires after 30 seconds.

Using ZooKeeper: Create an ephemeral sequential znode. The client with the lowest sequence number holds the lock. When the client disconnects (crashes), the ephemeral node is automatically deleted.

Why This Approach Wins

• Mutual exclusion: Only one client can hold the lock at a time.
• Deadlock freedom: If a client crashes while holding a lock, the lock must eventually be released (TTL).
• Fault tolerance: The lock system must survive node failures (use consensus-based stores like etcd, ZooKeeper).
• Redlock (Redis): Acquires the lock on N/2+1 Redis instances. Controversial — see Martin Kleppmann's critique.
• Fencing tokens: A monotonically increasing token issued with each lock. Storage systems reject writes with older tokens, preventing split-brain scenarios.

In Production

Stripe uses distributed locks to prevent double-charging customers during payment processing.

Google uses Chubby (distributed lock service) for leader election, metadata storage, and configuration management.

Kubernetes uses etcd leases for leader election among controller manager replicas.

Tradeoffs and Limitations

• Correctness vs Performance: Consensus-based locks (etcd) are correct but slower. Redis locks are faster but can have correctness issues.
• TTL: Too short = lock expires while client is still working. Too long = long wait if client crashes.
• Availability: The lock service must be highly available; if it goes down, all locked operations block.

Production Gotchas

1. Using a single Redis instance for distributed locks — no fault tolerance
2. Not using fencing tokens — stale lock holders can corrupt data
3. Setting lock TTL too short — long GC pauses can cause the lock to expire while the client is still processing

The Interview Angle

1. How do you implement a distributed lock?
2. What is the Redlock algorithm and why is it controversial?
3. What are fencing tokens and why are they needed?
4. How does ZooKeeper implement distributed locking?

Next Up

• consensus-algorithms
• distributed-lock-service
• idempotency
• chubby

Original DosiBridge notes for Distributed Locking

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Distributed Locking is to connect the concept to a production service where growth exposes the first weak assumption in the original design. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

• Membership: For Distributed Locking, decide the membership before choosing tools; otherwise the design looks clean but fails when distributed changes.
• Clock Drift: For Distributed Locking, decide the clock drift before choosing tools; otherwise the design looks clean but fails when locking changes.
• Retry Semantics: For Distributed Locking, decide the retry semantics before choosing tools; otherwise the design looks clean but fails when distributed changes.
• Partition Behavior: For Distributed Locking, decide the partition behavior before choosing tools; otherwise the design looks clean but fails when locking changes.
• Reconciliation: For Distributed Locking, decide the reconciliation before choosing tools; otherwise the design looks clean but fails when distributed changes.

Production review question

If a team adopts Distributed Locking, what observable signal proves it helped instead of only making the architecture diagram more fashionable?

Practical metric to track

Track distributed-locking health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Distributed Locking, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.