Distributed Locks in Microservices: Redlock vs etcd vs Postgres

What a distributed lock is for: efficiency versus correctness

A distributed lock gives one process, out of many running across separate machines, exclusive permission to do something for a while. In a microservices system, those processes are often replicas of the same service behind a load balancer. They share no memory, so a mutex in one replica means nothing to another. The lock has to live in a shared system that every replica can reach: a Redis instance, an etcd cluster, or a database.

Teams take a lock for one of two reasons, and they should decide which one applies before choosing a tool.

Efficiency locks

The lock prevents wasted work. Examples include a nightly report job, a cache rebuild, or a batch of reminder emails that only one replica should send. If the lock fails and two replicas run the job, the cost is duplicated compute, an extra email, or a cache warmed twice. That is annoying but recoverable. A lightweight lock that works almost all the time is often good enough here.

Correctness locks

The lock protects shared state. Examples include debiting an account balance, allocating the last unit of inventory, or writing to a file that two writers would interleave. If two holders run at once, data is corrupted, and nothing in the system detects it afterward.

This distinction drives the rest of the comparison. Efficiency locks can use whichever system is already in your stack. Correctness locks need stronger guarantees: the lock itself must be reliable, and the protected resource must be able to reject a holder whose lock has expired.

Why two clients can hold the same lock, and how fencing tokens fix it

A lease-based lock is a promise with an expiry date, and the holder cannot observe that date reliably. A client acquires a lock with a 10-second TTL, then hits a stop-the-world GC pause, a hypervisor steal, or a congested network path. The process is frozen, so it cannot renew. The lease expires, the lock service grants it to someone else, and when the first client resumes, its local state still says "I own this." Checking the clock right before writing does not help, because the pause can fall between that check and the write.

No TTL value closes this gap. Pauses have no reliable upper bound, so any TTL can be exceeded. The choice is only about failure rates. Short TTLs recover quickly after a crash but expire more often during ordinary pauses. Long TTLs survive pauses but leave the resource blocked for longer when a holder actually dies.

Fencing tokens

The fix moves the check to the resource. Every time the lock service grants the lock, it issues a number that only goes up. The client attaches that token to every write. The resource stores the highest token it has accepted and rejects any write that carries a lower one.

Here is the same failure with fencing:

1. Client A acquires the lock with token 33, then pauses.
2. A's lease expires. Client B acquires the lock with token 34 and writes. The resource records 34.
3. A resumes and writes with token 33. The resource rejects it as stale.

The limit is that the resource has to cooperate. Fencing protects a database you control, where you can add a token column and a conditional update. It does nothing for a payment provider, an email API, or a legacy system that accepts any well-formed request.

Copy +----------+       +--------------+       +----------+     +-------------+
 | Client A |       | Lock service |       | Client B |     |  Resource   |
 +----------+       +--------------+       +----------+     | (max token) |
      |                    |                    |           +-------------+
      |--- acquire ------->|                    |                  |
      |<-- token 33 -------|                    |                  |
      : A frozen           |                    |                  |
      : (GC / VM stall)    |                    |                  |
      :                    | lease 33 expires   |                  |
      :                    |<------ acquire ----|                  |
      :                    |------ token 34 --->|                  |
      :                    |                    |--- write(34) --->| max=34
      | resumes, believes  |                    |                  |
      | it still holds it  |                    |                  |
      |------------------------ write(33) ------------------------>| 33 < 34
      |<----------------------- rejected: stale -------------------| REJECT
      |                    |                    |                  |

Client A stalls holding token 33, its lease expires, Client B acquires token 34 and writes first, so the resource rejects A's late write because 33 is lower than the highest token it has already accepted.

How Redis locks and Redlock work, and where they fall short

A single-instance Redis lock is one command: SET with NX (only set if absent) and an expiry such as PX. The value is a random token unique to the client, and the expiry is what releases the lock if the holder crashes. The token matters at release time. A client whose lock has already expired must not delete a lock that someone else now holds, so release runs as a Lua script that compares the stored value to the client's token and deletes only on a match. The script is atomic, so nothing can change the key between the check and the delete.

Replication undermines this setup. Redis replicates asynchronously, which means the primary can acknowledge the SET, crash before the write reaches a replica, and fail over to a replica that never saw the key. A second client then acquires a lock the first client still believes it holds.

Redlock addresses failover by running N independent primaries, typically five, with no replication between them. A client tries to acquire the same key on each node using a short per-node timeout. The client holds the lock only if it succeeds on a majority of nodes and the total elapsed time is less than the TTL. Its remaining validity is the TTL minus the elapsed time minus an allowance for clock drift.

Martin Kleppmann criticized the design on two grounds. A process pause or a clock jump can let a lock expire while the holder keeps working, and Redlock produces no fencing token that would let downstream systems reject the stale holder. Salvatore Sanfilippo, the creator of Redis, responded that the timing assumptions are reasonable in practice. He pointed out that the algorithm checks elapsed time after acquiring the lock and that clocks can be configured to slew gradually rather than jump.

In practice, you take on five nodes to operate and still get no fencing token. Redlock rarely buys enough over a single node to justify that cost.

Copyimport secrets
import redis

# Single-instance Redis lock: SET NX PX to acquire, Lua compare-and-delete to release.
r = redis.Redis(host="localhost", port=6379)

LOCK_KEY = "lock:invoice:42"
TTL_MS = 10_000  # expiry frees the lock if the holder crashes

# Release only if the stored value is still our token. The script runs
# atomically, so no other client can take the key between GET and DEL.
RELEASE_SCRIPT = """
if redis.call('GET', KEYS[1]) == ARGV[1] then
    return redis.call('DEL', KEYS[1])
else
    return 0
end
"""

def acquire(key: str, ttl_ms: int) -> str | None:
    token = secrets.token_hex(16)  # random value unique to this client
    # nx=True: set only if absent; px: expiry in milliseconds
    if r.set(key, token, nx=True, px=ttl_ms):
        return token
    return None

def release(key: str, token: str) -> bool:
    # Returns False if our lock already expired and someone else may hold it
    return r.eval(RELEASE_SCRIPT, 1, key, token) == 1

token = acquire(LOCK_KEY, TTL_MS)
if token:
    try:
        pass  # critical section: must finish well within TTL_MS
    finally:
        release(LOCK_KEY, token)
# Caveat: async replication can still lose this key on failover,
# and nothing here gives downstream systems a fencing token.

A single-node Redis lock in Python with redis-py: SET with NX and PX plus a random token to acquire, and an atomic Lua compare-before-delete to release.

How etcd leases and revision-based fencing work

etcd starts from the guarantee Redlock lacks. Every write goes through Raft and is committed only when a majority of members agree, so all clients observe one linearizable order of changes. A lock built on etcd has three parts.

Leases. A client grants a lease with a TTL and keeps it alive by streaming keepalive requests. Keys attached to the lease are deleted when it expires or is revoked. A crashed holder therefore releases its lock automatically, and no client has to compare clocks.

Revisions as fencing tokens. etcd keeps one cluster-wide revision counter that increases on every write. Each key records the create_revision at which it was created and the mod_revision of its latest change. Because the lock key is created when the lock is acquired, its create_revision is unique and higher than any earlier holder's. The holder passes that value to the protected resource. The resource stores the highest value it has seen and rejects any request carrying a lower one.

Concurrency helpers. The Go client's concurrency package wraps a lease in a session. Its mutex creates a key under a shared prefix and waits until every key with a lower create_revision is deleted. Each waiter watches only its immediate predecessor, so a release wakes one client rather than all of them. Leader election uses the same mechanism.

Costs. You run another consensus cluster, usually three or five members, and have to size, back up, and monitor it. etcd is sensitive to fsync latency on its write-ahead log. Slow disks, GC pauses, or a partition can delay keepalives until the lease expires while the client still believes it holds the lock. Treat the session's done signal as an order to stop work immediately, and use fencing for the window before your code notices.

Copy#!/usr/bin/env bash
# Inspect the etcd primitives behind a lock: a lease, a key bound to it,
# and the key's create_revision used as a fencing token.
# Assumes etcdctl (v3 API) can reach the cluster and jq is installed.
set -euo pipefail

# 1. Grant a 15s lease. Output looks like: "lease 32695410dcc0ca06 granted with TTL(15s)"
LEASE=$(etcdctl lease grant 15 | awk '{print $2}')

# 2. Keep the lease alive from the background. If this stream stalls
#    (slow disk, GC pause, partition), the lease expires and the key is deleted.
etcdctl lease keep-alive "$LEASE" > /dev/null &
KEEPALIVE_PID=$!

# 3. Attach the lock key to the lease. A plain put does not enforce exclusivity.
#    The concurrency mutex instead writes a unique key per session under a
#    shared prefix and waits for keys with lower create_revisions.
etcdctl put --lease="$LEASE" /locks/billing-job/worker-a "host-a"

# 4. Read the key's revisions. create_revision is assigned once at creation and
#    comes from the cluster-wide counter, so a later holder always gets a higher one.
TOKEN=$(etcdctl get /locks/billing-job/worker-a -w json | jq '.kvs[0].create_revision')
echo "fencing token: $TOKEN"

# 5. Send $TOKEN with every write to the protected resource. The resource keeps
#    the highest token it has seen and rejects lower ones.

# 6. Check how much time remains on the lease.
etcdctl lease timetolive "$LEASE"

# 7. Release explicitly. Revoking the lease deletes every key attached to it.
kill "$KEEPALIVE_PID"
etcdctl lease revoke "$LEASE"

Using etcdctl to grant a lease, keep it alive, bind a lock key to it, read the key's create_revision as a fencing token, and release the lock by revoking the lease.

How Postgres advisory locks work and how to implement them safely

Postgres offers two scopes of advisory lock. Session-level locks, taken with pg_advisory_lock (blocking) or pg_try_advisory_lock (returns false instead of waiting), stay held until you call pg_advisory_unlock or the connection closes. They ignore transaction boundaries, so a rollback does not release them. They also stack: locking the same key twice requires two unlocks. Transaction-level locks, taken with pg_advisory_xact_lock and pg_try_advisory_xact_lock, are released automatically at commit or rollback, and there is no explicit unlock.

The transaction-scoped form avoids the fencing problem described earlier, as long as the protected data lives in the same database. Take the lock, perform the write, and commit, all in one transaction. If the client stalls and its connection is dropped, the server aborts the transaction, and the write never commits. No stale holder can complete a write after losing the lock, because the lock and the write succeed or fail together.

Connection poolers running in transaction mode break session-level locks. Your next statement may run on a different backend, so you might unlock on a connection that never held the lock, or the lock might stay on a pooled connection that another client then uses. Use transaction-scoped locks in that setup, or check out a dedicated, unpooled connection.

Keys are plain integers shared by every application using the database. Use the two-argument (int, int) form, with the first argument identifying the lock type, or hash a namespaced string into a bigint. This prevents unrelated services from colliding on the same key. Finally, every session lock occupies a connection for as long as it is held, so long-held locks reduce the capacity of your pool.

Copy-- Transaction-scoped advisory lock guarding a write in the same database.
-- Works behind a transaction-mode pooler: lock and write share one transaction.
BEGIN;

-- Two-argument (int, int) form: the first key names the lock type,
-- the second identifies the resource. 1001 = 'invoice-generation' (team convention).
SELECT pg_try_advisory_xact_lock(1001, 42) AS got_lock;

-- If got_lock is false, another worker holds it: the application should
-- ROLLBACK and retry later instead of proceeding.

-- Protected write. It commits only if this transaction commits, so a stalled
-- client whose connection is dropped cannot complete it after losing the lock.
UPDATE invoices
   SET status = 'generating', updated_at = now()
 WHERE customer_id = 42
   AND status = 'pending';

COMMIT;  -- the lock is released here (or on ROLLBACK); there is no unlock call

-- Alternative namespacing: hash strings into the two int keys.
-- hashtext() is an internal function, and hashes can collide, so reserve
-- this for cases where a fixed lock-type registry is impractical.
BEGIN;
SELECT pg_advisory_xact_lock(hashtext('billing-service'), hashtext('customer:42'));
-- ... protected write ...
COMMIT;

-- Inspect advisory locks currently held. With the two-int form,
-- classid holds the first key and objid holds the second.
SELECT pid, classid, objid, granted
  FROM pg_locks
 WHERE locktype = 'advisory';

A transaction-scoped advisory lock taken with a namespaced two-key form, with the protected write in the same transaction so the lock and the write commit or abort together.

Behavior under failure and a decision guide

| | Fencing | Timing assumptions | Holder dies | Partition | Failover / restart | Ops burden | |---|---|---|---|---|---| | Single Redis + TTL | None built in | Work finishes before TTL | Freed at TTL | Isolated holder outlives its TTL | Async replica may lack the key; restart without persistence drops it | Low | | Redlock | None | Bounded clock drift, pauses and network delay | Freed at TTL | Majority side can grant | Restarted node can forget a lock and grant it twice | Several independent nodes | | etcd lease | Key revision | Only for liveness | Freed at lease expiry | Minority side cannot write or renew | Raft preserves state | Quorum cluster on fast disks | | Postgres advisory | Not needed when writes share the lock's transaction | Connection liveness | Freed when the server sees the session end | Dead client holds the lock until TCP timeout | Locks are not replicated, but the transaction dies with them | None beyond Postgres |

Postgres is the right choice when the protected resource lives in the same database. Take the lock and do the writes in one transaction, and correctness follows from the transaction. Avoid it for resources outside that database. Also avoid it behind a transaction-mode pooler, because session locks leak across clients.

etcd fits cross-system correctness and leader election when you already run it or can justify running it. Pass the revision to the downstream system and have it reject stale values. Avoid it if no team will own a quorum cluster.

Single Redis with a TTL suits efficiency-only cases, such as deduplicating cron runs or suppressing cache stampedes, where an occasional double execution is harmless. Avoid it whenever two holders would corrupt data.

Redlock is rarely the answer. It costs more to run than single Redis but still provides no fencing. Use it only if you need availability beyond a single node and are prepared to accept its timing assumptions.

Alternatives to locking and the impact on cost and engineering time

Often the cheapest lock is no lock at all. A lock serializes every contender for a key. If the work under the lock takes 200 ms, that key handles at most five operations per second no matter how many replicas you add, and waiters queue behind slow holders. Several designs get correctness without that queue:

• Idempotency keys. Store a client-supplied key and the result in the same transaction as the side effect. A retry or duplicate delivery returns the stored result instead of repeating the work.
• Conditional writes (compare-and-set). Update with a version check such as WHERE version = $n. The loser sees zero rows affected and retries. The storage layer enforces the check, so a paused process cannot overwrite newer data. That is the guarantee fencing tokens exist to provide.
• Unique constraints. Let the database reject a second insert of the same order ID or job claim.
• Partitioning. Route each key to exactly one consumer, for example by queue partition, so two workers never compete for it.

These options reuse a database you already run. etcd adds a cluster that someone must provision, monitor, back up, upgrade and be paged for.

Illustrative estimate (assumed numbers, not a benchmark): Annual cost = (nodes × cost per node) + (ops hours × hourly rate) = (3 × $1,200) + (40 × $100) = $7,600, before any incidents.

Incident time costs more than either term. Lock bugs show up only during pauses and partitions, so they are hard to reproduce. A violated unique constraint fails immediately and the same way every time. Use a lock only when the work cannot be made idempotent or conditional.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to distributed locks microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.