Distributed Locks Fencing Tokens: Why Leases Alone Are Not Safe

Here is a failure that has corrupted real data: two workers both believe they own the same lock, both write to the same record, and the last writer wins with stale state. Nothing crashed. No network partition was declared. One process simply stopped for a few seconds, its lease expired, and when it woke up it carried on as if nothing had happened. Getting distributed locks fencing tokens right is the difference between a lock that is merely “usually exclusive” and one that protects your data when timing assumptions fail.

This matters more in 2026 than it did a decade ago. Fleets of IoT gateways, digital twin synchronizers and PLM integration workers all compete for the same device shadows, work orders and part revisions, and they run on garbage-collected runtimes, containers that get CPU-throttled, and VMs that get live-migrated. Every one of those can freeze a process at the worst moment.

You will leave with a precise model of why timeout-based locks fail, a fair reading of the Kleppmann and antirez Redlock debate, working recipes for etcd, ZooKeeper and PostgreSQL, runnable Python and Go code that enforces fencing at the storage layer, and a decision framework for when you should skip locks entirely.

What this covers: the failure model, the Redlock debate, fencing tokens, etcd leases and revisions, ZooKeeper sequential znodes, Postgres advisory locks, optimistic concurrency, timeline walk-throughs, and a recommendation checklist.

Context and Background

A lock in a single process is easy to reason about: the operating system guarantees that if a thread holds a mutex, no other thread does, and if the holder dies the process dies with it. Distribution breaks both guarantees. The lock lives on a different machine from the holder, the holder can be alive but unreachable, and the lock service cannot tell the difference between a dead holder and a slow one. This is the core impossibility behind failure detection in asynchronous systems: a timeout is a guess, not a fact.

The industry answer is the lease: a lock with an expiry. The service grants the lock for a bounded time, the holder must renew it before expiry, and if the holder disappears the lock frees itself. Leases were formalised by Gray and Cheriton in 1989 for cache consistency, and they remain the backbone of every practical lock service, including etcd, ZooKeeper sessions, Consul sessions and Redis SET NX PX locks.

The catch is that a lease is a promise made by the lock service about its own clock. The holder only knows what the service said at one moment in the past. Between that moment and the moment the holder actually touches the protected resource, an arbitrary amount of wall-clock time can pass, and the holder has no reliable way to notice. Martin Kleppmann’s 2016 essay How to do distributed locking and Salvatore Sanfilippo’s (antirez) reply Is Redlock safe? turned this gap into the most cited argument in the field, and we will treat both fairly below.

Two broad uses of locks must be separated from the start. An efficiency lock prevents redundant work, for example two workers both rebuilding the same cache entry. If it fails occasionally, you pay a little extra compute. A correctness lock protects an invariant, for example only one writer may advance a device’s firmware state machine. If it fails, data is wrong. Almost every dispute about distributed locking dissolves once you say which one you are building.

If you have read our analysis of Raft versus Paxos leader election failure modes, you already know that leader election is a lock with a different name: “I am the leader for term T” is exactly “I hold the lease with token T”. The same hazards and the same remedies apply, and this post develops them for application-level locks.

Why Naive Locks Break: Pauses, Skew and Delay

Direct answer: A lock with a timeout is unsafe for correctness because the holder can be paused, delayed or clock-skewed for longer than the lease without knowing it. After the lease expires another client acquires the lock, and the original holder’s late write is indistinguishable from a legitimate one unless the resource itself checks a fencing token.

The canonical naive implementation is a Redis single-instance lock:

# Naive lock: fine for efficiency, unsafe for correctness
import redis, uuid
r = redis.Redis()

def acquire(name, ttl_ms=10_000):
    token = str(uuid.uuid4())
    ok = r.set(name, token, nx=True, px=ttl_ms)
    return token if ok else None

def release(name, token):
    # compare-and-delete must be atomic, hence Lua
    lua = """
    if redis.call('get', KEYS[1]) == ARGV[1] then
        return redis.call('del', KEYS[1])
    else return 0 end
    """
    r.eval(lua, 1, name, token)

The release script is already a hint of the problem class. Without the compare, a slow client would delete a lock that now belongs to someone else. That bug is well understood. The deeper bug is that even a perfectly implemented release does nothing to stop a client whose lease expired from using the resource.

Process pauses are not rare

A process can stop executing between any two instructions. The common causes are stop-the-world garbage collection in the JVM, Go, .NET or Node runtimes, page faults and swapping, CPU throttling by container cgroup quotas, VM live migration, the hypervisor stealing time, a SIGSTOP from an operator debugging in production, and a laptop-class device going to sleep. Kleppmann’s essay uses a long GC pause as the leading example, and the mechanism is general: the client checks “do I hold the lock?”, the check returns true, the process freezes, and by the time it unfreezes the answer is false but the code path has already committed to the write.

The key insight is that checking the lease just before the write does not help. The pause can strike after the check and before the write reaches storage. The window may be a few microseconds wide, but a fleet making millions of writes per day will eventually land a pause in that window. Probability is not the issue; the absence of any guarantee is.

Network delay is a pause in disguise

The same hazard applies on the wire. A write can leave the client while the lease is valid, sit in a switch queue, a TCP retransmit buffer, or a retrying proxy, and arrive at storage after the lease has expired and another client has written. The client did nothing wrong by any local test. Packet delay is unbounded in asynchronous networks, so any scheme that relies on “the write will arrive within the lease” imports a timing assumption it cannot enforce.

Clock skew and clock jumps

Leases rely on clocks in two places: the service decides when a lease expires, and the client decides when to stop trusting its lease. If these clocks run at different rates, or one jumps, mutual exclusion silently weakens. NTP step corrections can move a wall clock forward or backward. Virtual machines can resume with a stale clock. A Redis node whose clock jumps forward expires keys early.

There is a useful distinction here. A monotonic clock (on Linux, CLOCK_MONOTONIC; in Go, the monotonic reading inside time.Time; in Python, time.monotonic()) never jumps backwards and is the right tool for measuring elapsed time on one machine. A wall clock is the right tool for timestamps shared between machines, but only to within an error bound that you cannot know without special hardware. Leases should measure durations on monotonic clocks, but that only fixes skew, not pauses: a paused process’s monotonic clock may or may not advance depending on the platform and virtualization layer.

The three-line timeline of every lock bug

Every incident in this family reduces to the same three events, and the figure below shows it end to end.

Figure 1: Failure timeline for a lease-based lock. Client A’s pause outlasts the lease, Client B legitimately acquires and writes, and then A resumes and overwrites B because storage cannot tell the writes apart.

In the sequence, nothing the lock service did was wrong. The lease expired exactly as specified, and B’s acquisition was legitimate. The flaw is architectural: the lock service and the storage service are separate systems, and only the lock service knows who the current holder is. Storage accepts whatever arrives. The rest of this post is about closing that gap.

The Redlock Debate: Kleppmann versus antirez

Redlock is the algorithm Redis’s documentation proposes for locks across multiple independent Redis masters. A client records the start time, tries SET resource random_value NX PX ttl on each of N instances (typically five) with a short per-instance timeout, and considers the lock acquired if it succeeded on a majority (N/2 + 1) and the elapsed time is less than the TTL. The remaining validity is the TTL minus elapsed time, minus a drift allowance. On failure it releases everything. The goal was a lock that survives the loss of a minority of nodes, an improvement over a single Redis instance that is a single point of failure and, with asynchronous replication, can lose a lock on failover.

Kleppmann’s critique

Kleppmann published his critique on 8 February 2016. His main points, which I summarise from the essay:

Redlock sits in an awkward middle. For efficiency locks it is heavier and costlier than a single Redis node. For correctness locks it is not safe enough.
It relies on timing assumptions for safety. Bounded network delay, bounded process pauses and bounded clock error must all hold. He gives scenarios where a clock jump on one Redis node, or a client pause after acquiring, lets two clients both believe they hold the lock.
It produces no fencing token. The random value Redlock stores is unique but not monotonic, so a storage service cannot use it to reject stale writers. He suspected that generating such a number would effectively need a consensus algorithm.
His recommendation: use a simple single-node Redis lock if occasional failure is acceptable, and for correctness use a consensus-backed system like ZooKeeper or a database with solid transactional guarantees, in both cases enforcing fencing tokens at the resource.

antirez’s reply

Antirez responded in Is Redlock safe?. Summarising his positions:

Clocks. Redlock needs only a semi-synchronous model in which processes measure elapsed time at roughly the same rate with bounded relative error, not synchronised absolute clocks. He conceded that wall-clock jumps are a real problem and said Redis and Redlock should move to a monotonic time API.
Pauses. Redlock rechecks elapsed time after collecting the majority, so a pause during acquisition simply reduces the remaining validity. The residual risk is a pause after the last check, which he argues affects every auto-releasing lock, including those Kleppmann recommends.
Fencing. If a store enforces increasing tokens, he argues, it is already effectively linearizable and the lock adds little. He also notes that many protected resources are non-transactional, such as physical devices or third-party APIs, where a token check is impossible. He proposes using Redlock’s random value for a check-and-set on the resource instead.

Where each side is right

Reading both carefully, the disagreement is narrower than the tone suggests. Both agree that a lease holder can be paused after its last validity check and that the resource must defend itself if correctness is at stake. Antirez’s position is, in effect, that a lock is a best-effort optimisation unless the resource participates, and that Redlock is acceptable when participation is impossible. Kleppmann’s position is that if the resource can participate, a monotonic token is the right primitive, and that Redlock cannot supply one.

I would put it this way (this is my reading, not either author’s wording). Redlock’s value claim, “stronger than a single Redis node”, is real for availability. Its safety claim for correctness workloads depends on timing assumptions that production environments violate, and it has no built-in path to a fencing token. If you cannot make the resource check anything, no lock protocol can save you from a paused holder, and you should instead design the operation to be idempotent or reduce its blast radius.

Redis’s own documentation still presents Redlock with a note pointing to this debate, and client libraries implement it. Whether you adopt it is a risk decision. For anything where two simultaneous writers corrupt state, choose a mechanism that yields a monotonic token and put the check in the storage layer.

Fencing Tokens: Moving the Check to the Resource

Direct answer: A fencing token is a number that increases every time a lock is granted. The client sends it with every write, and the resource rejects any write whose token is lower than the highest one it has seen. This makes safety independent of clocks and pauses, because stale writers are refused at the only place that matters.

Figure 2: With fencing, the lock service issues monotonically increasing tokens and storage remembers the maximum accepted. The delayed write carrying token 33 is rejected.

The mechanism in one invariant

Let the lock service issue token t(n) for the n-th grant, with t(n) < t(n+1). Let storage keep max_seen. On a write carrying token t, storage atomically checks t >= max_seen (or strictly greater for a new holder; see below), updates max_seen = t, and applies the write. Because a later grant always has a higher token, and the first write by the new holder raises max_seen, every subsequent write by the old holder fails the check.

The invariant protects against the pause, the network delay and the clock skew simultaneously, because it does not consult a clock. The only requirement is that the token sequence is monotonic across all lock holders, which in practice means it is generated by a linearizable component.

Check and update must be atomic at storage

A subtle point: the comparison and the write have to be one atomic step in the storage layer. If storage reads max_seen, compares in application code, and then writes, a second request can interleave. The implementation must be a single conditional update, a compare-and-set, a serializable transaction, or a single-threaded state machine. Section “Runnable Fencing at the Storage Layer” shows this in SQL and Go.

What generates monotonic tokens

Tokens come for free from consensus-backed systems that already number their writes:

Source	Token	Monotonic guarantee
etcd	`ModRevision` / `CreateRevision` of the lock key, or `Header.Revision`	Global, increasing per cluster, from Raft-ordered writes
ZooKeeper	`czxid` of the lock znode, or the sequence number suffix	zxid is globally ordered; sequence suffix ordered per parent
PostgreSQL	A `bigserial` or a counter row incremented in the lock transaction	Monotonic per sequence, with gaps allowed
Redis single node	`INCR` on a key	Monotonic only while that node’s state survives failover
Redlock	None built in	Random value is unique, not ordered

Notice what the strong sources share: they derive the token from the same ordered log that decides who holds the lock. The token and the ownership decision are the same fact, which is why they cannot disagree.

The requirement that scares people: the resource must cooperate

Fencing does not remove the need for a trustworthy resource; it moves the enforcement point. Your database, object store wrapper, or device gateway must accept and check a token. That is easy for a database you control: add a column. It is hard for third-party APIs that ignore your headers. For those, use idempotency keys on the request, or put a small proxy in front of the API that performs the fencing check, or accept that the lock is best-effort and design for repair.

For the same reason, fencing composes well with event-driven patterns. If your workflow coordinator handles multi-step operations, the token can be propagated through each step, as discussed in our write-up on saga orchestration versus choreography, so that a zombie orchestrator cannot trigger compensations after being replaced.

Strictly greater versus greater or equal

A holder issues many writes under one grant, so storage must accept a repeated token. The rule is therefore t >= max_seen. A new holder always has t > max_seen from the previous holder, so its first write raises the bar. If you use a strict > you will reject the holder’s own second write. This is a common bug and is why the example code below uses >=.

etcd Leases, Revisions and the Concurrency Package

etcd is a strong default for lock services because its writes go through Raft, so every key modification gets a cluster-wide, monotonically increasing revision. Its lease primitive and its concurrency package build locks and elections on top of that log. The details below come from the etcd client v3 concurrency package documentation; where I go beyond what the page states, I say so.

Leases and sessions

A lease in etcd is a server-side object with a TTL. Keys can be attached to a lease and are deleted when the lease expires or is revoked. The client keeps a lease alive by sending keep-alive requests. The concurrency.NewSession helper wraps this: it grants a lease, starts a keep-alive loop and exposes the lease through the session. Per the package docs, WithTTL sets the session TTL in seconds, and a value of zero or less selects the default of 60 seconds. WithLease lets you reuse an existing lease, which is useful for reclaiming leadership after a restart, and if the session’s context is cancelled before Close() completes, the lease is left to expire instead of being revoked.

The keep-alive loop is where the pause hazard returns. If the process freezes, keep-alives stop, the lease expires on the server after the TTL, and the lock keys vanish. The frozen process still thinks it holds the lock. The session exposes a Done() channel that closes when the lease is lost, but a frozen process cannot observe it until it resumes, by which time the damage path is already open.

How the mutex works

concurrency.NewMutex(session, prefix) creates a mutex under a key prefix. Conceptually (this is my description of the recipe; the documentation page does not spell out the key format) acquisition does the following:

Build a key from the prefix and the session’s lease ID, and attach it to the lease.
In one transaction, compare the key’s CreateRevision to zero. If absent, put it; otherwise fetch it. The transaction also fetches the key with the lowest creation revision under the prefix.
If the caller’s key has the lowest creation revision, it owns the lock. Otherwise it waits for each key with a lower creation revision to be deleted, then re-checks.

This is the same queue-by-order idea ZooKeeper uses, and it avoids the thundering herd because each waiter watches predecessors rather than a single hot key. Lock(ctx) blocks and, if the context is cancelled while waiting, tries to clean up its stale entry. TryLock(ctx) returns ErrLocked immediately if another session holds the lock. Unlock(ctx) deletes the key. Header() returns the response header etcd sent when the lock was acquired, and Key() returns the full key.

Using the revision as a fencing token

The documentation page does not mention fencing tokens, so treat the following as a design pattern rather than a documented feature. The lock key was created in a Raft-committed transaction, so its CreateRevision is unique and strictly greater than that of any earlier lock holder’s key. That number is a valid fencing token. The response header’s revision at acquisition is also monotonic and can serve the same role.

package main

import (
    "context"
    "fmt"
    "log"
    "time"

    clientv3 "go.etcd.io/etcd/client/v3"
    "go.etcd.io/etcd/client/v3/concurrency"
)

// Illustrative sketch against a real etcd endpoint; not run in this article's test harness.
func main() {
    cli, err := clientv3.New(clientv3.Config{
        Endpoints:   []string{"localhost:2379"},
        DialTimeout: 3 * time.Second,
    })
    if err != nil {
        log.Fatal(err)
    }
    defer cli.Close()

    sess, err := concurrency.NewSession(cli, concurrency.WithTTL(10))
    if err != nil {
        log.Fatal(err)
    }
    defer sess.Close()

    m := concurrency.NewMutex(sess, "/locks/device-7")
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    if err := m.Lock(ctx); err != nil {
        log.Fatal(err)
    }
    defer m.Unlock(context.Background())

    // Fetch the lock key's creation revision to use as the fencing token.
    resp, err := cli.Get(ctx, m.Key())
    if err != nil || len(resp.Kvs) == 0 {
        log.Fatal("lock key missing", err)
    }
    token := resp.Kvs[0].CreateRevision
    fmt.Println("fencing token:", token)

    // Pass `token` with every write to storage that enforces token >= max_seen.
    select {
    case <-sess.Done():
        log.Fatal("lease lost: abandon work")
    default:
    }
}

Notice the final select. Watching sess.Done() is a useful optimisation: it lets a healthy but partitioned client stop early. It is never a safety mechanism, because a frozen client cannot run it. Safety comes from the storage check on token.

etcd’s own caveat

etcd is built on Raft, so its locks inherit Raft’s properties: they stay correct as long as a majority of members are available and in contact, and the leader lease and election timeouts only affect liveness, not safety. But the application-level lease TTL is a different timer from Raft’s internal timers, and it is the lease TTL that decides when a holder is evicted. Choosing it is a trade-off: a short TTL recovers quickly from dead holders but evicts slow, healthy ones during hiccups such as compaction stalls or network blips; a long TTL tolerates hiccups but extends the outage when a holder dies. There is no universally correct number, so measure your real pause distribution before choosing.

ZooKeeper: Ephemeral Sequential Znodes

ZooKeeper provides the same ordering guarantees with a different vocabulary. Clients connect and establish a session with a negotiated timeout. A session stays alive through heartbeats; if the server does not hear from the client within the timeout, the session expires and all ephemeral znodes created by that session are deleted. Sequential znodes get a monotonically increasing counter suffix from the parent, assigned atomically by the server.

The lock recipe

The standard recipe, which Apache Curator implements as InterProcessMutex, is:

Create an ephemeral sequential znode under a lock path, for example /locks/device-7/lock-, receiving lock-0000000042.
List the children of /locks/device-7 and sort them.
If your node has the lowest sequence number, you hold the lock.
Otherwise set a watch on the node immediately preceding yours and wait for it to be deleted, then repeat from step 2.

Watching only the predecessor means a release wakes exactly one waiter, not the whole herd. Because the znode is ephemeral, a crashed holder’s lock disappears when its session expires.

Figure 3: The ZooKeeper and etcd lock recipes are structurally the same: create an ephemeral, ordered entry, compare order, wait on the predecessor, and reuse the ordering number as the fencing token.

zxid and the fencing token

Every state change in ZooKeeper carries a zxid, a 64-bit transaction identifier whose high bits hold the leader epoch and low bits a counter, assigned by the Zab atomic broadcast protocol. A znode’s czxid is the zxid of the transaction that created it. Kleppmann’s essay notes that the zxid or the znode version number can serve as a fencing token, and the reasoning matches etcd’s: the lock znode is created in a totally ordered transaction, so its czxid is larger than that of any previous holder’s znode.

Session expiry subtleties

ZooKeeper has a client-side wrinkle that is easy to miss. When a client loses its connection, it moves into a disconnected state but its session is still valid on the server until the timeout. Reconnecting to another server within the timeout preserves the session and the lock. But the client library cannot know whether the session has already expired on the server while it was partitioned. Well-written clients treat “disconnected” as “lock possibly lost” and stop work, which is the conservative reading. The same pause problem remains: a frozen client resumes holding stale beliefs, so writes still need a fencing check.

ZooKeeper deployments are also operationally heavier than etcd: a quorum ensemble to run, JVM tuning, and session timeouts bounded by the server’s configured minimum and maximum. Many teams keep it because Kafka, HBase or Solr already depend on it, and reusing the existing ensemble for locks is cheap. That is a legitimate reason; new standalone deployments usually pick etcd for the simpler API and Go-native tooling.

PostgreSQL Advisory Locks

If your application already depends on PostgreSQL, advisory locks are a pragmatic option with strong semantics inside one database. As the PostgreSQL explicit locking documentation describes, advisory locks are application-defined: the server stores and arbitrates them but gives them no meaning. They come in two scopes.

Session-level locks (pg_advisory_lock, pg_try_advisory_lock) are held until explicitly unlocked or the session ends. They do not honour transaction semantics: a lock taken inside a transaction that rolls back is still held afterwards.
Transaction-level locks (pg_advisory_xact_lock, pg_try_advisory_xact_lock) are released automatically at commit or rollback, and there is no unlock call.

-- Transaction-scoped lock plus a fencing counter, all in one transaction
BEGIN;
SELECT pg_advisory_xact_lock(hashtext('device-7'));

-- Issue a monotonic token and apply the write atomically
UPDATE device_state
   SET payload = $1,
       fence   = fence + 1
 WHERE id = 'device-7'
RETURNING fence;
COMMIT;

Why Postgres locks are safer than they look, and where they are not

A lock tied to a database session has an unusually strong coupling: the lock lives in the same system that stores the data. If the holder’s connection drops, the server releases the lock when it notices the session ended, and the transaction aborts with it. With transaction-level locks, the lock and the write commit or roll back together, so a stale holder cannot write after losing the lock inside the same transaction. This collapses the lock-service-versus-storage gap that causes Figure 1, because they are the same service.

The limits are real, however:

Scope. The guarantee only covers data in that one Postgres instance. If the protected resource is an object store, a message broker or a device, you are back to fencing.
Failover. Locks are not replicated to standbys in a way that survives promotion. After a failover, the new primary has no memory of locks held on the old one, and the old primary may still be running if it is not fenced off at the infrastructure layer.
Pooling. Session-level locks interact badly with connection poolers in transaction mode, since the session that holds the lock may be handed to another client. Prefer transaction-level locks behind poolers.
Idle connections. A holder that is paused but whose TCP connection is still open keeps the lock indefinitely. Postgres does not expire advisory locks; it releases them only when the session ends. idle_in_transaction_session_timeout and TCP keepalive settings are your leases here.
Capacity. Advisory locks share the shared-memory lock table, which is sized from max_locks_per_transaction and max_connections, so very large numbers of simultaneous locks can exhaust it.
A query pitfall. The documentation warns that LIMIT may not be applied before a locking function in the select list runs, which can lock more rows than intended. Wrap the limited query in a subquery.

The fencing counter pattern above is the portable core: a fence column that increments in the same transaction that acquires the logical right to write. Downstream systems then receive that value with each request.

Optimistic Concurrency: Often You Do Not Need a Lock

The cleanest answer to “how do I make my distributed lock safe” is frequently “remove it”. If the shared state lives in a store that supports conditional writes, you can use optimistic concurrency control (OCC): read the record with its version, compute the change, and write back only if the version is unchanged.

-- Compare-and-set with a version column
UPDATE work_order
   SET status = 'RELEASED', version = version + 1
 WHERE id = 4711 AND version = 12;
-- rowcount 0 means someone else won; reread and retry or give up

This is functionally a fencing check where the “token” is the version you read. A paused client that wakes up carries version 12; the row is now at 13; the write matches zero rows; and the stale client learns it lost. No lease, no clock, no lock service. The same idea appears as If-Match ETags in HTTP, conditional puts in DynamoDB (ConditionExpression), generation-match preconditions in object stores, and etcd’s transaction Compare on ModRevision.

OCC is not free. Under high contention on one record, retries pile up and throughput collapses, while a lock queue would serialise cleanly. It also only protects a single read-modify-write on one record; if the critical section touches several systems or performs side effects that cannot be rolled back, there is nothing to compare. That is the territory where leases and fencing, or a saga with compensations, take over.

My rule of thumb, opinion rather than universal law: if you can express the critical section as a single conditional write, do that. If the critical section is long, spans systems, or must serialise access to something that cannot check versions, use a lease with fencing. If neither is possible, the work should be idempotent and its duplication tolerated.

Figure 4: A decision flow. Start from the cost of a stale writer, then ask whether the store can check a version before reaching for a lock service.

Runnable Fencing at the Storage Layer

The code below is the smallest end-to-end demonstration of the invariant. I ran both snippets locally; the timings are artificial and the lock service is a toy, so read them as a demonstration of the check, not as a production design.

Python: a toy lease service plus a fenced SQL write

The lock service hands out a counter value on each grant, and the “storage” is an SQLite table whose UPDATE carries the fence condition in its WHERE clause. That makes the compare and the write one atomic statement.

import sqlite3, threading, time, itertools

db = sqlite3.connect(":memory:", check_same_thread=False, isolation_level=None)
db.execute("CREATE TABLE doc (id TEXT PRIMARY KEY, body TEXT, fence INTEGER NOT NULL)")
db.execute("INSERT INTO doc VALUES ('device-7', 'init', 0)")
mu = threading.Lock()

def fenced_write(doc_id, body, token):
    """Apply the write only if token >= highest token seen. One atomic UPDATE."""
    with mu:
        cur = db.execute(
            "UPDATE doc SET body = ?, fence = ? WHERE id = ? AND fence <= ?",
            (body, token, doc_id, token))
        return cur.rowcount == 1

class LockService:
    """Toy lease service with a monotonic counter, as etcd revisions would give."""
    def __init__(self, ttl):
        self.ttl, self.counter, self.owner, self.exp = ttl, itertools.count(1), None, 0.0
        self.mu = threading.Lock()
    def acquire(self, who):
        with self.mu:
            if self.owner is None or time.monotonic() > self.exp:
                self.owner, self.exp = who, time.monotonic() + self.ttl
                return next(self.counter)
            return None

svc = LockService(ttl=0.3)
results = {}

def client_a():
    tok = svc.acquire("A")
    time.sleep(0.6)                      # simulated GC pause, longer than the lease
    results["A"] = (tok, fenced_write("device-7", "stale-from-A", tok))

def client_b():
    time.sleep(0.4)                      # lease already expired
    tok = svc.acquire("B")
    results["B"] = (tok, fenced_write("device-7", "fresh-from-B", tok))

ta, tb = threading.Thread(target=client_a), threading.Thread(target=client_b)
ta.start(); tb.start(); ta.join(); tb.join()
print(results)
print(db.execute("SELECT * FROM doc").fetchone())

The output on my run was {'B': (2, True), 'A': (1, False)} followed by ('device-7', 'fresh-from-B', 2). Client A held token 1, slept past its lease and was refused when it tried to write; client B’s write landed with token 2. Remove the fence <= ? condition and A’s stale write would overwrite B’s, which is Figure 1 reproduced in thirty lines.

The condition reads fence <= token because the stored fence must be less than or equal to the incoming token for the write to pass. The WHERE ... AND fence <= ? clause is the entire safety mechanism; the lock service is only the source of the numbers.

Go: a fenced store with typed rejection

The same check in Go, as it would appear in a storage gateway in front of a device registry or an object store. The mutex here stands for whatever atomic primitive your real storage has (a conditional update, a CAS, a Raft-applied command).

package main

import (
    "errors"
    "fmt"
    "sync"
)

var ErrStale = errors.New("stale fencing token")

type Store struct {
    mu    sync.Mutex
    fence int64
    value string
}

// Put applies the write only if token >= the highest token seen.
func (s *Store) Put(token int64, v string) error {
    s.mu.Lock()
    defer s.mu.Unlock()
    if token < s.fence {
        return fmt.Errorf("%w: got %d, max seen %d", ErrStale, token, s.fence)
    }
    s.fence, s.value = token, v
    return nil
}

func main() {
    s := &Store{}
    fmt.Println(s.Put(34, "from B"))
    fmt.Println(s.Put(33, "from A"))
    fmt.Println(s.Put(34, "B again"))
}

Running it prints <nil>, then stale fencing token: got 33, max seen 34, then <nil>. The third call shows why the comparison is token < fence for rejection: the legitimate holder may issue many writes under the same token.

Persisting the fence

One production detail the toy examples skip: the maximum seen token must be durable and part of the same atomic update as the data. If you store the fence in memory in a gateway that restarts, max_seen resets to zero and a stale writer with an old token will be accepted. Store it next to the data it protects, in the same row, object metadata or transaction. Where the resource is a message queue or stream, put the token in the message and have consumers drop anything below the highest token they have processed for that key.

Failure-Mode Walk-throughs

Abstract arguments are easy to nod along with and hard to apply. These timelines show what each design does under specific faults. Times are illustrative, not measurements.

Scenario 1: GC pause with a naive Redis lock

Time	Event
t=0	A acquires `SET lock A NX PX 10000`
t=2	A checks it still holds the lock, passes, begins building the write
t=3	A’s runtime enters a 15-second stop-the-world pause
t=10	Redis expires the key
t=11	B acquires the lock and writes v2
t=18	A resumes and sends its write, which storage accepts

Result: lost update. The check at t=2 was correct and useless. Fix: storage fences on a token. A needs a monotonic token, which plain SET NX does not provide; an INCR counter on a single node can supply one, subject to the node’s own durability and failover behaviour.

Scenario 2: clock jump on a Redlock node

Following Kleppmann’s scenario: with five instances, A locks nodes 1, 2 and 3. Node 3’s clock jumps forward, so its key expires early. B then locks nodes 3, 4 and 5, a majority, while A still believes it holds a majority. Both proceed. antirez acknowledged that wall-clock jumps are a genuine hazard and said Redis should use monotonic time. Whether that closes the gap depends on whether you trust every deployed version and every operator’s NTP behaviour. Fix: a token. Redlock cannot generate one, so correctness workloads should use a consensus-backed source.

Scenario 3: network delay with etcd

A holds the etcd mutex with revision 120 on its lock key. A sends a write to storage, and the packet is delayed in a congested link for 12 seconds. A’s lease TTL is 10 seconds, but A’s keep-alives still succeed over a different path, so A’s session stays alive. Here the lock was never lost, and the write is legitimate even though it arrives late. Now flip it: keep-alives are also delayed, the lease expires, B acquires with revision 131 and writes with token 131. A’s delayed write arrives with token 120 and is rejected. The fence gave the right answer in both variants without anyone consulting a clock.

Scenario 4: ZooKeeper session expiry during a partition

A holds lock-0000000042. A is partitioned from the ensemble. After the session timeout, the ensemble expires A’s session and deletes the ephemeral node. B, watching its predecessor, sees lock-0000000042 vanish and acquires with lock-0000000043. When the partition heals, A’s library reports session expiry, but any write A already queued toward storage can still be in flight. If the storage layer checks czxid or the sequence number, A’s write fails. If it does not, it succeeds and corrupts state.

Scenario 5: Postgres failover

A holds a transaction-level advisory lock on the primary and is mid-transaction. The primary becomes unreachable, and an operator promotes a standby. The new primary has no record of A’s lock. B takes the “same” lock on the new primary and starts writing. If the old primary was merely partitioned and not shut down, A may still be writing to it. Fencing here means infrastructure fencing (STONITH-style isolation of the old primary) plus a token or version column that makes divergent writes detectable on reconciliation.

Scenario 6: the lease-renewal race

A subtle one: A renews its lease at t=9 of a 10-second TTL, but the renewal request is slow and reaches the lock service at t=10.2, after expiry and after B was granted the lock at t=10.1. The service must reject the renewal because the lease generation has changed. Services that renew by key name rather than by lease ID can accidentally extend B’s lease on A’s behalf. etcd avoids this by tying renewals to a lease ID; with homemade Redis locks, the renewal script must compare the stored random value before extending the TTL, exactly as release does.

The pattern across all six scenarios is the same. Timing hazards cannot be eliminated, so detect their consequences at the resource.

Trade-offs, Gotchas, and What Goes Wrong

Fencing needs a cooperating resource. The most common reason fencing is skipped is that the write target is a third-party API or a physical device. In that case consider a proxy that does the check, idempotency keys, or an explicit reconciliation job. Be honest in your design document that the lock is best-effort.

Token generation is itself a critical dependency. If your token source is a single Redis INCR and that node fails over to a replica that lagged, the counter can go backwards. A fence that regresses is worse than none, because it rejects the new legitimate holder or accepts a stale one. Prefer tokens derived from a consensus log.

Short TTLs cause churn, long TTLs cause outages. Leases that are too short produce spurious evictions during GC or network blips, causing lock flapping and wasted work. Leases too long delay recovery. Set the TTL from observed pause tails, not from intuition, and consider adding a renewal margin of several keep-alive intervals.

Fencing does not make the work safe, it makes stale writes visible. A rejected writer has done its computation for nothing and may have already triggered side effects such as emails, device commands or payments. Keep side effects after the fenced commit, make them idempotent, or route them through an outbox that is itself fenced.

Do not rely on Done() or isHeld() checks as safety. They shrink the window, which is valuable operationally, but cannot close it.

Lock-and-forget on release. Unlock paths that run on a timeout or error handler can delete a lock now owned by someone else. Always release by compare-and-delete with your own identity, or by lease ID in etcd, which Unlock handles for you.

Reentrancy and fairness. Many recipes are non-reentrant, and queue-based recipes (etcd, ZooKeeper) are fair while Redis polling locks are not. Fairness matters when a hot key could starve a waiter.

Over-locking. Teams reach for a lock because it is familiar, then discover that the hot lock is the throughput limit of the system. Partition the lock space per entity (per device, per work order) rather than using one global lock, and consider a single-writer-per-partition design with a message queue so that no lock is needed at all.

For IoT and digital twin pipelines, the ordering concern extends to events: if two writers produce updates to one twin, you also need a way to order the updates themselves. Our piece on hybrid logical clocks and vector clocks covers timestamps that give you a causal order without trusting raw wall clocks, a useful complement to fencing tokens.

Practical Recommendations

Start by classifying each lock in your system as efficiency or correctness, because the right tool differs. For efficiency, a single Redis SET NX PX with a safe release script is fine, and it is honest to document that it can fail. Spending a five-node Redlock deployment on an efficiency lock buys complexity without a corresponding safety gain, and for correctness it still lacks the token.

For correctness, pick one of three paths. If the critical section is a single-record read-modify-write, use optimistic concurrency with a version column and skip the lock. If you need ownership over a longer or multi-step operation, use etcd or ZooKeeper (or Consul sessions, if you already run it) and send the revision or czxid as a fencing token to every write. If you are already on PostgreSQL and the protected data is entirely inside it, use transaction-level advisory locks together with a fence column, and treat failover as a separate fencing problem at the infrastructure layer.

Whichever you choose, put the check where the data lives. A lock that exists only in the client’s head is a hint. A comparison inside the storage engine’s atomic update is a guarantee.

Checklist:

[ ] Each lock labelled efficiency or correctness in code comments and runbooks.
[ ] Correctness locks produce a monotonic token from a consensus log or a database sequence.
[ ] Every write path carries the token and the storage layer checks token >= max_seen atomically.
[ ] max_seen is persisted in the same atomic update as the data.
[ ] Lease TTL chosen from measured pause and latency tails, with keep-alive margin.
[ ] Client treats lost-session or disconnected events as “stop work now” (optimisation, not safety).
[ ] Release is compare-and-delete or lease-bound, never blind.
[ ] Side effects occur after the fenced commit, and are idempotent.
[ ] Fault-injection tests: SIGSTOP a holder past its lease and assert the stale write is rejected.
[ ] Lock space partitioned per entity to avoid a global hot lock.

Frequently Asked Questions

What is a fencing token in a distributed lock?

A fencing token is a number that increases every time a lock is granted. The holder includes it in every request to the protected resource, and the resource rejects any request whose token is lower than the highest it has already accepted. This prevents a client whose lease has expired, for example after a long GC pause, from overwriting data written by the new holder. The number should come from a linearizable source such as an etcd revision or a ZooKeeper zxid.

Is Redlock safe for distributed locking?

It depends on what the lock protects. For efficiency locks where an occasional double execution is harmless, it works and is more available than a single Redis node. For correctness locks, Martin Kleppmann argues it is unsafe because it relies on bounded delays, pauses and clock error and offers no monotonic fencing token. Salvatore Sanfilippo argues its timing assumptions are weaker than claimed and that a pause after the final check afflicts all expiring locks. Both agree resources must defend themselves for strict correctness.

Why do GC pauses break distributed locks?

A garbage collection pause can freeze a process between checking that it holds a lock and using the protected resource. If the pause outlasts the lease, the lock service hands the lock to another client while the first resumes with a stale belief. Re-checking the lease before writing cannot close the window because the pause can occur after the check. Only a resource-side check, such as a fencing token, makes the outcome safe.

Can I use an etcd or ZooKeeper lock without fencing tokens?

You can, and for efficiency locks that is reasonable. But a consensus-backed lock service only guarantees that it grants the lock to one session at a time; it cannot stop an expired holder from sending a late write. If correctness matters, send the etcd revision or ZooKeeper czxid with every write and verify it at the storage layer. The consensus protocol removes ambiguity about who the current owner is, which is what makes the token trustworthy.

Are PostgreSQL advisory locks good enough for distributed locking?

They are good when all protected data lives in the same PostgreSQL instance, because the lock and the data share fate: a dropped session releases the lock, and transaction-level locks end with the transaction. They fall short across failover, where a new primary does not know about locks from the old one, and for resources outside the database. Pair them with a fence or version column and ensure the old primary is fenced off.

When should I use optimistic concurrency instead of a lock?

Use optimistic concurrency when the critical section is a single read-modify-write on a record in a store supporting conditional updates, such as a SQL WHERE version = ?, an HTTP If-Match ETag, or a DynamoDB condition expression. It needs no lease or clock and is immune to pauses. Prefer a lock with fencing when the operation is long, spans several systems, or contention is so high that retries would thrash.