Hybrid Logical Clocks vs Vector Clocks: Event Ordering for Distributed IoT and Edge (2026)

Two sensors on opposite ends of a plant report a pressure spike and a valve closure. The gateway stamps both with its wall clock and the dashboard sorts by timestamp: the valve closed first, so the valve closure appears to have caused nothing. In reality the gateway’s clock had stepped backwards eleven seconds after a network time correction, and the cause landed after its effect. Every distributed IoT system eventually meets this bug, and “just sync the clocks” does not remove it.

Hybrid logical clocks (HLCs) are the pragmatic answer used by several production databases: a timestamp that stays close to physical time, never goes backwards, and respects causality in a fixed 64-bit or so footprint. Vector clocks give you exact causality detection but grow with the number of writers, and Lamport timestamps are tiny but unrelated to real time. This guide walks the full ladder, from why wall clocks lie to a working Python HLC, so you can choose a clock for your edge fleet on evidence.

What this covers: why physical clocks fail, Lamport and vector clocks, the HLC algorithm with a worked example and its drift bound, how databases use it, MQTT and OPC UA timestamp semantics, offline edge sync, last-writer-wins pitfalls, the CRDT relationship, code and failure modes.

Context and Background

Ordering events in a distributed system is the oldest problem in the field. Leslie Lamport’s 1978 paper “Time, Clocks, and the Ordering of Events in a Distributed System” reframed it: without a shared clock, the only ordering that is meaningful is the happened-before relation. Event A happened before B if they occurred on the same process in that order, if A is the sending of a message and B its receipt, or if the relation follows transitively. Events with no such chain are concurrent, and no honest system should claim to know which came first.

Physical time is the tempting shortcut. IoT stacks already carry timestamps: sensors stamp samples, gateways stamp batches, brokers stamp deliveries, historians stamp rows. The trouble is that these stamps come from different oscillators, disciplined by different protocols, with different failure behaviour. Our comparison of PTP (IEEE 1588), NTP and chrony for industrial time synchronization covers how tight you can make physical sync: sub-microsecond with hardware-assisted PTP on a well-engineered network, typically milliseconds on a good NTP path, and tens or hundreds of milliseconds (or worse) on a congested cellular backhaul. That is the best case. A battery-powered node that boots without a real-time clock battery may think it is 1970, or whatever its firmware default is, until its first sync.

The research response has three generations. Lamport clocks (1978) are scalar counters that respect causality in one direction. Vector clocks (Fidge and Mattern, 1988, independently) fix the other direction at the cost of one counter per participant. Hybrid logical clocks, introduced by Sandeep Kulkarni, Murat Demirbas and colleagues in the 2014 paper “Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases”, bolt a logical counter onto a physical timestamp so that the result is both causal and human-meaningful (the paper is hosted by the authors at the University at Buffalo).

A related track, which we cover elsewhere, avoids ordering questions by making merges commutative. If you are replicating digital-twin state across sites, read the CRDT guide for digital twin synchronization and offline-first edge replication alongside this one; the two techniques are complementary, and later sections show exactly where each fits.

Terminology used throughout: a process or node is any entity that produces events (a sensor, gateway, broker or cloud replica); an event is a local step, a send or a receive; skew is the instantaneous difference between two clocks; drift is how fast that difference grows; and causal ordering means that if A happened before B, every replica orders A before B.

Why Wall Clocks Lie, and What Each Logical Clock Buys You

Physical clocks fail in four distinct ways, and it is worth keeping them apart because each is fixed differently. First, offset: two clocks simply disagree by some amount, because their last sync was imperfect. Second, drift: crystal oscillators run fast or slow. A commodity quartz oscillator is commonly specified in the tens of parts per million; at 50 ppm a free-running clock gains or loses about 4.3 seconds per day (50 x 10^-6 x 86,400 s). That figure is illustrative arithmetic, not a measurement of any device, but it shows why a node that is offline for a week can be half a minute off.

Third, steps and slews: time daemons correct a clock either by gradually adjusting its rate (slewing) or by jumping it (stepping). A step backward makes time.time() non-monotonic, so two consecutive local events can receive timestamps in the wrong order. Fourth, leap seconds and wrong epochs: smearing versus stepping leap seconds differs by provider, and devices without a battery-backed real-time clock restart at a default date. Any of these can silently invert causality when you sort by wall time.

Lamport timestamps: causal in one direction

A Lamport clock is a single integer per process. On each local event increment it. On sending, attach the counter. On receiving a message with counter m, set the local counter to max(local, m) + 1. The guarantee is the clock condition: if A happened before B then L(A) < L(B).

The converse fails. L(A) < L(B) does not imply A happened before B; the events might be concurrent. Lamport timestamps therefore give you a consistent total order (break ties with a node ID) but cannot tell you whether two events are truly concurrent. They also carry no wall-clock meaning: a counter of 1,204,882 tells an operator nothing about when it happened, and a quiet node’s counter lags far behind a busy one’s, so “what happened around 14:00” is unanswerable.

Vector clocks: exact causality, linear cost

A vector clock keeps one counter per process: node i holds a vector V of length N. On a local event, increment V[i]. On send, attach the vector. On receive, take the element-wise maximum with the incoming vector and then increment own entry. Comparison is then exact: V(A) < V(B) if every entry is less than or equal and at least one is strictly less, which holds if and only if A happened before B. If neither vector dominates, the events are concurrent, and the system knows it.

This is exactly what a replicated store needs to detect conflicts, which is why Dynamo-style systems used vector clocks to surface sibling versions to the application. The price is metadata. With N writers, every message and every stored version carries N counters. For ten cloud replicas that is trivial. For a fleet of 50,000 field devices each writing, it is not, and clock entries for devices that have been decommissioned linger forever unless you prune them (pruning risks falsely ordering events, as the original Dynamo paper acknowledged).

A frequently used compromise is the version vector, which tracks one counter per replica rather than per event source, and the dotted version vector, which avoids false conflicts when many clients write through one server. The key design lever is to count replicas that coordinate writes (a handful of gateways or regional servers) rather than every device. Devices then attach their reads’ version context to writes, and the gateway mints the entry.

Hybrid logical clocks: bounded divergence from real time

An HLC timestamp is a pair (l, c). The component l is the largest physical time the node has seen, either its own or one carried in by a message. The component c is a small counter that breaks ties when l does not advance. Compare timestamps lexicographically: l first, then c. The authors show the pair can be packed into a single 64-bit integer by giving l the upper bits (for example, a millisecond-resolution physical time) and c the lower bits, which makes it a drop-in replacement for a wall-clock column.

What you gain over Lamport clocks is the same one-directional causality guarantee plus a bound: l never strays far from true physical time, so HLC values remain meaningful to humans and usable for “snapshot at time T” queries. What you give up relative to vector clocks is exact concurrency detection: two events with ordered HLCs may still be concurrent. HLC is a total order consistent with causality, not a causality detector.

Figure 1: Lamport, vector and hybrid logical clocks positioned by metadata size, causality guarantee and relationship to wall time.

The figure summarises the trade space: Lamport is constant-size and one-way causal, vector clocks are exact but linear, and HLC is constant-size, causally consistent and anchored to physical time. The sections that follow make that concrete.

The Hybrid Logical Clock Algorithm

Direct answer: a hybrid logical clock is a pair (l, c) where l tracks the maximum physical time seen and c is a counter for ties. On every local or send event, l becomes the max of its old value and the local wall clock, and c resets if l moved, otherwise increments. On receive, l becomes the max of local, message and wall time, with c adjusted to stay strictly greater than everything seen.

The update rules

Following the 2014 paper, let pt denote the node’s current physical clock reading. Each node maintains (l, c), initially (0, 0).

For a local event or send:

1. l_old = l
2. l = max(l_old, pt)
3. if l == l_old then c = c + 1, else c = 0
4. timestamp the event with (l, c)

For a receive of a message carrying (l_m, c_m):

1. l_old = l
2. l = max(l_old, l_m, pt)
3. if l == l_old == l_m then c = max(c, c_m) + 1
4. else if l == l_old then c = c + 1
5. else if l == l_m then c = c_m + 1
6. else c = 0
7. timestamp the receive with (l, c)

Reading the cases explains the design. If physical time has moved ahead of everything the node has seen, l jumps to pt and the counter resets: the timestamp is just wall time. If physical time has not advanced past what the node already knows (clock stepped back, or an incoming message came from a node whose clock is ahead), l stays put and c increments, preserving strict monotonicity without any lie about l beyond the amount already established by a peer.

A worked example

Take three nodes with millisecond physical clocks, deliberately skewed. Node A reads 100, node B reads 95 (five ms behind), node C reads 110. All start at (0, 0). (Illustrative values.)

1. A has a local event at pt = 100. l = max(0,100) = 100, changed, so c = 0. Timestamp (100, 0).
2. A sends a message. pt is still 100. l = max(100,100) = 100, unchanged, so c = 1. Message carries (100, 1).
3. B receives it at pt = 95. l = max(0, 100, 95) = 100 equals the message’s l, not B’s old l, so c = c_m + 1 = 2. B’s timestamp is (100, 2), even though its wall clock reads 95. The receive is correctly after the send.
4. B has a local event at pt = 96. l_old = 100, l = max(100, 96) = 100, unchanged, so c = 3. Timestamp (100, 3).
5. C has an unrelated event at pt = 110: (110, 0).
6. B sends to C at pt = 97: B’s state is still (100, 3); l stays 100, c = 4; message (100, 4). C receives at pt = 111: l = max(110, 100, 111) = 111, a fresh maximum equal to neither previous value, so c = 0. Timestamp (111, 0).

Notice what happened: B’s clock was five ms slow, yet B never produced a timestamp that went backwards, and the receive of A’s message sorted after the send. The cost was that B’s l ran ahead of its own wall clock by up to five ms for a while, until pt caught up. That “excess” l - pt is the quantity the paper bounds.

The drift bound

Let epsilon be the maximum clock skew between any two nodes’ physical clocks. The authors prove that l never exceeds the maximum physical clock reading of any node, so l - pt <= epsilon at every node. The counter c is bounded too: in a system with n nodes, and under the model where messages are processed at distinct physical instants, it is bounded in the order of n times the number of events within an epsilon window; in practice a few bits suffice. I will not quote the exact worst-case constant here; consult Theorem statements in the paper if you need it for sizing, and treat “16 bits for c” as an engineering choice rather than a proved value.

The consequence for engineers is the one that matters: an HLC’s l is never more than about the clock-sync error ahead of real time, so tighter physical sync directly gives a tighter and more useful HLC. HLC does not replace NTP or PTP; it degrades gracefully when they are imperfect, and it converts a synchronization error from a correctness bug into a bounded timestamp inflation.

Packing and resolution

A common packing uses 48 bits of milliseconds (good for thousands of years from the Unix epoch) and 16 bits for c, so that 65,536 events per millisecond per node can be ordered before the counter must stall. At a million events per second per node that headroom is roughly 65 times over, but it is a design check you should do for your own rates. Some systems instead keep l in nanoseconds and a separate 32-bit logical field (CockroachDB’s hlc timestamp has a wall time in nanoseconds plus a logical integer, to my knowledge; verify against current source before relying on the exact widths).

If the counter overflows the node must either wait for pt to advance one tick or fail the operation. Waiting is almost always right and costs at most one resolution unit of latency.

Hardening the receive path

Production implementations add two guards that the bare algorithm lacks. First, reject or clamp wildly future timestamps: if an incoming l_m exceeds local pt by more than a configured maximum offset, refuse the message instead of advancing l. Without the guard, a single device with a clock set to year 2099 poisons every node it talks to, because l is a max that never decreases. Second, persist the high-water mark across restarts, or wait out the maximum offset after a restart, so a rebooted node cannot reissue timestamps it already used.

Figure 2: The HLC update flow for local, send and receive events, including the guard that rejects timestamps beyond the maximum offset.

The flow shows a single decision tree: take the max of all three time sources, then pick the counter rule according to which source won. The guard on the receive branch is the difference between a textbook clock and a deployable one.

Deeper Analysis: Choosing a Clock for IoT and Edge Workloads

The right clock depends on three questions: how many writers produce events you must order, whether you need to detect concurrency or merely order consistently, and whether humans or time-range queries need the timestamp to mean something. The architecture below shows where each clock type typically lives in an edge-to-cloud pipeline.

Where HLC runs in a real pipeline

In a typical industrial deployment the leaf devices (PLCs, sensors, meters) are poor places for any logical clock. They are resource constrained, often stateless between samples, and their only meaningful clock is a sample timestamp. The sensible place to mint HLC timestamps is the first stateful hop: the edge gateway or edge broker. The gateway keeps one (l, c) pair, stamps each ingested batch or record, persists the high-water mark, and exchanges HLC values with cloud replicas on every sync.

Cloud-side stores (a time-series database, a document store, an event log) then receive records that already carry a total order consistent with causality across all gateways. Replication between regions piggybacks the HLC on each message, so a record created in region B after observing a record from region A is always ordered after it. This is the same property that makes HLC suitable for distributed transactions, and it is why systems such as CockroachDB and YugabyteDB use hybrid timestamps for multi-version concurrency control.

Figure 3: An edge-to-cloud pipeline. Devices carry source timestamps, gateways mint hybrid logical clock stamps, and cloud replicas merge using HLC order or version vectors.

The long-description version: devices emit raw samples with a source timestamp, the gateway wraps them with an HLC stamp and a replica ID, the broker transports them, and cloud replicas either order by HLC for last-writer-wins data or by version vector where concurrent updates must be surfaced.

How production databases use HLC

Several databases use hybrid clocks, with different safety envelopes. A reader should treat the following as a summary of public documentation and check current versions before relying on specifics.

CockroachDB uses HLC timestamps for MVCC versions and transaction ordering. It does not have a TrueTime-style bounded-error clock, so it declares a configurable maximum clock offset (500 ms by default in its documentation at the time of writing) and uses an uncertainty interval on reads: if a read finds a value whose timestamp is within the uncertainty window above the read timestamp, it cannot tell whether the write happened before the read, so the transaction restarts at a later timestamp. A node that detects its own clock is too far from the rest of the cluster shuts itself down rather than risk violating consistency; the docs describe the threshold as a fraction of the maximum offset. This is a deliberate choice: the failure is loud and local, not silent and global. Cockroach Labs’ own writing on clock management in CockroachDB explains the reasoning in more detail.

MongoDB uses a hybrid logical clock for its cluster time, a value exchanged between drivers, mongos routers and shards, and used to support causal consistency in sessions and ordering of oplog entries. Its Timestamp type is a 32-bit seconds-since-epoch field plus a 32-bit increment, which is the HLC shape with second-level resolution. Cluster time values are signed so that clients cannot arbitrarily advance the cluster’s clock, which is the same poisoned-future-timestamp problem described earlier and addressed with cryptography rather than an offset check.

YugabyteDB documents a “hybrid time” with a physical component and a logical component, plus a maximum clock skew parameter, following the same pattern. Other systems, including some CRDT libraries and local-first sync engines, use HLC-shaped timestamps as operation identifiers because they are compact, sortable and roughly time-like.

The shared lesson: HLC makes the correctness of a database robust to clock error, but not indifferent to it. Every one of these systems still needs NTP or PTP and still publishes a maximum offset, because large skew leads to long uncertainty windows, transaction restarts or refusals.

Choosing by decision matrix

Question	Lamport	Vector / version vector	Hybrid logical clock
Metadata per timestamp	One integer	O(writers)	One 64-bit-ish pair
Detects concurrent events	No	Yes	No
Causality guarantee	A before B implies L(A) < L(B)	Exact both directions	A before B implies HLC(A) < HLC(B)
Close to wall time	No	No	Yes, within clock-sync error
Supports “as of time T” queries	No	No	Yes
Handles unbounded writer set (fleet)	Yes	Poorly	Yes
Needs reasonable physical sync	No	No	Yes (bounds drift only)
Typical home	Teaching, simple logs	Dynamo-style stores, CRDT metadata	Distributed SQL, edge event stores

The matrix points to a pragmatic split. Use HLC for the ordering key of an event stream or of last-writer-wins registers, and use version vectors for the small set of objects where lost updates matter and the replica set is bounded (for example, a twin’s configuration, an asset’s maintenance record). Mixing is normal: an HLC can be embedded inside a version-vector entry as the “dot”.

Timestamp semantics in MQTT, Sparkplug and OPC UA

Protocols differ on who stamps what, and confusing the stamps is a frequent source of ordering bugs.

MQTT itself carries no timestamp. Neither MQTT 3.1.1 nor MQTT 5.0 defines a “time of publication” in the packet; MQTT 5 adds a Message Expiry Interval and user properties, but a broker is not required to stamp messages. Any time you see a timestamp on an MQTT message, it is in the payload or in a user property and its meaning is whatever the application defined. Ordering guarantees are per-topic-per-session at most, and they do not span publishers, so two publishers’ messages can arrive in either order with no way to tell which was first except by payload data.

Sparkplug B, a payload specification layered on MQTT, includes a timestamp in the payload and per metric, expressed as milliseconds since the Unix epoch; it identifies the time the value was sampled by the edge node, and it also includes a sequence number for detecting dropped messages in a session. The sequence number is a Lamport-flavoured per-session counter, not a global ordering.

OPC UA is the most explicit. A DataValue carries a SourceTimestamp (when the data was captured at the source, set by the device or server as close as possible to the sensor) and a ServerTimestamp (when the server last received or processed it), plus optional picosecond fields. A client reading history must decide which to trust: the source timestamp is the physically meaningful one, but it is only as accurate as the source’s clock, and a device that has never synchronised will produce confident nonsense. The server timestamp is trustworthy as a server-local ordering but reflects delivery delay. Our write-up on OPC UA PubSub over MQTT versus UADP shows how these fields travel in each transport.

The design rule that follows: keep the source timestamp as data, and add an HLC as ordering metadata. The source timestamp answers “when did the physical event occur, according to the sensor”; the HLC answers “in what order did the system observe and merge it”. Overwriting one with the other destroys information, and a historian that only keeps the HLC cannot tell you the real process time, while one that only keeps source time cannot order a replicated write.

Offline-First Edge Sync, Last-Writer-Wins and CRDTs

Edge sites go offline: cellular links drop, ships leave coverage, a factory network is segmented for maintenance. When the link returns, two sides that both accepted writes must merge. The ordering mechanism determines what the merge can promise.

The last-writer-wins trap

Last-writer-wins (LWW) resolves concurrent updates by keeping the one with the larger timestamp. With raw wall clocks this is dangerous in a precise way: whichever device has the fastest clock wins every conflict, regardless of which update was actually later. Suppose gateway A’s clock is 40 seconds ahead of gateway B’s. An operator at B changes a setpoint at true time T; a stale cached config push at A at true time T minus 30 seconds carries a stamp ahead of T. LWW keeps the stale value, and the operator’s change is silently lost.

HLC improves this in one important way and not in another. It guarantees that if the update at B observed the earlier state (by syncing first), its stamp exceeds that state’s stamp, so causally later writes beat causally earlier ones even if B’s wall clock is slow. It does not help when the two writes are genuinely concurrent: the winner is then decided by l, which is still physical time within the skew bound. Any LWW system silently drops one of two concurrent writes, and HLC only makes that choice less arbitrary and more explainable.

When to graduate beyond LWW

Use LWW with HLC for idempotent, overwrite-style data: the latest reading, the current status, a device shadow’s reported state. Overwriting is the correct semantic and losing a concurrent duplicate is harmless. Do not use it for accumulating or structured data: counters, lists of alarms, set membership, collaborative edits to a twin’s configuration. For these, losing one of two concurrent operations is a bug.

This is where CRDTs enter. A state-based or operation-based CRDT defines a merge that is commutative, associative and idempotent, so replicas converge regardless of delivery order and without a central arbiter. CRDTs still need identifiers and sometimes ordering: an LWW-register CRDT uses a timestamp (an HLC is the standard choice), a multi-value register uses version vectors to retain concurrent values, and sequence CRDTs use unique position identifiers derived from a replica ID and a counter. In short, the clock and the CRDT are layers of one design: the CRDT chooses what concurrent updates mean, and the clock supplies the order that makes “later” well defined. Our CRDT guide for digital twins covers the data-type side.

A worked edge-sync scenario

Consider (illustratively) 200 field gateways, each with a local store, syncing to three regional replicas. A pure vector-clock design would carry 203 counters in every version if each gateway is a writer, around 1.6 KB of clock metadata per record with 8-byte counters, which can exceed the payload for a small sensor reading. Restricting the vector to the three regional replicas shrinks it to 24 bytes, but then gateways cannot independently prove concurrency among themselves.

An HLC-only design uses 8 bytes per record plus a replica ID. For the setpoint-style data that dominates twin state, that is sufficient. For the small set of records where lost updates are unacceptable, add a three-entry version vector keyed by regional replica, and have gateways attach the version context they last read, so conflicts are detected at the region rather than at 200 endpoints. The result is roughly an 8-byte stamp on most records and 32 bytes on the few that need conflict detection.

Figure 4: Reconnect sequence for an offline gateway. The HLC is exchanged on handshake, the replica advances its clock, and concurrent writes are resolved by policy.

The sequence shows why the handshake exchange matters: by merging clocks before replaying queued writes, the replica guarantees that every record it accepts afterwards is stamped after everything the gateway had seen, which preserves causal order across the outage.

A Python Implementation of a Hybrid Logical Clock

The following is a compact, thread-safe HLC suitable for a gateway process. It packs into a 64-bit integer (48 bits of milliseconds, 16 bits of counter), enforces a maximum forward offset on received stamps, and never moves backward. It is a teaching implementation: it does not persist its high-water mark, which production code should.

import threading
import time

COUNTER_BITS = 16
COUNTER_MASK = (1 << COUNTER_BITS) - 1


class ClockDriftError(Exception):
    pass


class HybridLogicalClock:
    def __init__(self, max_offset_ms=500, now_ms=None):
        self._l = 0
        self._c = 0
        self._max_offset = max_offset_ms
        self._lock = threading.Lock()
        self._now = now_ms or (lambda: int(time.time() * 1000))

    def now(self):
        """Local event or send. Returns packed 64-bit timestamp."""
        with self._lock:
            pt = self._now()
            l_old = self._l
            self._l = max(l_old, pt)
            self._c = self._c + 1 if self._l == l_old else 0
            self._check_counter()
            return self._pack()

    def update(self, remote_packed):
        """Receive event. Merge a remote timestamp."""
        l_m, c_m = self.unpack(remote_packed)
        with self._lock:
            pt = self._now()
            if l_m - pt > self._max_offset:
                raise ClockDriftError(
                    f"remote ahead by {l_m - pt} ms, max {self._max_offset}"
                )
            l_old = self._l
            self._l = max(l_old, l_m, pt)
            if self._l == l_old and self._l == l_m:
                self._c = max(self._c, c_m) + 1
            elif self._l == l_old:
                self._c += 1
            elif self._l == l_m:
                self._c = c_m + 1
            else:
                self._c = 0
            self._check_counter()
            return self._pack()

    def _check_counter(self):
        if self._c > COUNTER_MASK:
            # Counter overflow: wait for physical time to advance.
            while self._now() <= self._l:
                time.sleep(0.0005)
            self._l = self._now()
            self._c = 0

    def _pack(self):
        return (self._l << COUNTER_BITS) | self._c

    @staticmethod
    def unpack(packed):
        return packed >> COUNTER_BITS, packed & COUNTER_MASK


if __name__ == "__main__":
    a = HybridLogicalClock(now_ms=lambda: 100)
    b = HybridLogicalClock(now_ms=lambda: 95)
    sent = a.now()
    recv = b.update(sent)
    assert recv > sent                       # receive sorts after send
    assert b.now() > recv                    # monotonic despite slow wall clock
    print(HybridLogicalClock.unpack(sent), HybridLogicalClock.unpack(recv))

Run it and the assertions reproduce steps 1 to 4 of the worked example: B’s receive carries l = 100 even though its wall clock reads 95, and B’s next local event keeps increasing the counter. Replace the injected now_ms with the real clock in production and add persistence of self._l on a timer, restoring it on boot (and waiting out the maximum offset if the persisted value is missing).

Two implementation notes. The packed integer compares correctly as an unsigned integer, so it can be stored in a database BIGINT (take care with signed 64-bit columns, which are fine until the year when 48 bits of milliseconds shifts into the sign bit, far beyond any practical horizon). And the lock matters: timestamps minted concurrently on one node must still be unique and increasing, which a mutex or atomic compare-and-swap guarantees.

Trade-offs, Gotchas, and What Goes Wrong

A single bad clock poisons the max. Because l is a maximum and never decreases, one device with a clock set far into the future drags every peer it talks to forward. The cost is permanent inflation: all later stamps carry the bad l until real time catches up, which could be years. Guard the receive path with a maximum-offset check, as in the code above, or sign the clock as MongoDB does for its cluster time. Quarantine the offender rather than merging its data blindly.

Restarts reissue timestamps. A gateway that reboots and reads a slightly earlier wall clock, with no persisted high-water mark, can mint stamps it has already used. Persist l periodically, and on boot either restore it or wait out the maximum offset before accepting writes. Skipping this is a classic source of duplicate-version bugs after power cycles in the field.

HLC does not detect concurrency. Teams sometimes read “causally consistent” as “I can tell if two updates conflicted”. You cannot. If an application must know that two edits were concurrent (so a human can reconcile), you need version vectors or a CRDT that keeps both values. HLC will cheerfully pick a winner.

Logical counters overflow under bursts. If a gateway ingests a large batch within one clock tick, c climbs quickly. With 16 counter bits and millisecond ticks, the ceiling is 65,536 stamps per millisecond per node. Bulk-load paths (replaying a buffered backlog after an outage) are where you meet it, so test that path explicitly and make the overflow behaviour a stall, not a wraparound.

Source timestamps are not trustworthy by default. A sensor that has not synced can report 1970 or the firmware build date. Validate source timestamps against the gateway’s receive time and flag implausible gaps instead of discarding or trusting them. Retain both values. When the offset is large and stable, record it as a device health signal, which is useful in itself: a persistent offset trend often precedes oscillator or battery failure.

Vector clock bloat and pruning. Per-device vectors grow without bound in fleets, and pruning old entries can create false ordering. If you must use them, restrict entries to coordinating replicas, track device context in a compact form, and set an explicit policy for retiring identities.

Skew still costs you. HLC bounds the consequence of skew, but a 5-second skew still means 5 seconds of timestamp inflation on the slow nodes, wide uncertainty windows in systems that use them, and “as of time T” queries that return slightly surprising snapshots. Invest in the synchronization layer discussed in the PTP and NTP article, and monitor offset as a first-class metric.

Network trust matters for the exchange. Clock values are metadata that influence global state. Transport them over authenticated, encrypted channels, since a forged far-future stamp is a denial-of-service lever. A mesh overlay such as those compared in our WireGuard, Tailscale, NetBird and Headscale edge-networking guide gives gateways an authenticated path for clock-bearing sync traffic.

Leap seconds and time zones. Always store UTC-based epoch values for l, never local time. Know whether your time source steps or smears leap seconds, and avoid mixing the two in one fleet; HLC survives a backward step of the wall clock, but a mixed fleet creates avoidable skew spikes.

Practical Recommendations

Start from the data, not the algorithm. Classify each class of record in your edge platform as overwrite-style (latest value wins), accumulating (merge required), or conflict-sensitive (a human must reconcile). Most telemetry and device-shadow state is overwrite-style, and an HLC ordering key with last-writer-wins is the right, cheap answer. Accumulating state belongs in CRDTs. Conflict-sensitive records, which are few, deserve version vectors scoped to a small set of coordinating replicas.

Mint HLC at the first stateful hop (gateway or edge broker), not on constrained leaf devices, and carry the source timestamp alongside it untouched. Treat the physical sync layer as a dependency with an SLO: measure offset between gateways and your time sources, alert before you approach the configured maximum offset, and choose that maximum deliberately rather than inheriting a default. Pair it with authenticated transport so clock-bearing messages cannot be forged.

Checklist:

• Classify each record type: overwrite, accumulate, or conflict-sensitive.
• Use an HLC as the ordering key for overwrite-style data; pack it into one 64-bit column.
• Keep source timestamp, server/receive timestamp and HLC as separate fields.
• Enforce a maximum forward offset on received stamps; reject or quarantine violators.
• Persist the HLC high-water mark and wait out the maximum offset after a restart if it is lost.
• Restrict version vectors to coordinating replicas, not individual devices.
• Test the backlog-replay path for counter overflow and stalls.
• Monitor clock offset per gateway and alert on trend, not just threshold.
• Store UTC epoch values only; document leap-second handling for the fleet.
• Run chaos tests that step clocks backwards and forwards by minutes and confirm ordering invariants.

Frequently Asked Questions

What is a hybrid logical clock?

A hybrid logical clock is a timestamp made of a physical-time component and a small logical counter. The physical part tracks the largest wall-clock time the node has seen, and the counter breaks ties when physical time does not advance. It preserves causal order like a Lamport clock while staying within the clock-synchronization error of real time, and it fits in a single 64-bit value, which makes it practical as a database ordering key.

How do hybrid logical clocks differ from vector clocks?

Vector clocks keep one counter per participant and can tell exactly whether two events are causally ordered or concurrent, but their size grows with the number of writers. Hybrid logical clocks are constant size and stay close to wall time, but they only guarantee that causally ordered events get ordered timestamps. They cannot prove two events were concurrent. Choose vectors when conflict detection matters and HLC when compact ordering matters.

Do I still need NTP or PTP if I use HLC?

Yes. An HLC remains correct, in the sense of respecting causality, even with bad clocks, but its usefulness depends on physical time being close to true time. The drift of l from real time is bounded by the clock skew between nodes, and systems built on HLC also use the skew bound to size uncertainty windows. Good synchronization makes HLC timestamps tighter and more meaningful, as covered in our PTP versus NTP guide.

Are Lamport timestamps still useful?

Yes, in narrow cases. A Lamport clock is the cheapest way to get a total order consistent with causality when you do not care about wall time, such as ordering entries in an internal log or building an operation ID. It cannot detect concurrency and cannot answer time-range queries, so for IoT telemetry that humans will query by time, an HLC is usually the better single-integer choice.

Can HLC replace CRDTs for edge sync?

No. HLC supplies an order, while a CRDT defines how concurrent updates combine. Using HLC with last-writer-wins is a simple register-style CRDT and is fine for overwrite data like latest status. For counters, sets or collaborative documents, you need data types whose merge preserves every update. In practice the two are layered, with the CRDT using an HLC as its timestamp source.

What does MQTT guarantee about message order and time?

MQTT defines no timestamp in the protocol, and its ordering guarantee is limited to messages from one client on one topic within a session at a given QoS level. Across publishers there is no global order. If you need ordering or time semantics, carry a source timestamp and, if useful, an HLC stamp in the payload or a user property, and decide explicitly which one consumers should trust.

Further Reading

• PTP IEEE 1588 vs NTP vs chrony for industrial time synchronization: how tight physical sync can get, and what bounds the HLC drift.
• CRDTs for digital twin synchronization and offline-first edge replication: the merge-semantics layer that sits on top of an ordering clock.
• OPC UA PubSub over MQTT vs UADP: how source and server timestamps travel on each transport.
• WireGuard vs Tailscale vs NetBird vs Headscale for edge fleet networking: authenticated transport for clock-bearing sync traffic.
• Kulkarni, Demirbas et al., Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases (2014): the original HLC paper.
• Cockroach Labs, Clock management in CockroachDB: maximum offset, uncertainty intervals and clock-skew handling in production.

By Riju — about