Digital Twin Event Time: Out-of-Order Sensor Data, Watermarks and Late Events

A digital twin is only as truthful as the order in which it believes things happened. A pump’s vibration spike that arrives after the shutdown command it caused will, in a naive pipeline, be applied last, and the twin will report a healthy running machine that is in fact stopped. Nothing crashed, no message was lost, and every dashboard is green. The twin is simply wrong about the past, and therefore wrong about the present.

This is the central problem of digital twin event time: separating when a measurement was taken from when the platform happened to see it. Gateways buffer, radios retry, brokers rebalance, and edge clocks drift. By the time a reading reaches the twin store it can be seconds, hours or days stale, and it will often arrive after newer readings.

This article builds the full alignment layer: clock and timestamp discipline, watermarks and allowed lateness, idempotent last-writer-by-event-time upserts, a bitemporal record for audit and replay, a runnable worked example, and the failure modes that bite in production.

What this covers: event time versus processing time, the sources of disorder, watermark mechanics in Apache Flink and Kafka Streams, twin state reconciliation, replay and backfill, and a decision checklist.

Context and Background

Industrial telemetry is rarely delivered in a straight line. A typical path runs from a sensor through a PLC or gateway, over a fieldbus or wireless link, into an edge buffer, across a WAN to a broker, and finally through a stream processor into the twin. Each hop can delay, batch, duplicate or reorder messages. Our article on MQTT store-and-forward at the edge shows how a gateway that loses its uplink for an hour will happily replay sixty minutes of data in a burst once the link returns. Those messages carry old timestamps but fresh arrival times.

Stream processing frameworks formalised the distinction years ago. Apache Flink’s documentation defines event time as the timestamp of when each event occurred on its producing device, and processing time as the wall-clock time of the machine executing the operator. Event-time results are deterministic and reproducible when historical data is reprocessed, at the cost of waiting for lateness. Processing-time results are fast and approximate and vary with arrival and processing speed (Apache Flink documentation on time and watermarks).

Most digital twin platforms inherit the processing-time habit by accident. A REST call such as “update property X to value Y” has no notion of when Y was true. Azure Digital Twins, for example, applies property updates in the order the service receives them; unless you carry the observation time as your own property, the platform has no way to know that an update is older than the one already stored. The same is true for any generic key-value twin store. The ordering problem is pushed to you.

Three adjacent topics deserve a pointer rather than a repeat. Timestamps are only as good as the clocks that wrote them, which is the subject of PTP versus NTP versus chrony. When no trustworthy wall clock exists, causal ordering needs different tools, covered in hybrid logical clocks and vector clocks. And the ingest side of historian data is covered in OPC UA historian to digital twin ingestion. This post takes a distinct angle: what the twin should do once timestamps exist and arrive in the wrong order.

My thesis, stated up front so you can argue with it: a twin should never store “the current value”. It should store the current best-known value for each point in event time, plus the processing-time at which it learned it. Every other design choice in this article follows from that sentence.

Why disorder is the default, not the exception

It is tempting to treat out-of-order delivery as a rare fault. In a fleet of battery-powered or cellular devices it is the normal case. Consider four independent sources of disorder:

Transport retries. MQTT with QoS 1 may redeliver a message, and a client that reconnects may publish queued messages after newer live ones.
Parallel paths. A device that publishes over both a primary and fallback link can deliver the same reading twice by different routes.
Partitioning. A Kafka topic guarantees order only within a partition. Two sensors on one machine keyed differently can land on different partitions and be consumed at different speeds.
Edge buffering. Store-and-forward gateways flush in bulk, so a backlog of old events is interleaved with live ones.

None of these is a bug. They are consequences of building reliable systems on unreliable links, and the correct response is to design the twin to tolerate them.

Reference Architecture for Event-Time Alignment

Direct answer: event-time alignment means every reading is stamped with the time it was measured, a stream processor tracks progress through a watermark that says “events older than this are complete”, and the twin store accepts updates idempotently, keyed by entity, property and event time. Late data is either absorbed within a lateness window, corrected through a retraction path, or diverted for backfill.

Figure 1: Reference architecture for digital twin event time alignment, from sensor stamping through watermarking to a bitemporal twin store.

The diagram shows five stages. Stamping happens at the edge, where each reading gets an event timestamp plus a sequence number. The transport and broker tier preserves per-device ordering as far as it can but is allowed to reorder. A stream processor assigns watermarks and groups data into event-time windows. A reconciliation function writes into the twin store with deterministic conflict resolution. A late-event path handles everything the watermark has already passed. Each stage owns one responsibility, and that separation is the point: ordering logic buried inside five different microservices is the classic way these systems become unfixable.

Stage 1: stamp at the source, and say which clock you used

An event needs at least three fields beyond the value: an entity identifier, an event timestamp and a monotonic per-source sequence number. The timestamp should be in UTC with millisecond or better resolution, and the schema should record the clock’s quality. A reading stamped by a GPS-disciplined clock and one stamped by a free-running microcontroller RTC are not equally trustworthy, and downstream logic can use that.

Where the device has no reliable clock, stamp at the nearest trustworthy point, usually the gateway, and record that the timestamp is “gateway-observed”. The error you introduce is the sensor-to-gateway latency, which for wired fieldbuses is typically small compared with the WAN delay you are avoiding. Never let the cloud ingress timestamp silently stand in for event time; that is processing time wearing a disguise.

The sequence number matters more than people expect. Two readings from one sensor in the same millisecond cannot be ordered by timestamp alone. A per-source counter that increments on each reading gives a total order for that source, lets you detect gaps (lost data), and lets you detect duplicates cheaply. Combine them into an idempotency key: (entity_id, property, source_seq) or (entity_id, property, event_ts).

Stage 2: choose the time domain per use case

Not every consumer wants event time. A live operator HMI showing “last seen” wants processing time; a quality report wants event time; a billing meter wants event time with strict closure rules. Provide both on every record and let the consumer pick. The twin’s schema should therefore carry event_time (when it happened), ingest_time (when the platform first saw it) and optionally commit_time (when it became visible in the twin). We return to why three clocks, not two, in the bitemporal section.

Stage 3: watermarks as the progress contract

A watermark is a statement about completeness, not about time. Flink’s documentation describes it as a marker flowing through the stream with timestamp t, declaring that all events with timestamp up to t have already arrived, so windows ending at or before t may close and emit results. In a parallel pipeline an operator’s event time is the minimum watermark across its inputs (Apache Flink documentation).

The watermark is a heuristic you choose, not a fact you discover. A bounded-out-of-orderness watermark says: “no event will be more than D older than the newest event I have seen.” If your network violates that, events become late. Choosing D is therefore the central tuning decision, and the next section treats it quantitatively.

Stage 4: reconcile into the twin store deterministically

Reconciliation is the step that makes the twin consistent. It must satisfy three properties: idempotent (applying the same event twice changes nothing), commutative for the same property (the final state does not depend on arrival order) and monotonic in event time (an older event never overwrites a newer one). A last-writer-wins register keyed on event time satisfies all three for scalar properties. We cover its limits for composite state later.

Stage 5: a first-class path for late data

Events behind the watermark are not garbage. They are information the pipeline had given up waiting for. Route them to a late-event path that can (a) correct previously emitted aggregates, (b) update the point-in-time history, and (c) trigger a re-evaluation of derived state such as alerts. If the answer to “what happens to an event that arrives an hour late?” is “it is logged and dropped”, then your twin has an undocumented accuracy ceiling.

Watermarks, Lateness and the Cost of Waiting

Watermarks trade completeness for latency, and there is no setting that gives you both. A small out-of-orderness bound produces fast results and many late events. A large bound produces accurate results and a twin that lags reality by the bound. The right value is a property of your network and your use case, and it should be measured rather than guessed.

Figure 2: Watermark advance, allowed lateness and the side output path. Events behind the watermark but inside allowed lateness refine results; events beyond it divert to backfill.

The figure follows one window as the watermark advances. While the watermark is below the window end, events are buffered. When it passes the window end the window fires and emits a result. During the allowed-lateness period the window state is retained, so a late event triggers a refined, updated result. After that, the state is dropped and anything further is diverted to a side output for offline correction.

How Flink defines the watermark

Flink’s bounded-out-of-orderness generator emits a watermark equal to the highest timestamp seen so far, minus the configured bound, minus one millisecond (Apache Flink documentation, generating watermarks). In code, the strategy is built with WatermarkStrategy.forBoundedOutOfOrderness(Duration), and timestamps and watermarks are expressed in milliseconds since the Java epoch. If the stream already carries timestamps, as Kafka and Kinesis records do, a timestamp assigner is optional; if you supply one it overwrites the existing timestamp.

Three details in the Flink documentation matter for twins. First, with a Kafka source, watermarks are generated inside the consumer per partition and then merged, the way they are on a stream shuffle, so per-partition ordering can yield tight overall watermarks. Second, withIdleness(Duration) marks an input idle after a period with no events, so a silent partition stops holding back the global watermark. Third, watermark alignment, configured through withWatermarkAlignment(group, maxDrift, updateInterval), bounds how far one source can run ahead of the slowest in its group; the documentation notes it works with FLIP-27 sources and that split-level alignment is supported from Flink 1.17.

Idleness deserves special attention in twin workloads. A decommissioned machine, a night-shift line or a device in deep sleep produces no events for hours. Without idleness handling, that one silent partition pins the minimum watermark and every window on every other machine stops closing. With idleness handling, the silent source is excluded from the minimum, but if it wakes and sends old data, that data may be late. That is the correct trade, but you must know you made it.

Allowed lateness and what it costs

Flink’s allowedLateness keeps a window’s state alive for an extra period after the watermark passes the window end, and late events inside that period trigger a recomputation of the window result. Larger values improve correctness at the price of more state and more result revisions (Apache Flink documentation). By default, events whose timestamp is at or below the current watermark once the window has fired are dropped, which is why you should explicitly attach a side output for them rather than rely on the default.

The downstream consequence is that your twin store will receive multiple results for the same window: a first provisional one and subsequent refinements. That is only safe if the sink is idempotent and window results are keyed so that the latest refinement replaces the earlier one. Treat the stream as a changelog of upserts, not a log of final facts. This is exactly the discipline of Kafka exactly-once semantics, where transactional writes make the repeated emission safe.

Kafka Streams and Azure Stream Analytics use different vocabulary

Kafka Streams expresses the same trade-off as a grace period on windowed operations. Records are processed in offset order, and a window accepts out-of-order records until stream time, the maximum timestamp observed so far, passes the window end plus the grace period; later records are dropped. I am describing this from the Kafka Streams model rather than quoting a specific version’s defaults, which have changed across releases, so check the grace-period default for the version you deploy before relying on it.

Azure Stream Analytics exposes two explicit policies. Its documentation lists a default late arrival tolerance of 5 seconds and a default out-of-order tolerance of 0 seconds, applied only when you process by event time using TIMESTAMP BY; by default the job uses arrival time. A late event is either dropped or has its event time adjusted to arrival time minus the late-arrival value, and an out-of-order event beyond tolerance is adjusted to the tolerance boundary. Raising the out-of-order tolerance delays the first output by at least that value, and a partition that receives no events can delay output by the late-arrival value (Azure Stream Analytics event ordering documentation).

Notice the philosophical difference. Flink and Kafka Streams default to dropping or side-routing late data, preserving the timestamp’s truth. Azure Stream Analytics offers to adjust the timestamp, which keeps the event in the aggregate but quietly lies about when it happened. For a twin that supports audit, adjusting is a last resort. If you adopt it, record the original timestamp alongside.

Sizing the bound from data, not intuition

Pick D from an empirical distribution of lateness, defined per event as ingest_time - event_time. Collect a week of production telemetry, compute percentiles, and decide what fraction of events you are willing to treat as late. The numbers below are illustrative and are not measurements from any real fleet; they show the shape of the decision.

Percentile of lateness	Illustrative value	If D equals this value
p50	0.4 s	Half of events are late. Unusable.
p95	3 s	5% late; fast; suitable for live dashboards
p99	12 s	1% late; reasonable for control-adjacent twins
p99.9	90 s	0.1% late; good for quality and energy reports
p99.99	20 min	Rare late events; heavy state and latency

Notice the knee. Moving from p95 to p99 costs 9 seconds of added latency and removes four-fifths of the late events. Moving from p99.9 to p99.99 costs eighteen and a half minutes for a thousandth of the data. This is why a single watermark rarely suits a twin, and why the next pattern exists.

The two-tier pattern: fast path and correction path

Run two tiers over the same stream. The fast path uses a tight bound (say p99) and feeds live state, alerts and the HMI. The correction path uses a generous allowed lateness or a periodic batch recomputation and feeds history, reports and model training. The twin exposes both: a current view that is fast and sometimes revised, and a settled view that is slower and rarely changes. Consumers choose.

This mirrors the lambda-style split familiar from lakehouse designs, but without duplicating logic if you write the aggregation once and run it with two configurations. If you use a streaming SQL engine, the same applies; see RisingWave vs Materialize vs Flink for how those engines express watermarks and emit-on-close semantics.

Reconciling Twin State: Idempotent Upserts and Conflict Rules

Figure 3: Sequence of an out-of-order update arriving at the twin store and being rejected, merged or recorded as a correction.

Watermarks decide what the stream processor emits. They do not by themselves guarantee the twin ends up right. The final guard is the write itself, and it must be safe against duplicates, reordering and retries.

The last-writer-by-event-time register

For a scalar property such as temperature, the rule is simple: store the value with its event time, and accept a new write only if its event time is greater than the stored one. If equal, break ties with the source sequence number, then with a deterministic rule such as lexicographic order of a writer identifier. This is a last-writer-wins register with a logical clock supplied by the physical measurement. It is idempotent, because replaying the same event is rejected as “not newer”. It is commutative, because any permutation of arrivals converges on the maximum.

In a relational twin store this is one statement. The example below uses PostgreSQL syntax and a conditional update in the conflict clause:

-- twin_property: one row per (entity, property), holds the current best-known value
CREATE TABLE twin_property (
  entity_id   text        NOT NULL,
  property    text        NOT NULL,
  value       jsonb       NOT NULL,
  event_time  timestamptz NOT NULL,
  source_seq  bigint      NOT NULL,
  ingest_time timestamptz NOT NULL,
  PRIMARY KEY (entity_id, property)
);

INSERT INTO twin_property AS t
  (entity_id, property, value, event_time, source_seq, ingest_time)
VALUES ($1, $2, $3, $4, $5, now())
ON CONFLICT (entity_id, property) DO UPDATE
SET value = EXCLUDED.value,
    event_time = EXCLUDED.event_time,
    source_seq = EXCLUDED.source_seq,
    ingest_time = EXCLUDED.ingest_time
WHERE (EXCLUDED.event_time, EXCLUDED.source_seq)
    > (t.event_time, t.source_seq);

The WHERE clause on the conflict branch makes the update conditional, so an older event leaves the row untouched, and the row-value comparison handles the tie-break. A document store can do the same with a conditional write on an attribute; a graph twin can use a compare-and-set on the node property. The principle is portable: the database, not the application, enforces monotonicity, because the application has several writers.

When last-writer-wins is not enough

Scalar registers fail in three cases. First, composite state: an AGV’s position, heading and speed are one physical fact. If three independent upserts each carry their own timestamp, a reader can see a position from t=10, a heading from t=12 and a speed from t=11, a state that never existed. Write composite readings as one atomic record with a single event time, or version the whole aggregate.

Second, accumulating state such as a counter or energy meter. Last-writer-wins on a cumulative counter is correct; on an incremental delta stream it is wrong, because dropping a late delta permanently loses energy. For deltas, convert to cumulative readings at the edge, or store deltas as an append-only event log and compute the sum.

Third, multi-writer state where two systems legitimately update the same property, for example a PLC and an operator override. Event-time ordering cannot arbitrate intent. Use separate properties with an explicit precedence rule, or a mergeable data type; our piece on CRDTs for digital twin synchronization covers designs where several replicas must converge without coordination.

Estimation versus ordering

There is a subtler point. A twin is often not a register of raw readings but an estimate of hidden state, produced by a filter. A Kalman filter assumes measurements arrive in time order; feeding it a late measurement naively applies it at the wrong time. The correct approach is either to buffer behind a watermark so the filter sees ordered input, or to use an out-of-sequence-measurement update that rewinds to the measurement’s time. This is discussed in the state estimation article on Kalman and particle filters. The watermark bound D then becomes a direct input to estimator latency, which is why it should be chosen with the control engineers, not alone by the data team.

Bitemporal Twins, Replay and Backfill

A last-writer-wins register answers “what is the best-known value now?” It cannot answer “what did the twin believe at 14:00 yesterday?” or “what was actually true at 14:00 yesterday, as we know it today?” Those are different questions, and incident reviews, warranty claims and regulated manufacturing need both.

Figure 4: Bitemporal storage. Valid time is event time; transaction time is when the twin learned the fact. Replay reads the log to rebuild either view.

Two time axes, not one

A bitemporal model keeps two timestamps per fact. Valid time is when the fact was true in the physical world, which is your event time. Transaction time is when the twin recorded it, which is your commit or ingest time. Late data adds a new row with an old valid time and a new transaction time; it does not overwrite anything. A query “as of transaction time T, what was the state at valid time V?” reproduces what the twin showed an operator at T. A query “as of now, state at V” gives the corrected truth.

The financial-audit world solved this long ago, and the pattern transfers directly. See event sourcing and bitemporal data for audit trails for the ledger-oriented treatment. For twins, the key practical benefit is explainability: when an alert fired at 02:13 on data that was later corrected, you can show precisely what the system knew.

SQL:2011 added standardised temporal table syntax, and several relational engines implement parts of it, but support differs by product and I would verify the specific features of your database before relying on them. A pragmatic implementation needs no special engine: an append-only twin_event table with (entity_id, property, valid_from, recorded_at, value) plus a materialised current-state table built by the idempotent upsert shown earlier.

The log is the source of truth

Make the immutable event log, a Kafka topic with long retention or an object-store lake, the system of record, and treat the twin store as a derived view. This gives you three capabilities that otherwise cost a rewrite. You can replay the log through a corrected pipeline after fixing a unit bug or a bad calibration. You can backfill a newly modelled property by running historical events through the new logic. And you can rebuild a corrupted twin store without asking the plants to resend data.

Replay has its own ordering hazards. When you re-read a long period at full speed, event-time progress is driven by the data, so a deterministic event-time job produces the same output as the original run, which is precisely the benefit Flink’s documentation attributes to event time. A job that mixed in processing-time logic, for example now() calls in a UDF or processing-time timers, will not reproduce. Audit your pipeline for hidden wall-clock dependence before you depend on replay. The related table-format question, where the replayed history lands, is covered in Iceberg versus Delta versus Hudi.

Replay also interacts with watermark alignment. Without alignment, a replay that reads several partitions of unequal depth lets one source race far ahead in event time, so the others’ events look late and are dropped. Flink’s watermark alignment feature exists for this reason: it bounds the drift between sources in a group (Apache Flink documentation). For backfills, set a small maximum drift, or replay with a very large lateness bound, then compare.

Late-event policy table

A written policy prevents arguments during an incident. Here is a reasonable default for an industrial twin; adapt the thresholds.

Lateness of event	Action	Twin effect
Inside the watermark bound	Normal processing	Current state updated in order
Behind watermark, inside allowed lateness	Refine window result	Provisional result revised; upsert is idempotent
Beyond allowed lateness, within retention	Side output to correction job	History updated; current state only if newer
Beyond retention or schema-invalid	Dead-letter queue	Logged with reason; operator review

Worked Example: A Pump Twin Under Disorder

This example is deliberately small and its numbers are illustrative, not measured. A pump publishes pressure_kpa once per second. A gateway buffers during an uplink outage and then flushes. The twin needs a 10-second tumbling average per pump, tolerating up to 20 seconds of disorder, with one extra minute of allowed lateness, and it must route anything later to a correction topic.

The Flink job

import java.time.Duration;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.util.OutputTag;

public class PumpTwinJob {
  static final OutputTag<Reading> LATE = new OutputTag<>("late-readings"){};

  public static void main(String[] args) throws Exception {
    var env = org.apache.flink.streaming.api.environment
        .StreamExecutionEnvironment.getExecutionEnvironment();

    WatermarkStrategy<Reading> wm = WatermarkStrategy
        .<Reading>forBoundedOutOfOrderness(Duration.ofSeconds(20))
        .withTimestampAssigner((r, ts) -> r.eventTimeMs)
        .withIdleness(Duration.ofMinutes(1));

    DataStream<Reading> src = env.fromSource(kafkaSource(), wm, "pump-telemetry");

    var windowed = src
        .keyBy(r -> r.pumpId)
        .window(TumblingEventTimeWindows.of(Time.seconds(10)))
        .allowedLateness(Time.minutes(1))
        .sideOutputLateData(LATE)
        .aggregate(new AvgPressure());

    windowed.sinkTo(idempotentTwinSink());          // upsert by pump+window start
    windowed.getSideOutput(LATE).sinkTo(correctionTopicSink());
    env.execute("pump-twin");
  }
}

Every call above is a documented Flink DataStream API element: the bounded-out-of-orderness strategy, the timestamp assigner, idleness, allowedLateness and sideOutputLateData. The two helper sources and sinks are placeholders you must provide. The crucial design choice is the idempotent sink keyed by (pump_id, window_start): because allowedLateness re-fires a window with a refined value, a non-idempotent sink would double count.

Walking the timeline

Suppose the watermark bound is 20 seconds and a window covers 12:00:00 to 12:00:10. At 12:00:35 wall-clock, the highest event time seen is 12:00:34, so the watermark is about 12:00:14, which is past the window end, and the window fires a provisional average from the readings it has. At 12:01:00 the uplink returns and the gateway flushes readings stamped 12:00:03 to 12:00:09. The watermark is already beyond them, but the window state is retained for one more minute of watermark time, so each late reading triggers a refined average. The twin’s upsert for window 12:00:00 is overwritten with each refinement.

If the gateway had instead flushed at 12:03:00, the watermark would have passed 12:01:10, the window state would be gone, and the readings would flow to the late-readings side output. The correction job then reads that topic, recomputes the affected window from the log, and upserts the corrected value into both the history and, if the window is the latest for that pump, the current state. At no point did the system silently discard data.

Handling the duplicate flush

Gateways sometimes resend a batch after a timeout. The same reading arrives twice with an identical (pump_id, source_seq). In the window aggregate this would be a double count. The fix is to de-duplicate before aggregation, keeping a keyed state of recent sequence numbers per pump with a time-to-live equal to your lateness horizon. State is bounded because you only need to remember sequences that could still plausibly arrive. This is the stream-processing analogue of idempotency keys in payment systems; see idempotency keys and exactly-once retries for the underlying pattern.

Trade-offs, Gotchas, and What Goes Wrong

Clock skew corrupts everything silently. If a gateway’s clock is five minutes fast, its events look five minutes in the future. A bounded-out-of-orderness watermark advances to that future timestamp and then treats every correctly stamped event from other devices as late. One bad clock can poison a shared watermark for a whole partition. Defend with clock-quality metadata, a sanity check rejecting events more than a threshold ahead of ingest time, and per-device or per-partition watermarks.

Idle sources stall or distort time. Covered earlier, but it bears repeating that both extremes hurt: no idleness handling freezes output, aggressive idleness handling drops late wake-up data. Choose per fleet segment, since a duty-cycled sensor and a continuous one need different settings.

State growth from generous lateness. Allowed lateness keeps window state alive for every key. For a million devices with one-minute windows and an hour of lateness, state size scales with devices times retained windows. Estimate it before shipping, and note that the cost multiplies if you hold raw events rather than aggregates.

Non-idempotent sinks. The most common production defect I see described is a pipeline that is event-time correct but writes through an INSERT or an increment. A refinement or a retry then double counts. Every sink in the twin path should be an upsert or a transactional write.

Late data that changes decisions already taken. If an alert fired or a control action was issued on a provisional value, a later correction may contradict it. The twin cannot un-send the setpoint. Decide in advance whether corrections generate retractions, annotations or nothing, and record the knowledge state in the transaction-time column so the audit trail shows why the action was reasonable.

Mixing event time and processing time in one query. Joining a stream windowed by event time against a table refreshed by processing time yields results that vary run to run. Make time semantics explicit in every join, and test by replaying the same log twice and diffing outputs.

Over-trusting the watermark. A watermark is a heuristic. The statement “all events up to t have arrived” is only as good as your bound. Monitor the late-event rate as a first-class SLO; a rising rate is an early warning that the network or a gateway has changed, long before dashboards look wrong.

Practical Recommendations

Start by measuring. Instrument ingest_time - event_time per source for a week before choosing any watermark bound, and keep that metric running afterwards. Pick the fast-path bound near a high percentile of that distribution and a longer allowed lateness for the correction path, then write the late-event policy down in the table format shown above.

Stamp events at the source with UTC time, a quality flag and a per-source sequence number, and make event time a required schema field. Keep the immutable log as the source of truth, and make the twin store a derived view built with conditional upserts so the database enforces monotonicity. Store both valid time and transaction time, because the day you need one of them is the day you cannot reconstruct it.

Test disorder deliberately. Build a replay harness that shuffles a recorded day within a range of delays, injects duplicates and drops a gateway’s clock forward by minutes, then asserts the twin converges to the same state each run.

A short checklist:

Every event carries entity, property, event time, sequence number and clock quality.
Watermark bound derived from measured lateness percentiles, with idleness configured per source type.
Late events routed to a side output, never silently dropped.
All sinks idempotent; composite state written atomically.
Valid time and transaction time both stored; log retained long enough to replay.
Late-event rate, watermark lag and rejected-future-timestamp counts monitored.
Deterministic-replay test run before every pipeline change.

Disclaimer for related regulated uses: this article describes engineering patterns only and is not legal or compliance advice.

Frequently Asked Questions

What is event time in a digital twin?

Event time is the moment a measurement or state change actually occurred in the physical asset, as opposed to the moment the platform received or processed it. Digital twin event time lets the twin order readings correctly even when the network delays or reorders them. It is carried in the data as a timestamp, ideally UTC with millisecond resolution, and drives windows, comparisons and history. Processing time, by contrast, depends on delivery and load.

How do you handle out-of-order sensor data?

Stamp every reading with an event time and sequence number, then use a stream processor with a watermark that tolerates a bounded amount of disorder. Within the bound, events are reordered into the correct windows. At the store, apply conditional upserts that only accept newer event times, so older readings never overwrite newer ones. Events beyond the bound go to a side output for correction rather than being dropped.

What is a watermark in stream processing?

A watermark is a progress marker that travels through a stream and asserts that events with timestamps up to a given value have already arrived, so event-time windows ending before it can close and emit results. Apache Flink’s bounded-out-of-orderness strategy sets it to the highest timestamp seen minus a configured delay. It is a heuristic, so events older than the watermark can still arrive and are treated as late.

How long should allowed lateness be?

Base it on the measured distribution of ingest delay, not a guess. A tight bound near the 99th percentile serves live state, while a longer allowed lateness or periodic recomputation serves history and reporting. Longer lateness improves accuracy but increases state size and causes more result revisions. Many teams run two tiers: a fast provisional path and a slower correction path, and expose both views to consumers.

Should a digital twin store both event time and ingest time?

Yes. Event time is when something was true; ingest or transaction time is when the twin learned it. Keeping both gives a bitemporal record, so you can answer what the twin believed at a moment in the past and what was actually true at that moment as known today. Without both, late corrections overwrite history, and incident reviews cannot reconstruct why a decision was taken.

Can late-arriving events be applied after a window has closed?

Yes, within limits. If allowed lateness is configured, a late event inside that period triggers a refined result for the closed window. Beyond it, the window state is discarded, so the event should be sent to a side output and applied by a correction job that recomputes from the immutable log. The twin store must accept these refinements idempotently, keyed by entity and window, so repeats do not double count.