MQTT Store-and-Forward at the Edge: Offline Buffering, Broker Bridging and QoS Under Disconnects (2026)
Most MQTT deployments are designed on the assumption that the network is up, and then tested only in the demo room where it is. The factory reality is different: a cellular modem reboots, a firewall change drops the uplink for forty minutes, a ship leaves coverage for three days. Whether your telemetry survives that gap depends less on the QoS number you picked than on a handful of decisions about where messages wait, how long they are allowed to wait, and what happens when the link returns and every device tries to talk at once.
MQTT store and forward is the discipline of making those decisions explicitly. The protocol gives you building blocks: acknowledged delivery levels, persistent sessions, message expiry, and broker bridges. None of them, on its own, is a durable outage buffer, and several of them fail silently at default settings. A broker queue that quietly stops at 1,000 messages, a client library that keeps unsent messages only in RAM, and a bridge that replays a retained flood on reconnect are all ordinary behaviours that cost real data.
What this covers: what each QoS level promises across a disconnect, how MQTT 5 session state and expiry work, the real limits of broker queues and client libraries, three edge gateway buffer designs with sizing math, bridging with Mosquitto and EMQX, ordering and duplicate handling, backfill versus live data after reconnect, Sparkplug birth and death semantics, observability, a runnable Python spool-and-forward example, and the failure modes that bite in production.
Context and Background
MQTT was designed for constrained, unreliable links, and its reliability features reflect that heritage. The OASIS MQTT 5.0 specification defines three Quality of Service levels, a session mechanism that lets a broker remember a client across connections, and, since version 5, explicit expiry controls for both sessions and individual messages. What it does not define is a general-purpose durable queue with capacity guarantees. Brokers are free to choose how many messages they hold for an offline subscriber and how they persist them. Mosquitto’s documentation, for example, states a per-client default of 1,000 queued QoS 1 and 2 messages and calls the option to queue QoS 0 messages “non-standard”.
That gap is where store and forward lives. In the industrial setting, the pattern has a recognisable shape: sensors and PLCs talk to an edge gateway over a local protocol such as OPC UA or Modbus, the gateway normalises the data and publishes it to a local or central broker, and a long-haul link carries it to a cloud or data-centre platform. The outage you care about is usually on the long-haul link, not the local one. The edge component must therefore keep accepting data while the uplink is dead, and then deliver it later without losing, duplicating or misordering what matters.
Two incumbent approaches dominate. The first is client-side buffering: the publishing library holds unacknowledged messages, perhaps in memory, perhaps in a file store, and retries on reconnect. The second is broker-side buffering: a local broker at the edge receives everything from local clients and a bridge forwards it upstream, queuing while the bridge is down. A third family, covered later, puts a purpose-built durable queue (SQLite, a write-ahead log, an embedded queue library) between acquisition and publishing. If you are weighing a different transport for the edge altogether, our comparison of NATS JetStream and Kafka for edge IIoT telemetry covers brokers with native replay, which changes the buffering calculus considerably.
One framing point matters throughout. MQTT delivery guarantees are hop by hop. HiveMQ’s widely cited explainer on QoS misconceptions makes this point directly: a publisher’s QoS governs only the publisher-to-broker leg, and the broker then forwards to each subscriber at that subscriber’s granted QoS, which can be lower. A QoS 2 publish reaching a QoS 0 subscription is delivered at most once. Any design that treats the QoS number as an end-to-end contract is already broken before the first outage.
Reference Architecture: Where Messages Wait During an Outage
The short answer: a message can wait in exactly four places during an uplink outage: the publishing client’s in-flight window, the client library’s outgoing queue, a local broker’s session queue, or an application-level durable spool. Only the last two survive a process restart by design, and only the last gives you capacity and policy you control.
Figure 1: Reference architecture for MQTT store and forward. Data is acquired, written to a durable spool, then published by a forwarder; a local broker with a bridge is an optional second buffer.
Figure 1 shows the layered layout this post argues for. Southbound devices feed an acquisition process at the edge. The acquisition process writes each reading to a durable spool before anything else happens. A forwarder reads from the spool, publishes to the upstream broker, and deletes a record only after the broker acknowledges it. A local broker with a bridge can sit alongside, serving local consumers such as a dashboard or a control loop that must keep working while the uplink is down. This is the fundamental asymmetry of store and forward: local consumers need low latency and availability, while the upstream consumer needs completeness.
Why the spool comes first
Putting the durable write before the network publish inverts the usual instinct, which is to publish and buffer on failure. The failure-first approach has a subtle hole: the buffering code path is exercised only during outages, so it is the least tested code in the system, and it runs at the worst possible time. A spool-first design makes the disk path the only path. The forwarder is a separate consumer that happens to be fast when the link is healthy and slow when it is not. Normal operation and outage operation differ only in how far behind the forwarder is, which is a number you can graph.
The cost is write amplification and a small latency floor. Every message touches storage once on the way in. On modern industrial gateways with eMMC or SSD that is cheap when writes are batched, and expensive if you fsync per message on cheap flash. The sizing section below quantifies it. For genuinely latency-critical paths, keep a parallel fire-and-forget publish for the live view and let the spool carry the authoritative history.
Where the QoS levels fit
The QoS you choose applies to the forwarder-to-broker hop only. QoS 0 is fire and forget: no acknowledgement, no retransmission, no packet identifier. If the TCP connection is down when you call publish, the message is not sent, and whether it is kept at all is an implementation detail of your client. QoS 1 requires a PUBACK from the receiver and guarantees at-least-once delivery, so duplicates are possible. QoS 2 runs a four-step PUBLISH, PUBREC, PUBREL, PUBCOMP exchange that gives exactly-once semantics for that hop, at the price of extra round trips and held state at both ends.
For a spool-based forwarder, QoS 1 is almost always the right setting. The spool already gives you the “keep until acknowledged” semantics, and application-level idempotent keys, described later, give you effective exactly-once at the consumer. QoS 2 adds a second round trip per message and doubles the state to recover after a crash, for a guarantee the consumer can obtain more cheaply. QoS 0 is reasonable only for data you are happy to lose, such as a high-rate vibration preview stream where the spool already holds the authoritative summary.
Persistent sessions: what the broker actually keeps
A persistent session is the broker’s memory of a client. In MQTT 3.1.1 it is controlled by the Clean Session flag. In MQTT 5 that single flag is split into Clean Start and a Session Expiry Interval, and the two interact. The specification states that when Clean Start is 1, “the Client and Server MUST discard any existing Session and start a new Session” (MQTT-3.1.2-4), and that the Session Expiry Interval, if absent, is treated as 0, meaning the session ends when the network connection closes. A value of 0xFFFFFFFF (UINT_MAX) means the session does not expire. The client and server must store session state after the connection closes if the interval is greater than 0 (MQTT-3.1.2-23).
What the session holds matters for store and forward. For a subscriber, it includes its subscriptions and the QoS 1 and 2 messages the broker has queued for it while it was offline, plus any in-flight handshake state. For a publisher, the relevant state is the client’s own unacknowledged outgoing QoS 1 and 2 messages, which it must retain and retransmit on reconnect. HiveMQ’s documentation adds a useful reading of the expiry rule: when a session expires, its queued messages expire with it, even if the individual messages carried longer expiry intervals.
A persistent session therefore protects the subscriber side of the outage. If your edge broker or your cloud ingestion client is the one that goes offline, a persistent session with a non-zero expiry is how it resumes without gaps. It does nothing for a publisher that has no connection and no spool, which is the situation most people mean when they say “store and forward”.
Figure 2: What QoS 0, 1 and 2 do across a disconnect, and where a persistent session is required for retransmission.
Figure 2 traces a QoS 1 publish interrupted by a drop. The client sent the PUBLISH and never saw the PUBACK. On reconnect with the session preserved, it retransmits the same packet identifier, flagged as a duplicate, and the broker may deliver the message to subscribers a second time. The MQTT 5 specification restricts retransmission to the moment a connection resumes: HiveMQ’s explainer quotes the rule as “Clients and Servers MUST NOT resend messages at any other time”, which rules out timer-based retry within an ongoing connection. If you want a retry timeout, you build it above MQTT, which is one more reason a spool with explicit acknowledgement tracking is easier to reason about.
Retained messages are not a buffer
Retained messages come up in every outage discussion and are almost always the wrong tool. A retained message is the broker’s single last-known value for a topic, delivered to new subscribers on subscribe. It is state, not history. If a gateway publishes ten thousand readings to one topic during a reconnect burst with the retain flag set, a subscriber that arrives later sees only the last one. Retain is correct for “current value” and “I am alive” topics, and Sparkplug uses it for exactly that purpose, but it cannot replay the outage. The MQTT 5 message expiry interval does apply to retained messages as well, per the HiveMQ essentials: the same interval controls how long the retained message stays on the topic.
Deeper Analysis: Broker Queues, Client Libraries and Buffer Designs
Everything above is protocol. What actually happens in an outage is decided by implementation defaults, and those defaults differ in ways that matter. This section walks through the main places a message can wait, with the documented limits of each, then compares three durable gateway designs and finishes with the sizing arithmetic.
Broker-side limits you will hit first
Take Mosquitto, the most common edge broker. Its documented defaults (from the mosquitto.conf manual page) are max_queued_messages of 1000 per client, counting QoS 1 and 2 messages beyond those in flight; max_queued_bytes of 0, meaning no byte cap; max_inflight_messages of 20; and queue_qos0_messages false. Once the queue limit is reached, the manual says subsequent messages are silently dropped. Persistence is also off by default: persistence is false, and when enabled the database is written at autosave_interval, which defaults to 1800 seconds. A broker restart between autosaves loses whatever arrived since the last save.
Run the arithmetic for a subscriber that receives 20 messages per second. The default queue holds 1000, so it fills in 50 seconds of subscriber downtime. After that, the 51st second onward is gone, with no error visible to anyone. This is not a bug; it is a defensible protection against a dead consumer exhausting broker memory. It means a persistent session in default Mosquitto is a short-outage tool, good for a consumer restart or a brief Wi-Fi drop, not a multi-hour buffer. Raising the limit is possible, and the manual calls an unlimited setting “not recommended”, with good reason: the queue is in RAM.
Enterprise brokers differ in mechanism but not in principle. They keep per-client queues with configurable depth and overflow strategy, usually drop-oldest or drop-newest, and many can persist them to disk. The question to ask of any broker is the same three-part one: what is the queue limit, what happens at the limit, and does the queue survive a broker restart. If the vendor cannot answer all three in writing, treat it as best-effort. Verify against your specific broker version, because these defaults change between releases.
Client library behaviour while offline
Client libraries are inconsistent, and the inconsistency is the source of many production surprises.
The Eclipse Paho Python client documents that publish() returns a result code of MQTT_ERR_SUCCESS or MQTT_ERR_NO_CONN if the client is not currently connected, and does not state whether a message published while disconnected is kept. Its README adds two relevant warnings: with clean_session=False the session lives only in memory and is lost on restart, so unacknowledged QoS 1 and 2 messages may be lost, and messages passed to publish() may be lost in general unless you confirm each with on_publish or wait_for_publish. The outgoing queue can be capped with max_queued_messages_set, where the documentation says 0 means unlimited, and in-flight messages default to 20. Reconnect delay defaults to a minimum of 1 second doubling to a maximum of 120 seconds. The practical lesson: the Paho in-memory queue is a retransmission aid, not a store.
MQTT.js documents queueQoSZero (default true: queue outgoing QoS 0 messages when the connection is broken), reconnectPeriod (default 1000 ms), and clean (default true; set to false to receive QoS 1 and 2 messages while offline). It documents outgoingStore and incomingStore options and describes its bundled mqtt.Store as an in-memory implementation. The README says the client supports starting to publish before being connected, but does not spell out in-memory queueing for QoS 1 and 2; verify against your version’s behaviour. Because the store is pluggable, a disk-backed store is possible, but then you own its correctness.
HiveMQ’s MQTT Client library and the Eclipse Paho Java and C clients each have their own offline-queue and persistence options, and I have not verified their current defaults for this post, so check their documentation before relying on them. The general conclusion holds across all of them: any buffering inside the client library is bounded by process memory unless you explicitly configure a persistent store, and none of them gives you the operational visibility (depth, age, bytes) you need to run an outage buffer.
Three durable designs for an edge gateway
When the requirement is “survive hours or days offline and a power cycle”, you need your own durable queue. Three designs recur.
Embedded SQLite in WAL mode. One table of pending messages, an autoincrement primary key as the sequence, and a forwarder that selects the oldest rows, publishes them, and deletes on acknowledgement. SQLite in write-ahead-log mode gives you atomic commits, crash recovery, and concurrent readers while one writer appends. It is easy to inspect with the sqlite3 CLI at 3 a.m., supports secondary indexes for priority queries, and handles retention with a plain DELETE. Throughput is excellent when inserts are batched in transactions, and poor if every message is its own fsynced transaction on slow flash.
Append-only segmented log. Messages are appended to fixed-size segment files with an offset index; the forwarder keeps a durable cursor; fully acknowledged segments are deleted whole. This is the Kafka-log model in miniature and gives the highest sustained write rate with minimal write amplification, because deleting a segment is a file unlink rather than row-by-row deletion. The costs are that you write more code, priority and selective drop are awkward, and corruption recovery at a torn tail is your problem.
Ring buffer on a preallocated file. A fixed-size circular file with head and tail pointers gives a hard capacity ceiling and oldest-first overwrite. It is attractive when the policy is “keep the most recent N hours, always”, for example on a device whose flash must never fill. The cost is silent data loss by design when the ring wraps, so the overwrite counter must be exported as a metric and alarmed.
| Property | SQLite WAL | Segmented log | Ring buffer |
|---|---|---|---|
| Crash safety | Built in | Needs careful tail repair | Needs careful pointer commit |
| Write amplification | Moderate | Low | Low |
| Priority and selective drop | Easy (indexes, SQL) | Awkward | Not supported |
| Capacity behaviour at full | Policy in code | Policy in code | Overwrites oldest |
| Inspectability | Excellent | Needs tooling | Needs tooling |
| Implementation effort | Low | High | Medium |
For most teams, SQLite is the correct default and the other two are optimisations you adopt when measurements demand them. The runnable example later in this post uses SQLite for that reason.
Sizing the buffer: worked math
Sizing a buffer is arithmetic, and skipping it is how teams discover a full disk during the outage they bought the buffer for. The numbers below are illustrative, not measurements; replace them with your own.
Assume a gateway aggregating 2,000 tags, with about 20 percent changing each second on average, which gives 400 messages per second. Assume a 180-byte payload, a 45-byte topic string and roughly 10 bytes of framing and metadata, about 235 bytes of raw content per message. A row store typically roughly doubles that with index and page overhead, so budget about 500 bytes per stored message. That is 400 x 500 = 200,000 bytes per second, or 720 MB per hour.
| Outage length | Messages buffered | Disk at 500 B per message |
|---|---|---|
| 1 hour | 1,440,000 | about 0.72 GB |
| 8 hours | 11,520,000 | about 5.8 GB |
| 24 hours | 34,560,000 | about 17.3 GB |
| 72 hours | 103,680,000 | about 51.8 GB |
If compression or batching reduces stored size by 3x, the 72-hour figure falls to roughly 17 GB. The more important number is drain time. Suppose the uplink and broker can sustain 1,500 messages per second of backfill on top of the 400 per second of live traffic that continues to arrive. The net drain rate is 1,100 messages per second, so clearing a 72-hour backlog of 103.68 million messages takes about 94,000 seconds, or 26 hours. Backfill is not instantaneous; it is a second, long outage in which you are serving two classes of traffic. That insight drives the priority design later.
Add headroom: the disk must hold the outage plus the drain period’s new arrivals, plus filesystem overhead and free space the flash controller needs for wear levelling. A common rule of thumb is to size the spool partition at no more than 60 to 70 percent of the device’s usable flash and to set the full-disk policy deliberately, as discussed in the failure-modes section.
Figure 3: Reconnect sequence. The forwarder resumes with jittered backoff, ships live data on a priority lane, and drains the backlog at a shaped rate.
Reconnect behaviour: jitter, rate shaping and the thundering herd
Figure 3 shows what should happen when the link returns. Two separate problems appear.
The first is the reconnect storm. If five hundred gateways lose the same backhaul and regain it at the same instant, they will all attempt TLS handshakes, authentication and session resumption within the same few seconds, and then all begin draining backlogs at line rate. The broker’s accept queue, its auth backend and its ingest pipeline each see a spike orders of magnitude above steady state. The remedy is the same one that fixes every thundering herd: exponential backoff with randomised jitter on the reconnect timer. Paho’s documented reconnect delay doubles from 1 to a 120-second cap, but doubling alone synchronises clients that failed together; add full jitter (a uniformly random delay between zero and the current cap) on top. Mosquitto’s bridge documentation shows the same idea: restart_timeout accepts a jittered form with a base and a cap, defaulting, per the manual, to a base of 5 seconds and a cap of 30 seconds.
The second problem is backfill starving live data. If the forwarder drains the spool strictly oldest-first, a dashboard that is supposed to show the current state sees ancient data for hours. The fix is two lanes: live messages bypass or jump the queue on a high-priority path, while backlog drains in the remaining capacity under a token-bucket rate limit. In the SQLite design, this is a priority column and an ORDER BY; in the log design, it is two cursors. Choose the backfill rate to leave headroom on both the uplink and the broker: a good starting point is a fraction of measured capacity, increased as broker lag metrics stay flat.
Broker Bridging: Mosquitto, EMQX and What the Bridge Really Buffers
A broker bridge makes a local broker act as a client of a remote broker and relays selected topics in one or both directions. It is the lowest-effort store-and-forward mechanism, because the local broker already accepts connections from every local client and the bridge is configuration, not code. It is also the mechanism whose buffering guarantees are the easiest to over-estimate.
Mosquitto bridge
A Mosquitto bridge is declared with a connection block. The documented options relevant to outages are these: topic takes a pattern, a direction of out, in or both, a QoS, and optional local and remote prefixes; cleansession defaults to false, which keeps remote subscriptions across drops; bridge_protocol_version defaults to mqttv311 and may be set to mqttv50; start_type is automatic by default, with lazy starting the bridge only when queued messages exceed a threshold (default 10) and stopping after an idle_timeout (default 60 seconds); keepalive_interval defaults to 60 seconds; and notifications publishes bridge state to $SYS/broker/connection/<remote_clientid>/state. For MQTT 5 bridges, bridge_session_expiry_interval defaults to 0, which expires the remote session immediately, and 0xFFFFFFFF means it never expires.
# /etc/mosquitto/conf.d/bridge-central.conf (illustrative; verify against your Mosquitto version)
connection edge-site01-to-central
address central-broker.example.com:8883
bridge_protocol_version mqttv50
bridge_cafile /etc/mosquitto/certs/ca.pem
bridge_certfile /etc/mosquitto/certs/edge.crt
bridge_keyfile /etc/mosquitto/certs/edge.key
# local telemetry/# goes out at QoS 1, prefixed with the site id on the remote side
topic telemetry/# out 1 "" site01/
cleansession false
local_clientid site01-bridge-local
start_type automatic
restart_timeout 5 30
keepalive_interval 30
notifications true
notification_topic site01/bridge/state
# in the main mosquitto.conf
persistence true
persistence_location /var/lib/mosquitto/
autosave_interval 60
max_queued_messages 200000
max_queued_bytes 536870912
queue_qos0_messages true
Read this configuration for what it does and does not do. The bridge queue is the bridge client’s session queue inside the local broker, so everything said about broker queue limits applies. The documentation does not give a separate offline-queue limit for bridges, and I have not found a statement of exactly how the global queue options interact with the bridge connection, so test it: disconnect the bridge, publish past your configured limit, reconnect, and count. Note also the interplay between QoS values. Per the manual, when bridge topics use QoS 1 or 2, incoming QoS 0 messages on those topics are queued only if queue_qos0_messages is true. If local publishers use QoS 0 and the option is false, the bridge happily forwards live traffic and silently buffers nothing through an outage.
The persistence caveat from the previous section applies with full force. With the built-in persistence, queued messages are written on the autosave schedule, so a power loss between saves loses the recent tail of the queue; the example sets a short interval, at the cost of more disk writes. The manual itself recommends a plugin-based persistence alternative over the built-in file, which is worth evaluating if your version supports it.
EMQX: rules, connectors and buffer workers
EMQX approaches the same problem with its data integration layer. An MQTT connector links to a remote broker; an MQTT sink action publishes the output of a rule to it; an MQTT source subscribes remotely and republishes locally. The EMQX documentation describes a per-resource buffer queue with a capacity limit in which, when it is full, the oldest data is discarded first. Disk cache files go to data/bufs for non-Kafka sinks. It also describes an async request mode (on by default) and a max_inflight setting for some sinks, noting that to preserve per-client ordering in async mode you set it to 1. Buffered messages are retried until their request TTL expires or the buffer overflows, and the metrics separate the drop reasons: dropped.expired, dropped.queue_full, dropped.resource_stopped and dropped.resource_not_found. Newer releases add fallback actions (documented as of 5.9.0) that run when the primary action fails, for example republishing to an alternative target.
Two cautions. First, I could not confirm from the EMQX pages I reviewed the exact config names for choosing memory versus disk buffer mode or the maximum buffer size; check the current reference for your version rather than copying names from memory. Second, EMQX’s drop-oldest overflow policy is a design decision with consequences: during a long outage you retain the newest data and lose the oldest. For many telemetry use cases that is the right default, because recent data is worth more. For audit or compliance records it is the wrong one.
Bridge versus spool: when to use which
A bridge shines when you already run a local broker, the data volume is moderate, the acceptable outage is minutes to a few hours, and the loss of the tail on a power failure is tolerable. It fails you when the outage is days, when you need guaranteed capacity accounting, when you need to prioritise or selectively drop, or when the local broker itself is not trusted to survive a power cut with its queue intact. The spool in front of the forwarder is the more defensible design for those cases. Many production edge stacks run both: local clients publish to the local broker for real-time use, a spool-based forwarder subscribes to the broker with a persistent session and writes everything to disk, and only then publishes upstream.
Ordering, Duplicates and Idempotent Keys
Outages break two properties that engineers often assume: order and uniqueness. MQTT gives you ordering only in bounded circumstances. The specification defines ordering guarantees per topic and QoS within a session, and Mosquitto’s manual notes that setting max_inflight_messages to 1 guarantees in-order delivery. With larger in-flight windows, a retransmitted message after a reconnect can overtake or trail newer ones. Receive Maximum, a CONNECT property that defaults to 65,535 if absent, limits concurrent QoS 1 and 2 publications on the connection, and the specification notes there is no mechanism to limit QoS 0 publications a server might try to send.
Duplicates come from at-least-once delivery. If a PUBACK is lost after the broker has processed the message, the client retransmits on reconnect and the broker delivers a second copy. Paho’s README is candid about a related point: with clean_session=True, QoS above zero messages are republished across network reconnections, which it describes as a departure from the MQTT standard, so QoS 2 messages can arrive twice; it advises clean_session=False if you need single delivery for QoS 2.
The robust answer is not to chase exactly-once in the transport, but to make duplicates harmless at the consumer. Give every message an idempotency key that is stable across retries and unique per logical reading. A good key is the concatenation of a durable source identifier, the metric name, and the source timestamp, or a per-device monotonically increasing sequence number that persists across restarts. The consumer writes with an upsert or a unique constraint on that key. Then at-least-once plus idempotent write is effectively exactly-once in the database, regardless of how many times the network replays.
Be careful about sequence numbers that reset. A counter held in RAM restarts at zero after a reboot, and a naive consumer then discards new data as duplicates of old. Persist the counter in the same transaction as the spool insert, or include a boot epoch in the key.
Late Data: Timestamps, Clock Skew and Event Time
When data arrives hours after it was produced, the question of “when” has three answers: when the sensor measured it, when the gateway received it, and when the platform ingested it. Store and forward is only coherent if the first one is captured at the source and carried in the payload. Never use the broker or ingestion time as the measurement time for backfilled data: the whole outage would collapse into a single timestamp at reconnect.
Edge devices often lack reliable clocks. A gateway that boots without network time can stamp readings with 1970 or with the last persisted time, and an NTP step correction minutes later puts them out of order. Useful countermeasures are to record a monotonic offset alongside wall-clock time and to flag readings stamped before time sync, to correct them in the backend once the sync event is known, and to export clock offset as a metric so skew is visible. Time-series databases need to accept out-of-order and late writes within a defined window, and downstream aggregations (rollups, alerts) must either wait for a lateness bound or be recomputable. A real-time alert computed from backfilled data is an alert about the past; mark backfilled records with a flag, in a payload field or user property, so alerting logic can ignore them.
MQTT 5 Message Expiry Interval is the matching control on the delivery side. It lets a publisher declare that a message is worthless after N seconds, so a stale reading does not clog a recovering broker queue. Use it for data whose value decays, such as live status, and omit it for data you must keep. Per the HiveMQ essentials, a message with no expiry interval is held for matching offline subscribers with no time limit, which in an outage means unbounded queue growth.
Sparkplug B: Birth, Death and Reconnect Semantics
If you use Sparkplug, store and forward interacts with its state model. Sparkplug defines NBIRTH and NDEATH messages for edge nodes and DBIRTH and DDEATH for devices. The death certificate is registered as the MQTT Last Will and Testament so the broker publishes it if the node disconnects ungracefully. As summarised in HiveMQ’s overview of Sparkplug 3.0, a Sparkplug-aware server stores NBIRTH and DBIRTH messages as they pass through, makes them available under a $sparkplug/certificates/... topic with the retain flag set, and may replace the NDEATH timestamp with the UTC time it attempts delivery, because the timestamp inside a Last Will is otherwise unreliable.
The consequence for outages: when the uplink drops, the broker publishes the NDEATH and the host application marks every metric of that node stale. When the node reconnects it must publish a fresh NBIRTH so the host can re-establish the full metric set before any NDATA is meaningful. Buffered NDATA from the outage must therefore not be flushed before the rebirth, and the backlog must be marked as historical so the host does not treat it as live. Sparkplug payloads support a historical flag at the metric level; I have not verified the exact field names in the 3.0 specification text for this post, so confirm them in the normative document before implementing. The practical rule is: reconnect, birth, then drain, with the backlog carried as historical data. For the broader trade-offs between Sparkplug and plain topic design, see our Sparkplug B versus plain MQTT topics analysis.
Runnable Example: A SQLite Spool and Forwarder in Python
The following is a compact, working reference for the spool-first design: durable insert, a forwarder that deletes only on PUBACK, live-first ordering, a token-bucket backfill limit, a disk cap with oldest-first shedding, and a dead-letter table for poison messages. It targets paho-mqtt 2.x, whose README documents the CallbackAPIVersion.VERSION2 callback signatures used here. Treat it as a starting point: it omits TLS options, metrics export and graceful shutdown for brevity.
import json, random, sqlite3, threading, time
import paho.mqtt.client as mqtt
from paho.mqtt.properties import Properties
from paho.mqtt.packettypes import PacketTypes
DB_PATH = "/var/lib/edge/spool.db"
MAX_DB_BYTES = 8 * 1024**3 # hard cap for the spool file
LIVE_WINDOW_MS = 60_000 # rows younger than this are "live"
BACKFILL_RATE = 800 # msgs/sec token bucket for backlog
MAX_ATTEMPTS = 5
class Spool:
def __init__(self, path):
self.db = sqlite3.connect(path, check_same_thread=False, isolation_level=None)
self.lock = threading.Lock()
self.db.execute("PRAGMA journal_mode=WAL")
self.db.execute("PRAGMA synchronous=NORMAL") # trade: tiny loss window on power cut
self.db.execute("""CREATE TABLE IF NOT EXISTS spool(
id INTEGER PRIMARY KEY AUTOINCREMENT, key TEXT UNIQUE,
topic TEXT NOT NULL, payload BLOB NOT NULL,
ts_ms INTEGER NOT NULL, attempts INTEGER NOT NULL DEFAULT 0)""")
self.db.execute("""CREATE TABLE IF NOT EXISTS dead(
id INTEGER, key TEXT, topic TEXT, payload BLOB, ts_ms INTEGER, reason TEXT)""")
self.dropped = 0
def put_many(self, rows): # rows: (key, topic, payload_bytes, ts_ms)
with self.lock:
self.db.execute("BEGIN")
self.db.executemany(
"INSERT OR IGNORE INTO spool(key,topic,payload,ts_ms) VALUES(?,?,?,?)", rows)
self.db.execute("COMMIT")
self._enforce_cap()
def _enforce_cap(self):
size = self.db.execute(
"SELECT page_count*page_size FROM pragma_page_count(), pragma_page_size()").fetchone()[0]
if size > MAX_DB_BYTES: # shed oldest first; count it, alarm on it
cur = self.db.execute(
"DELETE FROM spool WHERE id IN (SELECT id FROM spool ORDER BY id LIMIT 5000)")
self.dropped += cur.rowcount
def next_batch(self, n, skip_ids):
cutoff = int(time.time() * 1000) - LIVE_WINDOW_MS
with self.lock:
rows = self.db.execute(
"SELECT id,key,topic,payload,ts_ms FROM spool "
"ORDER BY (ts_ms < ?), id LIMIT ?", (cutoff, n + len(skip_ids))).fetchall()
return [r for r in rows if r[0] not in skip_ids][:n]
def ack(self, row_id):
with self.lock:
self.db.execute("DELETE FROM spool WHERE id=?", (row_id,))
def fail(self, row_id, reason):
with self.lock:
self.db.execute("UPDATE spool SET attempts=attempts+1 WHERE id=?", (row_id,))
self.db.execute(
"INSERT INTO dead SELECT id,key,topic,payload,ts_ms,? FROM spool "
"WHERE id=? AND attempts>=?", (reason, row_id, MAX_ATTEMPTS))
self.db.execute("DELETE FROM spool WHERE id=? AND attempts>=?", (row_id, MAX_ATTEMPTS))
class Forwarder:
def __init__(self, spool, host, port=8883):
self.spool, self.host, self.port = spool, host, port
self.pending = {} # mid -> spool id
self.connected = threading.Event()
c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2,
client_id="site01-forwarder", protocol=mqtt.MQTTv5)
c.max_inflight_messages_set(100)
c.reconnect_delay_set(min_delay=1, max_delay=60)
c.on_connect, c.on_disconnect, c.on_publish = self.on_connect, self.on_disconnect, self.on_publish
self.c = c
def on_connect(self, c, ud, flags, rc, props):
if rc.is_failure:
return
if not flags.session_present:
self.pending.clear() # broker lost our session; spool still has the rows
time.sleep(random.uniform(0, 5)) # jitter before draining (reconnect storm control)
self.connected.set()
def on_disconnect(self, c, ud, flags, rc, props):
self.connected.clear()
def on_publish(self, c, ud, mid, rc, props):
row_id = self.pending.pop(mid, None)
if row_id is None:
return
if rc.is_failure:
self.spool.fail(row_id, str(rc))
else:
self.spool.ack(row_id) # delete only after PUBACK
def run(self):
p = Properties(PacketTypes.CONNECT)
p.SessionExpiryInterval = 3600
self.c.connect_async(self.host, self.port, keepalive=30, properties=p)
self.c.loop_start()
tokens, last = float(BACKFILL_RATE), time.monotonic()
while True:
if not self.connected.wait(timeout=5):
continue
now = time.monotonic()
tokens = min(BACKFILL_RATE, tokens + (now - last) * BACKFILL_RATE)
last = now
batch = self.spool.next_batch(100, set(self.pending.values()))
if not batch:
time.sleep(0.2); continue
cutoff = int(time.time() * 1000) - LIVE_WINDOW_MS
for row_id, key, topic, payload, ts_ms in batch:
live = ts_ms >= cutoff
if not live:
if tokens < 1:
break
tokens -= 1
props = Properties(PacketTypes.PUBLISH)
props.UserProperty = [("idem", key), ("hist", "0" if live else "1")]
info = self.c.publish(topic, payload, qos=1, properties=props)
if info.rc == mqtt.MQTT_ERR_SUCCESS:
self.pending[info.mid] = row_id
def make_row(device, metric, value, ts_ms):
key = f"{device}/{metric}/{ts_ms}" # stable across retries
topic = f"telemetry/{device}/{metric}"
return (key, topic, json.dumps({"v": value, "t": ts_ms}).encode(), ts_ms)
Several details in that listing are deliberate. The idempotency key travels as an MQTT 5 user property, so the consumer can deduplicate without parsing payloads. The hist property marks backfilled messages so alerting can ignore them. Live rows skip the token bucket entirely, which is the two-lane policy from Figure 3 expressed as one ORDER BY clause. The failure path counts attempts and parks a message in dead after five, so a single bad record cannot block the queue. Because paho may retransmit its own in-flight packets after a reconnect while the spool also still holds the row, the same message can reach the broker twice; that is precisely what the idempotency key is for.
Test it as you would any recovery code: kill the network with iptables, kill the process with SIGKILL mid-batch, fill the disk, and restart with a six-hour backlog. The assertion is not “no errors” but “the count of distinct keys at the consumer equals the count produced”.
Observability: What to Measure During an Outage
A buffer you cannot see is a buffer you will not trust. The minimum viable metric set for any store-and-forward component is small and should be exported from every gateway.
- Spool depth in messages and bytes. The primary health signal. Alert on sustained growth rather than a fixed value, since a nonzero depth is normal during a healthy publish cycle.
- Oldest message age. The most useful single number: it is the real lag of your data. A depth of one million messages is a minute of data at 16,000 messages per second and a day at 12 per second; age tells you which.
- Drain rate versus arrival rate. If drain does not exceed arrival after reconnect, the backlog never clears. This is the signal that your backfill limit is too conservative or the broker is saturated.
- Drops by reason. Cap shedding, expiry, dead-lettered, and broker rejection each get their own counter. EMQX’s separate
dropped.queue_fullanddropped.expiredcounters are a good model. - Link state and flap count. A connection that flaps every ten seconds produces a very different failure pattern from one clean outage, and it exercises the retransmission paths harder.
- Clock offset. The difference between gateway time and a trusted source, so skew is a graph rather than a surprise.
- Disk free and write latency on the spool partition.
On the broker side, watch the equivalents: per-session queue depth, messages dropped due to queue limits, and for bridges the connection state topic. Mosquitto publishes the bridge state to the $SYS/broker/connection/<remote_clientid>/state topic, with 1 meaning connected and 0 meaning failed, per its manual; subscribe to it and alert on duration of the 0 state, not merely its occurrence.
Do synthetic end-to-end checks. A canary publisher at each site that emits a sequence-numbered heartbeat every few seconds lets the central side compute gaps per site directly, independent of any internal counter. Gaps that are later filled by backfill are a successful outage recovery; gaps that persist are lost data, and the difference is invisible without the canary.
Trade-offs, Gotchas, and What Goes Wrong
Most store-and-forward incidents are not protocol bugs. They are capacity, policy and clock problems that appear only under outage conditions.
Figure 4: Common failure modes and the mitigation attached to each stage of the buffer lifecycle.
Disk full. The spool fills during a long outage, and now the acquisition process cannot write. The behaviours available are block, drop-newest, or drop-oldest, and the wrong default is whichever one nobody chose. Dropping oldest keeps the freshest data and is usually right for telemetry; dropping newest preserves a complete prefix and may suit audit trails; blocking acquisition can crash control logic. Whatever you pick, count it, alarm on it, and size so it is a last resort. A tiered policy also works: shed low-value tags first, keep alarms and events.
Poison messages. A message the broker will always reject, because it exceeds the broker’s maximum packet size, violates an ACL, or fails a schema check, will block a strictly ordered queue forever if retried forever. Cap attempts and dead-letter, as the example does, and keep the dead-letter table small and inspectable. MQTT 5 helps by returning reason codes on PUBACK, so you can distinguish permanent from transient failures.
Clock skew and time steps. Discussed above: stamp at the source, carry the stamp, flag pre-sync readings, and tolerate late writes downstream.
Reconnect storms. Backoff without jitter synchronises clients. Add jitter, shape the backfill, and stagger fleet-wide events such as firmware updates and certificate rotations.
Session expiry shorter than the outage. A persistent session with a one-hour expiry does not survive a six-hour outage: the broker discards the session and its queue, and the reconnecting client is told Session Present is 0. Set expiry to exceed your design outage, within any server-side maximum the broker enforces, and remember the broker’s own queue limit still applies.
Retained-message floods. A bridge or client that reconnects with a clean session resubscribes and receives every retained message again. The Mosquitto manual warns of this directly for cleansession true on bridges. Keep sessions persistent and limit retained messages to genuine state topics.
Silent QoS downgrade. A QoS 1 publish delivered to a QoS 0 subscription is a QoS 0 delivery, so the end-to-end path can lose messages even when the publisher-to-broker leg is acknowledged. Audit subscription QoS on the consumers that matter.
The fsync trade-off. Setting SQLite synchronous=NORMAL in WAL mode, as the example does, keeps writes fast but accepts a small window of loss on power failure, whereas FULL narrows that window at a throughput cost. This is a business decision about how many seconds of data you can afford to lose, not a technical default; make it explicitly per site class.
Over-trusting the broker as a buffer. A broker is optimised for fan-out, not for being the system of record for an outage. If you find yourself raising queue limits to millions of messages to survive an outage, you are building a spool in the wrong place.
Practical Recommendations
For most industrial gateways, the defensible default is a layered design: a durable local spool written before publishing, a forwarder using QoS 1 with a persistent MQTT 5 session and a session expiry longer than your design outage, idempotency keys on every message, and a local broker only where local consumers need it. Use a bridge as a convenience for moderate-volume, short-outage sites, and verify its queue behaviour by experiment rather than by reading defaults.
Decide the numbers before the outage does. State the design outage in hours, compute the disk (the worked math above is a template), compute the drain time, and write down the overflow policy. Treat backfill as a first-class traffic class with its own rate limit and its own marker in the data. Test the full cycle regularly, not once: pull the uplink on a canary site monthly and check that distinct keys at the consumer match distinct keys produced.
- Write to a durable spool first; publish from the spool, delete on PUBACK.
- Use QoS 1 plus idempotent consumers; reserve QoS 2 for cases where the extra round trips are justified.
- Set MQTT 5 Session Expiry Interval above the design outage, and Message Expiry on perishable data.
- Configure and test broker queue limits, persistence and autosave; do not trust defaults (Mosquitto defaults: 1000 queued messages, persistence off).
- Jitter every reconnect; shape backfill; keep live data on a priority lane.
- Carry source timestamps, device sequence or idempotency keys, and a historical flag in every message.
- Pick and alarm the full-disk policy; dead-letter poison messages.
- Export spool depth, oldest-message age, drain rate and drops by reason; run canary heartbeats.
- For Sparkplug, rebirth before drain and mark backlog as historical.
Note for readers in regulated settings: this post is a systems analysis, not compliance guidance. Retention, integrity and audit requirements for buffered data vary by industry and jurisdiction.
Frequently Asked Questions
Does MQTT QoS 1 or 2 guarantee delivery if the connection drops?
Only partly. QoS 1 and 2 guarantee delivery of a message once it is handed to a session that keeps it, but retransmission happens only when the client reconnects with its session intact. If the client process restarts and its state was in memory, the unacknowledged messages are gone. Also, the guarantee covers a single hop: the broker may deliver to a subscriber at a lower QoS. For outages longer than minutes, add a durable application spool.
What is the difference between clean start and session expiry in MQTT 5?
Clean Start decides whether to discard any existing session at connect time: when it is 1, the server and client must discard the old session and begin a new one. Session Expiry Interval decides how long the session survives after disconnect. If absent it is 0, so the session ends when the connection closes; 0xFFFFFFFF means it never expires. A resumable session needs Clean Start 0 plus a non-zero expiry.
How many messages can an MQTT broker queue for an offline client?
It depends entirely on the broker and its configuration. Mosquitto documents a default of 1000 queued QoS 1 and 2 messages per client, with no byte cap by default, and drops further messages silently once the limit is reached. QoS 0 messages are not queued unless the non-standard queue_qos0_messages option is enabled. Other brokers have their own limits and overflow strategies, so check your version’s documentation and test it.
Is a Mosquitto bridge enough for store and forward at the edge?
For short outages and moderate volume, often yes: with a persistent bridge session, QoS 1 on the bridge topics and raised queue limits, it buffers inside the local broker. It is weaker for multi-day outages, because the queue is bounded by broker memory and persists only on the autosave schedule. For guaranteed capacity, priorities and crash safety, put a disk-backed spool in front of the forwarder.
How do I avoid duplicate messages after a reconnect?
You cannot fully prevent them at the transport level with QoS 1, and even QoS 2 has caveats across client implementations. Make duplicates harmless instead: give each message a stable idempotency key, such as device, metric and source timestamp, and write with an upsert or unique constraint. The consumer then converges on one record per key no matter how many times the network replays a message.
What happens to Sparkplug data when the edge node loses its connection?
The broker publishes the node’s NDEATH Last Will, and the host application marks the node’s metrics stale. After reconnecting, the node must send a new NBIRTH before data is meaningful again. Buffered data from the outage should follow the rebirth and be marked historical so it is not mistaken for live values. Confirm the exact historical-flag fields in the Sparkplug specification for your version.
Further Reading
- OPC UA PubSub over MQTT versus UADP: architecture comparison for how the transport choice affects delivery and buffering.
- Edge LLM benchmark on Jetson Orin: Llama, Phi and Gemma for sizing compute alongside buffers on constrained edge hardware.
- Sparkplug B versus plain MQTT topics for birth and death semantics and topic design.
- NATS JetStream versus Kafka for edge IIoT telemetry for brokers with native replay.
- OASIS MQTT Version 5.0 specification, the normative source for sessions, expiry and delivery rules.
- Mosquitto configuration manual (mosquitto.conf) for queue, persistence and bridge options.
- Eclipse Paho Python client documentation for publish, queue and reconnect behaviour.
- EMQX data integration documentation for sinks, buffer workers and fallback actions.
By Riju — about
