IoT Data Compression: Zstandard vs LZ4 vs Snappy vs Gorilla Encoding, Dictionaries and Bandwidth Cost (2026)
Run Zstandard over a single 137-byte JSON telemetry message and you save about 6% of the bytes. Train a 4 KB dictionary on yesterday’s traffic and the same message shrinks to about 31 bytes. Batch 100 messages into a columnar block of delta-encoded integers and you get under 3 bytes per sample. IoT data compression is mostly decided by framing and representation. The choice of codec comes third.
This matters in 2026 because cellular IoT fleets are billed per megabyte. Battery devices pay for every radio-on millisecond, and LTE-M and NB-IoT uplinks are narrow. Teams still argue “zstd vs lz4” while the TCP, TLS and MQTT headers around each message often outweigh the payload.
You will leave with a layered model of where bytes actually go and a runnable Python harness. You also get worked cellular cost maths and a decision matrix for gateways and microcontrollers.
What this covers: codec internals (Zstandard, LZ4, Snappy, Deflate, Brotli), Gorilla, Chimp and ALP time-series encodings, trained dictionaries, MCU memory limits, a benchmark harness, bandwidth economics and the failure modes that bite in production.
Context and Background
General-purpose lossless compressors, the workhorses of IoT data compression, fall into two families. LZ77-style compressors find repeated byte sequences inside a sliding window and replace them with back-references. Entropy coders then squeeze the literal bytes and match lengths using Huffman or finite-state-entropy tables. Deflate (RFC 1951) pairs LZ77 with Huffman coding and is what gzip and zlib wrap. Zstandard, specified in RFC 8878 (February 2021, obsoleting RFC 8478), pairs LZ77 with Huffman for literals and finite-state entropy for sequences. LZ4 and Snappy skip entropy coding entirely and trade ratio for raw speed.
Time-series databases took a different path. Facebook’s Gorilla paper (Pelkonen et al., VLDB 2015) showed that monitoring data compresses well when you exploit its structure. Timestamps arrive at near-fixed intervals, so the delta of the delta is usually zero. Consecutive float values are often identical or close, so XOR-ing them leaves mostly zero bits. Gorilla reported an average of about 1.37 bytes per 16-byte timestamp-value pair, roughly 12 times smaller. Its descendants, Chimp (VLDB 2022) and ALP (SIGMOD 2024), improve on the float half. We cover the bit-level mechanics in depth in our Gorilla delta-of-delta and XOR compression deep dive.
The gap this post fills is IoT data compression on the wire. Storage engines see millions of points per series in two-hour blocks. A gateway publishing over MQTT sees one 100 to 500-byte message at a time, wrapped in protocol headers and encrypted. Storage-engine algorithms and general codecs behave very differently at that scale. If you want the storage side, read our time-series database internals comparison of InfluxDB, TimescaleDB and QuestDB.
Two adjacent techniques are out of scope here, and you should apply them first. Lossy filtering (deadband, swinging door) removes samples that carry no new information; see our historian deadband and swinging-door guide. Compact binary serialization replaces JSON’s repeated keys; see our CBOR vs Protobuf vs MessagePack comparison. This post assumes you have done both and still need to cut bytes.
The Layered Model of IoT Data Compression
Direct answer: For IoT data compression on the wire, batching and representation beat codec choice. Individual messages under about 200 bytes barely compress with any general-purpose codec unless you use a trained dictionary. Batched, columnar, delta-encoded blocks compressed with Zstandard at a low level usually win on ratio. LZ4 wins on CPU-starved devices.

Figure 1: The six layers where telemetry bytes are removed, in the order they must be applied.
Figure 1 shows the pipeline as layers, each feeding the next. Samples first pass a deadband or exception filter. Survivors are serialized compactly, collected into a batch window, and transposed into columns with delta-of-delta and XOR transforms. A byte codec then runs, optionally with a shared dictionary. Encryption comes last, and the cloud reverses each step before re-encoding for storage. The order is not negotiable: each layer works only on the redundancy left by the layer before it.
Why single messages barely compress
LZ77 compressors need history. A back-reference points to an earlier occurrence of the same bytes, and in a 137-byte message there is very little earlier occurrence to point to. The keys in {"deviceId":"press-07","ts":...,"temperatureC":...} appear once per message, so the compressor sees each for the first time. On top of that, every format adds framing. A Zstandard frame starts with a 4-byte magic number (0xFD2FB528) and a frame header. An LZ4 frame has its own 4-byte magic number (0x184D2204), a descriptor and an end mark.
The result, which you can reproduce with the harness later in this post, is counter-intuitive. Per-message LZ4 frames and Snappy output were larger than the raw JSON in our example run. Zstandard at level 3 saved about 6%. Deflate did slightly better because zlib’s header overhead is small. None of these numbers justify the CPU and code size on a device.
Why batching changes everything
Put 100 of those messages into one buffer and the picture inverts. The second message can reference the keys, device ID and firmware string of the first. In our example run, batched Zstandard at level 3 averaged 8.8 bytes per message, down from 137. That is about a 15-times reduction, versus the 6% from compressing each message alone.
Batching also amortizes protocol overhead, which on a cellular link is the bigger prize. Each MQTT PUBLISH over TLS over TCP carries a fixed tax of headers and acknowledgements regardless of payload size. Cutting the number of publishes by 100 cuts that tax by 100. The cost is latency: a 100-sample batch at 10-second sampling holds data for over 16 minutes. Our MQTT store-and-forward gateway guide covers how to buffer that safely across outages.
Why representation beats codec
General codecs see bytes, not numbers. A temperature of 61.23 encoded as JSON text is five ASCII characters that change unpredictably in the last two positions. The same value as a scaled integer delta from the previous sample is usually a single small number. Zigzag-varint encoding packs that number into one byte.
Transposing to columns puts all timestamps together, then all temperatures, then all vibration values. Each column then becomes a run of near-identical small integers, which any entropy coder handles well. In our example run, a 100-sample batch of three fields encoded this way and passed through Zstandard came to about 2.5 bytes per sample. That is roughly 3.5 times better than compressing the same batch as JSON, although the JSON batch also carries constant fields the columnar version drops. Parquet formalizes the same idea with its DELTA_BINARY_PACKED and BYTE_STREAM_SPLIT encodings, as described in the Parquet encodings specification.
Where Gorilla, Chimp and ALP fit
In IoT data compression terms, Gorilla-family encodings are columnar transforms with a built-in bit packer, not general codecs. The Gorilla paper encodes a timestamp’s delta-of-delta of zero as a single 0 bit. Small deviations go into tagged buckets of 9, 12 or 16 bits, and anything larger takes 36 bits. Values are XOR-ed with their predecessor. An identical value costs one bit, and a change costs a 2-bit control code plus the meaningful bits, sometimes with an 11-bit header for a new leading/trailing zero window. In Facebook’s data, about 96% of timestamps compressed to one bit and about 51% of values were identical to the previous one.
Chimp (Liakos, Papakonstantinopoulou and Kotidis, PVLDB 15(11), 2022) redesigns the flag codes and leading-zero encoding. Its Chimp128 variant XORs against the best of the previous 128 values. The authors report Chimp128 output about 40% smaller than Gorilla on average across their datasets. ALP (Afroozeh, Kuffo and Boncz, SIGMOD 2024) takes a different route. It detects doubles that started life as decimals and encodes them as integers, falling back to a vectorized scheme otherwise. ALP is DuckDB’s default float compression and appears as a preview encoding in the Parquet specification.
The decimal insight matters for IoT data compression. Most industrial telemetry is decimal: a sensor reports 61.23, not an arbitrary IEEE 754 double. XOR-ing the binary forms of 61.23 and 61.24 produces many meaningful bits, because neither number is exact in binary. In our harness, a simplified Gorilla bit count on two-decimal temperatures came to about 7 bytes per point. That is much worse than the paper’s 1.37-byte average and much worse than scaled-integer deltas. If you control the encoder, scale to integers first.
Codec Internals That Matter on Devices
Each codec makes a different IoT data compression bargain between ratio, speed, memory and code size. The details below are the ones that change engineering decisions on gateways and microcontrollers.
Zstandard: levels, windows and dictionaries
The reference zstd command-line tool exposes levels 1 to 19, with 3 as the default (zstd manual). The --ultra flag unlocks levels 20 to 22, which use considerably more memory, including at decompression. The --fast=N modes trade ratio for more speed. The current release at the time of writing is v1.5.7. Python 3.14 added a standard-library compression.zstd module, so gateways on recent Python no longer need a third-party package.
Window size is the memory knob that matters on constrained hardware. RFC 8878 allows a window from 1 KB up to terabytes. It recommends that decoders support windows up to 8 MB and that encoders not exceed 8 MB for general interoperability. For a gateway this is irrelevant. For a microcontroller it is the whole story, because the decoder must hold a window-sized history buffer. A small windowLog (10 means 1 KB) bounds that memory, at some cost in ratio on long inputs.
Dictionaries are where Zstandard earns its place in IoT data compression. A dictionary is a pre-shared blob of typical content plus pre-computed entropy tables. Both sides load it before compressing, so the first message already has history to reference. RFC 8878 defines a dictionary format with its own magic number (0xEC30A437) and a 4-byte Dictionary_ID. A frame can carry that ID so the decoder knows which dictionary to load.
The zstd --train command builds dictionaries from samples. The manual recommends more than 100 samples, ideally about 100 times the target dictionary size in total. It defaults to a maximum of 112,640 bytes (110 KiB) and uses only the first 128 KiB of large samples. In our example run, a 4 KB dictionary trained on 5,000 synthetic messages cut a 137-byte message to about 31 bytes, a 4.4-times reduction on a single message.
LZ4: speed and a 64 KB window
LZ4 is an LZ77 compressor with no entropy stage, designed for decompression speed. The LZ4 frame format wraps blocks of up to 64 KB, 256 KB, 1 MB or 4 MB. It adds an optional xxHash-32 content checksum and an optional 4-byte dictionary ID. Linked blocks can reference data up to 64 KB back. LZ4 also has a high-compression mode (LZ4HC) with levels that trade compression time for ratio while keeping decompression fast.
On microcontrollers, the bare LZ4 block format matters more than the frame format. It has no magic number and no header, and the caller supplies the decompressed size out of band. The compressor’s hash table is sized by a compile-time LZ4_MEMORY_USAGE constant, 16 KB by default. Decompression needs no state beyond the input and output buffers. LZ4 accepts a dictionary of up to 64 KB, so the shared-dictionary trick also works on devices that cannot fit Zstandard. In our example run, LZ4 block mode with a 4 KB dictionary reached about 33 bytes per message, close to Zstandard with a dictionary.
Snappy, Deflate and Brotli
Snappy, from Google, targets very high speed with “reasonable” compression. Its README cites roughly 250 MB/s compression and 500 MB/s decompression on a single Core i7 core. It reports output 20% to 100% larger than zlib’s fastest mode. Snappy appears in Kafka, Parquet and many databases, so it shows up in IoT pipelines through the cloud side rather than on devices. Its main IoT relevance is that cloud consumers already speak it.
Deflate (RFC 1951) inside zlib (RFC 1950) or gzip (RFC 1952) remains the compatibility default. Almost every embedded toolchain and cloud service can decode it. Its 32 KB window and Huffman stage give middling ratio at middling speed. Zlib does support preset dictionaries, which is easy to forget, but it has no built-in trainer.
Brotli (RFC 7932) ships a built-in static dictionary of common web text of roughly 120 KB, which helps HTML and JSON-like text without training. It shines on static web assets compressed once and served many times. For telemetry encoded on the device per message, its slow high levels and large dictionary rarely pay off compared with a trained Zstandard dictionary.
Walk-through: Benchmark Harness You Can Run
The IoT data compression harness below generates synthetic machine telemetry, trains a dictionary on one day of messages, and measures a separate test set. It compares per-message and batched compression, plus a columnar delta transform. It also includes a simplified Gorilla bit counter so you can see how decimal-origin floats behave. Replace make_msgs with a dump of your real traffic before you trust any number it prints.
"""Telemetry compression harness: per-message vs batched, with and without a zstd dictionary.
pip install zstandard lz4 python-snappy cbor2"""
import json, random, time, zlib, statistics, platform, struct
import zstandard as zstd
import lz4.block, lz4.frame
try:
import snappy
except ImportError:
snappy = None
try:
import cbor2
except ImportError:
cbor2 = None
random.seed(42)
def make_msgs(n, device="press-07", start=1_763_164_800_000, period_ms=10_000):
temp, vib = 61.2, 0.84
out = []
for i in range(n):
temp += random.gauss(0, 0.05)
vib = max(0.0, vib + random.gauss(0, 0.01))
out.append({
"deviceId": device,
"ts": start + i * period_ms + random.choice([0, 0, 0, 1, -1]),
"temperatureC": round(temp, 2),
"vibrationRmsMmS": round(vib, 3),
"spindleRpm": 1450 if i % 50 else 1452,
"status": "RUNNING",
"fw": "4.2.1",
})
return out
train = [json.dumps(m, separators=(",", ":")).encode() for m in make_msgs(5000, start=1_763_000_000_000)]
test_objs = make_msgs(2000)
test = [json.dumps(m, separators=(",", ":")).encode() for m in test_objs]
dict_data = zstd.train_dictionary(4096, train)
c_plain = zstd.ZstdCompressor(level=3, write_content_size=False, write_checksum=False)
c_dict = zstd.ZstdCompressor(level=3, dict_data=dict_data, write_content_size=False,
write_checksum=False, write_dict_id=False)
d_dict = zstd.ZstdDecompressor(dict_data=dict_data)
lz4_dict = dict_data.as_bytes()[-4096:]
codecs = {
"raw JSON": lambda b: b,
"zlib-6 (deflate)": lambda b: zlib.compress(b, 6),
"zstd-3": c_plain.compress,
"zstd-3 + 4 KB dict": c_dict.compress,
"lz4 block": lambda b: lz4.block.compress(b, store_size=False),
"lz4 block + dict": lambda b: lz4.block.compress(b, store_size=False, dict=lz4_dict),
"lz4 frame": lz4.frame.compress,
}
if snappy:
codecs["snappy"] = snappy.compress
def per_message(fn, msgs):
t0 = time.perf_counter()
sizes = [len(fn(m)) for m in msgs]
dt = (time.perf_counter() - t0) / len(msgs) * 1e6
return statistics.mean(sizes), dt
print(f"python {platform.python_version()}, zstandard {zstd.__version__} "
f"(libzstd {zstd.ZSTD_VERSION}), lz4 {lz4.__version__}")
print(f"mean raw message: {statistics.mean(map(len, test)):.1f} B")
print("\n-- per message --")
for name, fn in codecs.items():
size, us = per_message(fn, test)
print(f"{name:22s} {size:7.1f} B {us:6.2f} us/msg")
if cbor2:
cb = [cbor2.dumps(o) for o in test_objs]
print(f"{'CBOR (no compression)':22s} {statistics.mean(map(len, cb)):7.1f} B")
# Round-trip check: never ship a codec path you have not decoded.
assert d_dict.decompress(c_dict.compress(test[0]), max_output_size=4096) == test[0]
print("\n-- batches of 100 messages (newline-delimited) --")
batches = [b"\n".join(test[i:i + 100]) for i in range(0, len(test), 100)]
for name in ["raw JSON", "zlib-6 (deflate)", "zstd-3", "zstd-3 + 4 KB dict", "lz4 frame"] + (["snappy"] if snappy else []):
size, us = per_message(codecs[name], batches)
print(f"{name:22s} {size/100:7.1f} B/msg {us:7.1f} us/batch")
def gorilla_bits(ts, vals):
"""Simplified Gorilla (VLDB 2015) bit count for timestamps + float64 values."""
bits = 64 + 64
prev_delta = None
for i in range(1, len(ts)):
d = ts[i] - ts[i - 1]
if prev_delta is not None:
dd = d - prev_delta
bits += (1 if dd == 0 else 9 if -63 <= dd <= 64 else
12 if -255 <= dd <= 256 else 16 if -2047 <= dd <= 2048 else 36)
else:
bits += 14
prev_delta = d
prev = struct.unpack("<Q", struct.pack("<d", vals[0]))[0]
lead_prev, trail_prev = -1, -1
for v in vals[1:]:
cur = struct.unpack("<Q", struct.pack("<d", v))[0]
x = prev ^ cur
if x == 0:
bits += 1
else:
lead = min(64 - x.bit_length(), 31)
trail = (x & -x).bit_length() - 1
if lead_prev >= 0 and lead >= lead_prev and trail >= trail_prev:
bits += 2 + (64 - lead_prev - trail_prev)
else:
bits += 2 + 5 + 6 + (64 - lead - trail)
lead_prev, trail_prev = lead, trail
prev = cur
return bits
def zigzag(n): return (n << 1) ^ (n >> 63)
def varints(nums):
out = bytearray()
for n in nums:
n = zigzag(n)
while n >= 0x80:
out.append((n & 0x7F) | 0x80); n >>= 7
out.append(n)
return bytes(out)
print("\n-- one batch of 100, columnar (ts + temperature + vibration) --")
col_sizes, gor_sizes = [], []
for i in range(0, len(test_objs), 100):
b = test_objs[i:i + 100]
ts = [m["ts"] for m in b]
t = [round(m["temperatureC"] * 100) for m in b]
v = [round(m["vibrationRmsMmS"] * 1000) for m in b]
dts = [ts[0]] + [ts[j] - ts[j - 1] for j in range(1, len(ts))]
ddts = dts[:2] + [dts[j] - dts[j - 1] for j in range(2, len(dts))]
payload = (varints(ddts) + varints([t[0]] + [t[j] - t[j-1] for j in range(1, 100)])
+ varints([v[0]] + [v[j] - v[j-1] for j in range(1, 100)]))
col_sizes.append(len(c_plain.compress(payload)))
gor_sizes.append((gorilla_bits(ts, [m["temperatureC"] for m in b]) +
gorilla_bits(ts, [m["vibrationRmsMmS"] for m in b])) / 8)
print(f"delta+zigzag varint + zstd-3: {statistics.mean(col_sizes)/100:.2f} B/msg (3 fields)")
print(f"Gorilla bits (ts counted twice, 2 series): {statistics.mean(gor_sizes)/100:.2f} B/msg")
Example run and how to read it
The table below is one example run of this harness, not a benchmark. The environment was a shared cloud container with 2 vCPUs (Intel Xeon at 2.10 GHz) running Python 3.13.18, zstandard 0.25.0 bundling libzstd 1.5.7, lz4 4.4.5 and python-snappy 0.7.3. Timings include Python call overhead, which dominates at these sizes, so treat them as relative only. The data is synthetic and far more regular than real plant data, which flatters every batched result.
| Method (example run, synthetic data) | Bytes per message | Time per call |
|---|---|---|
| Raw JSON, per message | 136.8 | n/a |
| CBOR, no compression | 116.0 | n/a |
| zlib level 6, per message | 128.2 | ~10 µs |
| zstd level 3, per message | 129.2 | ~2.9 µs |
| zstd level 3 + 4 KB dictionary, per message | 30.8 | ~0.8 µs |
| LZ4 block, per message | 138.8 | ~0.6 µs |
| LZ4 block + 4 KB dictionary, per message | 33.0 | ~1.4 µs |
| LZ4 frame, per message | 159.8 | ~0.5 µs |
| Snappy, per message | 140.8 | ~0.4 µs |
| zlib level 6, batch of 100 | 9.5 | ~57 µs per batch |
| zstd level 3, batch of 100 | 8.8 | ~20 µs per batch |
| zstd level 3 + dictionary, batch of 100 | 9.7 | ~20 µs per batch |
| LZ4 frame, batch of 100 | 17.9 | ~9 µs per batch |
| Snappy, batch of 100 | 18.6 | ~10 µs per batch |
| Columnar delta + zigzag varint + zstd-3, batch of 100 (3 fields) | 2.47 | n/a |
| Simplified Gorilla bit count, 2 float series with timestamps | 14.07 | n/a |
Three IoT data compression patterns are worth carrying away, and they should survive on real data even though the exact numbers will not. First, per-message compression without a dictionary is roughly break-even, and framed formats can expand data. Second, a trained dictionary recovers most of the batching benefit for single messages, about a 4-times reduction here. Third, the dictionary stops helping once batches are large enough to supply their own history. The batched dictionary result was slightly worse than plain batched Zstandard in this run.
The Gorilla line needs a caveat. It counts timestamps once per series, and it runs on doubles that came from two- and three-decimal rounding. That is close to the worst case for XOR encoding, as the ALP authors point out. Gorilla on true sensor doubles, or with timestamps shared across series, will do better. The point is narrower: on decimal telemetry you control, scaled-integer deltas beat XOR floats.
Methodology caveats
For IoT data compression testing, synthetic random walks are a weak proxy. They have no outliers, missing samples, mode changes or string fields with real variety. Real fleets have all of these, and each one costs ratio. Time per call in Python mostly measures interpreter overhead, not codec throughput; use the C command-line tool’s -b benchmark mode, or your target’s native build, for throughput. Dictionary results depend heavily on the similarity between training and test traffic. Train on one week and test on the following week, never on a random split of the same day.
Bandwidth and Cost Maths for a Cellular Fleet
To see why framing dominates IoT data compression economics, put the example-run sizes into a fleet model. Every figure in this section is illustrative. The protocol overheads are first-principles estimates, the fleet is hypothetical, and the tariff is a placeholder you should replace with your contract rate.
Assume 5,000 cellular gateways, each sampling once every 10 seconds. That is 8,640 messages per device per day, or 259,200 per 30-day month. Assume an always-on TLS 1.3 session over TCP, MQTT QoS 0, and a 30-byte topic name. Per publish, the overhead estimate is about 34 bytes of MQTT (fixed header, topic length and topic) and 22 bytes of TLS 1.3 record (5-byte header, 1-byte inner content type, 16-byte AES-GCM tag). Add 40 bytes of IPv4 and TCP headers and roughly 40 bytes for the returning TCP acknowledgement, which most carriers also bill. That totals about 136 bytes of overhead per publish, before any keep-alives, retransmissions or reconnect handshakes.
| Scenario (illustrative) | Bytes on wire per publish | Publishes per device per month | MB per device per month | Fleet GB per month | Fleet cost at an illustrative $0.05/MB |
|---|---|---|---|---|---|
| Raw JSON, one message per publish | 273 | 259,200 | 70.8 | 354 | ~$17,700 |
| CBOR, one message per publish | 252 | 259,200 | 65.3 | 327 | ~$16,300 |
| zstd + dictionary, one message per publish | 167 | 259,200 | 43.3 | 216 | ~$10,800 |
| Raw JSON, 100 per publish | 13,916 | 2,592 | 36.1 | 180 | ~$9,000 |
| LZ4 frame, 100 per publish | 1,926 | 2,592 | 5.0 | 25 | ~$1,250 |
| zstd level 3, 100 per publish | 1,016 | 2,592 | 2.6 | 13 | ~$660 |
| Columnar delta + zstd, 100 per publish | 383 | 2,592 | 1.0 | 5 | ~$250 |
The ordering is the useful output, not the dollar figures. Moving from JSON to CBOR saves about 8%, because the overhead dwarfs the key savings. A trained dictionary saves about 39% without changing latency. Batching uncompressed JSON saves about half, simply by paying protocol overhead 100 times less often. Batching plus a codec cuts traffic by more than an order of magnitude. Batching plus a columnar transform cuts it by about 70 times.
Two second-order effects push the same way. Fewer, larger transmissions let the radio sleep longer, which on LTE-M and NB-IoT is usually worth more battery life than the CPU cost of compression. Our guide to cellular IoT power saving with PSM, eDRX and RAI explains why. On narrow links, smaller payloads also cut the airtime per message. Our LoRaWAN vs NB-IoT vs LTE-M comparison shows how payload limits differ by radio. On LoRaWAN, with payloads capped at tens to a couple of hundred bytes depending on data rate, a trained dictionary can be the difference between one frame and two.
Dictionary Lifecycle: Zstandard Dictionary Compression in Production
Zstandard dictionary compression turns a codec into a protocol. Both ends must agree on exactly which bytes form the dictionary, forever, for every message in flight or in storage. That is a versioning problem, not a compression problem, and it is where most teams get hurt.

Figure 2: Dictionary lifecycle, from training on real traffic to rollout, decoding and retraining on ratio drop.
Figure 2 traces one dictionary generation. A trainer builds dictionary 41 from samples captured on firmware 4.2 and publishes it to a registry. Gateways receive it, signed, with their configuration update and announce the active ID. Every frame carries that ID, so the decoder can fetch an uncached dictionary. When the decoder’s measured ratio falls, the trainer builds dictionary 42 while 41 stays available.
Identify the dictionary in every message
RFC 8878 lets a Zstandard frame carry the Dictionary_ID in its header, using 0, 1, 2 or 4 bytes. The zstd manual notes that IDs below 256 or 65,536 need fewer header bytes, so allocate small IDs deliberately. The LZ4 frame format has an equivalent 4-byte Dict-ID field. If you strip the ID to save bytes, as the harness does with write_dict_id=False, you must carry it elsewhere. Options include an MQTT 5 user property, a topic segment, or a one-byte prefix in your own envelope.
MQTT 5 gives you two standard hooks for describing a compressed payload. The Payload Format Indicator is 0 for an unspecified byte stream and 1 for UTF-8. A compressed payload must use 0, because compressed bytes are not valid UTF-8. The Content Type property is a free-form UTF-8 string, so application/zstd (the media type registered by RFC 8878) is a sensible value. A dictionary-ID suffix is your own convention, not a standard. Remember that each property costs bytes on every publish, which is why a compact one-byte envelope often wins on constrained links.
Train on the right data
Train on traffic from the same firmware, schema and message type you will compress. A dictionary trained on mixed message types spends its budget on content that most messages never reference. Prefer one small dictionary per message family over one large dictionary for everything. For gateways, sizes from 2 to 16 KB usually capture field names and enum strings. Larger dictionaries mostly help when messages contain long repeated text, and they cost memory on every decoder.
Use a time-based holdout. Train on days 1 to 7 and measure on day 8, then compare with your current dictionary on the same data. Promote the new dictionary only if it improves the ratio by a meaningful margin, because every promotion has rollout cost.
Keep old dictionaries forever
Data compressed with dictionary 41 can only be decompressed with dictionary 41. If you archive compressed frames in object storage, or a device buffers them during a week-long outage, the decoder must still have that dictionary years later. Treat the registry like a schema registry: immutable entries, signed content, and no deletion. In practice, most teams decompress at ingestion and re-encode for storage, which limits the long-term dependency to the ingestion window.
Decision Matrix: Zstd vs LZ4 vs Snappy vs Gorilla-Style Encoding
The “zstd vs lz4” IoT data compression debate resolves quickly once you know the device class and whether you can batch. Figure 3 is the decision flow we use.

Figure 3: Decision flow by latency budget, payload shape and device class.
Read Figure 3 top-down. If you can batch within your latency objective and the fields are mostly numeric series, use a columnar delta transform followed by low-level Zstandard. If you can batch but the data is irregular, batch and apply Zstandard level 3, or LZ4 on weak CPUs. If you cannot batch and messages are small and similar, use a shared dictionary: Zstandard on Linux gateways, LZ4 block mode on microcontrollers. If you cannot batch and messages are dissimilar, skip the codec and rely on a compact encoding.
| Option | Ratio on small messages | Ratio on batches | Compression speed | Decoder memory | Dictionary support | Best fit |
|---|---|---|---|---|---|---|
| Zstandard, levels 1 to 3 | Poor without dictionary, strong with | Strong | Fast | Window plus context; small windows possible | Yes, with a trainer (--train) and frame dictionary ID |
Linux gateways, cloud ingestion |
| Zstandard, levels 19 to 22 | Same as above | Strongest of the general codecs | Slow | High at ultra levels | Yes | Cold archives, not live telemetry |
| LZ4 block or frame | Poor without dictionary, good with | Moderate | Very fast | Minimal; up to 64 KB history | Yes, up to 64 KB, no official trainer | MCUs, CPU-starved gateways |
| LZ4HC | Same as LZ4 | Better than LZ4 | Slow | Same as LZ4 | Yes | Data compressed once, decoded on MCUs |
| Snappy | Poor | Moderate | Very fast | Small | No | Cloud pipelines that already use it |
| Deflate (zlib or gzip) | Poor | Good | Moderate | 32 KB window | Preset dictionary, no trainer | Maximum compatibility |
| Brotli | Fair on text, via static dictionary | Strong on text | Slow at high levels | Window plus static dictionary | Built-in static; custom is newer and less portable | Static web assets, rarely telemetry |
| Gorilla, Chimp, ALP | Not applicable per message | Excellent on regular numeric series | Fast | Tiny | Not applicable | Columnar blocks and TSDB storage |
| Delta plus zigzag varint, then a codec | Not applicable per message | Excellent on decimal telemetry | Fast | Tiny plus codec | Codec-dependent | Batched gateway uploads |
A few cells deserve justification. LZ4’s lack of an official trainer is less of a handicap than it sounds. A Zstandard-trained dictionary’s raw content section, or simply a concatenation of typical messages, works as an LZ4 dictionary, as the harness does. Zstandard’s small-message advantage over LZ4 comes from its entropy stage and pre-computed tables in the dictionary. On the 137-byte example the gap was only about 7%. On larger or more varied payloads the gap usually widens in Zstandard’s favour.
For time series compression, Gorilla-style encodings are not competitors to Zstandard; they stack with it. Storage engines often apply a Gorilla-like transform and then a general codec on top, and Parquet combines BYTE_STREAM_SPLIT or delta encodings with a page-level codec such as Zstandard or Snappy. On the wire, a gateway can do the same thing. Transpose and delta-encode the batch, then hand the result to the codec. For a deeper look at how columnar formats layer these steps, see our Parquet vs ORC vs Lance vs Vortex comparison.
Microcontroller memory budgets
On a Cortex-M class device with 64 to 256 KB of RAM, memory decides the IoT data compression codec before ratio does. LZ4 block compression needs its hash table (16 KB at the default LZ4_MEMORY_USAGE, tunable down at a ratio cost) plus input and output buffers. A dictionary lives in flash and costs no RAM if the library can reference it in place. LZ4 decompression needs only the buffers, which is why many bootloaders use it for firmware images.
Zstandard can run on microcontrollers, but plan it deliberately. Compression and decompression contexts scale with the window and the chosen parameters. The library exposes ZSTD_estimateCCtxSize and ZSTD_estimateDCtxSize style functions in its advanced API so you can size buffers at build time; measure for your exact parameters rather than trusting a blog figure. Code size is also larger than LZ4’s, which matters on parts with small flash. If your device only sends and never receives compressed data, the device needs the compressor only, and the large-window decoder lives in the cloud.
Ultra-constrained parts below about 32 KB of RAM often get the best IoT data compression from no general codec at all. A hand-written delta and varint encoder over a fixed schema is a few dozen lines, needs bytes of state, and captures most of the redundancy in numeric telemetry. Tiny LZSS libraries designed for embedded use, with windows of a few hundred bytes, sit between the two.
Trade-offs, Gotchas, and What Goes Wrong
IoT data compression changes your system’s failure surface. The five failure modes in Figure 4 are the ones we see most often in field telemetry.

Figure 4: Common IoT data compression failure modes and the mitigation for each.
Dictionary drift. A firmware update renames temperatureC to temp_c or adds a field, and the dictionary’s matches stop matching. Nothing breaks and every message still decodes, but the ratio silently falls toward the no-dictionary baseline and the data bill climbs. Track compressed-to-raw ratio per dictionary ID and firmware version as a first-class metric. Alert when it degrades, and tie dictionary versions to schema versions in your release process.
Compressing after encryption. Encrypted output is indistinguishable from random bytes, so compressing it wastes CPU and adds framing. Compression must happen before encryption. TLS 1.3 removed protocol-level compression entirely, so on modern stacks the application layer is the only place it can happen. Our guide to TLS 1.3 handshakes and mTLS on constrained IoT devices covers the record-layer overheads that make batching matter.
Mixing secrets and attacker-influenced data in one compression context. The CRIME (2012) and BREACH (2013) attacks showed that compressed size leaks information. If an attacker can inject chosen bytes into the same compressed stream as a secret, they can guess the secret by watching lengths. Telemetry is usually low risk because it rarely contains secrets. Command and configuration channels, which may carry tokens alongside user-supplied strings, are a different matter. Do not compress those, or keep secrets in a separate context.
CPU and energy on battery devices. Compression costs cycles, and cycles cost charge. On cellular devices the radio usually dominates, so fewer, smaller transmissions win, but this is not universal. On short-range radios with cheap transmissions, or on very slow cores, the CPU cost can exceed the radio saving. Measure energy per byte saved on your hardware with a current probe before you commit.
Frame overhead on tiny payloads. Framed formats add magic numbers, headers, checksums and end marks. A 20-byte sensor reading can grow by a third once wrapped in an LZ4 frame. Use raw block formats on constrained links, carry sizes and dictionary IDs in your own compact envelope, or do not compress messages below a measured threshold.
Two smaller IoT data compression gotchas round this out. Decompression bombs are real: a hostile or corrupted frame can claim a huge output size, so always cap output size, as the harness’s max_output_size does. And Zstandard’s content checksum is optional. If you disable it to save four bytes, make sure your transport or envelope provides integrity, which TLS does in transit but not in storage.
Practical Recommendations
Start any IoT data compression project by measuring where your bytes go, not by picking a codec. Capture a week of real payloads from each message family, then compute overhead per publish on your actual stack. On most cellular fleets, that single exercise reveals that framing and protocol overhead dominate. Telemetry bandwidth reduction then becomes a batching and representation project first and a codec project second.
When you do add a codec, keep the IoT data compression configuration boring. Zstandard level 1 to 3 on gateways, LZ4 block mode on microcontrollers, and trained dictionaries for any path that cannot batch. Avoid high levels for live telemetry, because the ratio gains on small batches are marginal and the CPU cost is not. Keep compression at the edge symmetric with what your cloud ingestion already decodes. Kafka, for example, supports gzip, Snappy, LZ4 and Zstandard natively, which our MQTT to Kafka bridge tutorial builds on.
Finally, treat IoT data compression as part of your data contract. Version dictionaries with schemas, publish ratio metrics, and test round-trips in CI. A compression change that saves 40% of the data bill but silently corrupts one message in a million is not a saving.
Checklist:
- Apply deadband or exception filtering before any IoT data compression step.
- Replace JSON with CBOR or Protobuf where you control both ends.
- Batch to the largest window your latency objective allows.
- Transpose batches into columns and delta-encode numeric fields as scaled integers.
- Use Zstandard levels 1 to 3 on gateways and LZ4 block mode on MCUs.
- Train per-message-family dictionaries for unbatched paths, with time-based holdouts.
- Carry a dictionary ID in every message and never delete old dictionaries.
- Set MQTT 5 Payload Format Indicator to 0 and a meaningful Content Type.
- Compress before encryption, and never mix secrets with attacker-influenced data.
- Cap decompressed output size and monitor the IoT data compression ratio per dictionary ID.
Frequently Asked Questions
Is Zstandard or LZ4 better for IoT data compression?
For IoT data compression, it depends on the device class and payload shape. On Linux gateways, Zstandard at levels 1 to 3 usually gives a better ratio at acceptable CPU cost, especially with a trained dictionary. On microcontrollers, LZ4 block mode is simpler, has a small fixed memory footprint and decompresses with almost no state. In our example run on 137-byte messages, both reached about 31 to 33 bytes with a 4 KB dictionary. Without a dictionary, neither saved meaningful bytes. Measure both on your real payloads before deciding.
Why does compressing a single small MQTT message barely reduce its size?
LZ77-style compressors save bytes by referencing earlier occurrences of the same data, and a single small message has almost no earlier data to reference. Every field name and value appears for the first time. Framed formats also add their own headers, magic numbers and checksums, which can make the output larger than the input. The two fixes are batching, so later messages reference earlier ones, and trained dictionaries. A dictionary pre-loads typical content so even the first message finds matches.
How does Gorilla time series compression work?
Gorilla, from Facebook’s VLDB 2015 paper, compresses timestamps and float values separately. Timestamps use delta-of-delta encoding: if samples arrive at a steady interval, each one costs a single zero bit, with tagged buckets for small deviations. Values are XOR-ed with the previous value; an identical value costs one bit, and a changed value stores only its meaningful bits. The paper reported about 1.37 bytes per 16-byte point on Facebook’s monitoring data. Results on decimal-origin sensor data are usually worse, which motivated later schemes such as Chimp and ALP.
How big should a Zstandard dictionary be for telemetry?
For telemetry messages under 1 KB, dictionaries of about 2 to 16 KB usually capture field names, enum strings and common values. The zstd command-line trainer defaults to a 110 KiB maximum. It recommends more than 100 samples, about 100 times the dictionary size in total. Larger dictionaries rarely help short structured messages and cost memory on every decoder. Train one dictionary per message family and evaluate on a later time window than the training data.
Should I compress before or after TLS encryption?
Always compress before encryption. Encrypted data looks random, so compressing it achieves nothing and wastes CPU. TLS 1.3 no longer offers protocol-level compression, so compression must happen in your application payload before it is handed to the TLS stack. Be careful when one compressed stream mixes secrets, such as tokens, with data an attacker can influence. The CRIME and BREACH attacks showed that compressed length can leak those secrets. Telemetry channels are usually low risk; command channels deserve more care.
Does MQTT have built-in payload compression?
No. MQTT treats the payload as opaque bytes, and IoT data compression is left to the application. MQTT 5 helps you describe compressed payloads. Set the Payload Format Indicator to 0 (unspecified bytes), because compressed data is not valid UTF-8, and use the Content Type property, for example application/zstd. You can carry a dictionary ID in a user property or in your own envelope. Each property adds bytes to every publish, so on constrained links a compact one-byte envelope often works better.
Further Reading
- MQTT store-and-forward edge gateways: offline buffering, broker bridging and QoS
- Time-series database internals: InfluxDB vs TimescaleDB vs QuestDB
- LoRaWAN vs NB-IoT vs LTE-M for industrial IoT
- Gorilla delta-of-delta and XOR time-series compression for IoT telemetry
- RFC 8878: Zstandard Compression and the application/zstd Media Type
- Gorilla: A Fast, Scalable, In-Memory Time Series Database (VLDB 2015)
By Riju — about
