Time Series Compression for IoT Telemetry: Gorilla, Delta-of-Delta, XOR and Swinging-Door

A single temperature sensor sampled once per second produces 86,400 points a day. Stored naively as a 64-bit timestamp plus a 64-bit float, that is 16 bytes a point, or about 1.4 MB per sensor per day. Multiply by 10,000 sensors and you are writing roughly 13.8 GB a day before indexes, replicas or backups. The surprising part is that most of those bits carry no information: the timestamps are almost perfectly regular, and the values barely move between samples.

Time series compression exploits exactly that redundancy, and the best-known scheme, Facebook’s Gorilla, reported an average of 1.37 bytes per data point, a 12x reduction, on production monitoring data. It matters now because edge fleets keep growing while cellular, satellite and historian storage budgets do not. This post takes the mechanisms apart at the bit level, shows where they break on real IoT signals, and ends with working Python you can run.

You will leave knowing how delta-of-delta and XOR encoding work, why a decimal value like 21.6 is expensive in a float, how lossy swinging-door trending interacts with lossless coding, and which scheme to put at the edge versus in the database.

What this covers: the Gorilla bit layout, a worked bit-count example, lossy methods (deadband, swinging door), newer float codecs (Chimp, Elf, ALP), how Prometheus, TimescaleDB and InfluxDB apply them, edge-side design, failure modes, and runnable encoders.

Context and Background

Telemetry has two properties that general-purpose compressors such as gzip or zstd handle poorly in a streaming setting. First, it arrives as an append-only stream of (timestamp, value) pairs that a database must make queryable within seconds, so you cannot wait to compress a large file. Second, the structure lives in numeric relationships between neighbours, not in repeated byte strings. A general compressor sees 8-byte floats whose bytes look random; a specialised one sees that today’s value differs from the last by a handful of low mantissa bits.

The reference design is the Gorilla paper, “Gorilla: A Fast, Scalable, In-Memory Time Series Database” by Pelkonen et al. (Proceedings of the VLDB Endowment, vol. 8, no. 12, 2015). Gorilla kept the most recent 26 hours of monitoring data in memory, grouped into two-hour blocks, and compressed each series with two tricks: delta-of-delta for timestamps and XOR for values. The authors report that about 96% of timestamps compress to a single bit and about 51% of values compress to a single bit. The paper is the origin of what most engineers now simply call Gorilla compression. You can read it at the VLDB proceedings.

Gorilla’s ideas now sit inside most mainstream engines. Prometheus’s TSDB chunk format stores a variable-width delta-of-delta timestamp and a variable-width XOR value per sample, as documented in the Prometheus chunk format notes. TimescaleDB uses delta-of-delta with Simple-8b and run-length encoding for integer-like columns and a Gorilla-style XOR scheme for floats, per Timescale’s own explainer on compression algorithms. InfluxDB’s TSM engine uses delta encoding with Simple-8b or run-length encoding for timestamps and Facebook’s float encoding for floats, as described in the original TSM design post. For how these engines organise storage around the codecs, see our internals comparison of InfluxDB, Timescale and QuestDB and the industrial IoT time-series platform architecture.

Industrial historians took a different road, long before Gorilla. Plant historians such as the PI System use exception reporting and compression based on swinging-door trending, a lossy method that keeps only the points needed to rebuild a signal within a tolerance. Those two families, lossless bit-packing and lossy point selection, solve different problems and, as shown later, they interact in ways that surprise people who stack them.

One framing runs through the rest of the post. A time series codec is a bet about your data. Gorilla bets that timestamps are regular and values change slowly or repeat. When the bet is right you pay one or two bits per field. When it is wrong, the same scheme can cost more than raw storage. Knowing your signal’s shape matters more than picking the fashionable algorithm.

The Core Architecture: Splitting Timestamps from Values

Time series compression in the Gorilla family treats timestamps and values as two independent streams: timestamps are stored as the change in their spacing (delta-of-delta), and values are stored as the XOR against the previous value, with leading and trailing zero bits dropped. Both streams are bit-packed into one append-only block per series.

Figure 1: The Gorilla-style pipeline. Each sample is split, encoded with a field-specific scheme, and appended to a bit stream that is sealed at a time boundary.

Figure 1 shows the flow. A sample enters, its timestamp and value take separate paths, and the resulting variable-length bit strings are concatenated by a bit packer into a block covering a fixed window, two hours in the original system. When the window closes, the block is sealed and flushed. The encoder is O(1) in time and memory per sample, which is why it works for streaming ingestion on small hardware.

Delta-of-delta for timestamps

Delta encoding stores the difference between consecutive timestamps. Delta-of-delta stores the difference between consecutive deltas. If a sensor reports every 10 seconds, the deltas are all 10 and the delta-of-delta values are all 0. The formula from the paper is D = (t_n – t_(n-1)) – (t_(n-1) – t_(n-2)).

In Gorilla, a block header holds the start time aligned to the two-hour window, and the first timestamp is stored as a 14-bit delta from that start. Every later timestamp is coded by the size of D, using a prefix code. As I read the paper’s table (the page summaries I fetched did not reproduce it, so verify against Figure 3 of the paper before copying constants):

D = 0: the single bit 0.
D in [-63, 64]: control bits 10 followed by 7 value bits.
D in [-255, 256]: control bits 110 followed by 9 value bits.
D in [-2047, 2048]: control bits 1110 followed by 12 value bits.
Anything else: control bits 1111 followed by 32 value bits.

The design is a Huffman-flavoured prefix code chosen empirically: small jitter is common, large gaps are rare. A regular series costs 1 bit per timestamp, which is a 64x reduction from a raw 64-bit timestamp. A series with 1 ms of jitter around a 1-second period costs 9 bits for most points, still a 7x gain.

Prometheus uses the same idea with different buckets. Its chunk documentation says delta-of-delta values use 1 to 68 bits and XOR values 1 to 77 bits, which is consistent with a 4-bit prefix plus a 64-bit escape (68) and the 2-bit prefix, 5-bit leading-zero field, 6-bit length field and 64 bits of payload (77). I recall the intermediate bucket widths in the source as 14, 17 and 20 bits, but I did not verify that in this run, so treat those as unconfirmed.

XOR for floating-point values

Floating-point numbers follow IEEE 754: one sign bit, an 11-bit exponent and a 52-bit mantissa for a double. Two nearby values share the sign, usually share the exponent, and share the high mantissa bits. XOR of their raw 64-bit patterns therefore has many leading zeros. If the values are identical the XOR is zero.

Gorilla’s value coding works like this. The first value in a block is stored raw. For each later value, compute x = bits(v_n) XOR bits(v_(n-1)).

If x is 0, write a single 0 bit.
Otherwise write 1, then a second control bit. If the meaningful bits (from the first set bit to the last set bit) fit inside the previous meaningful window, write 0 and then just the bits of that window.
If they do not fit, write 1, then 5 bits for the leading-zero count, 6 bits for the length of the meaningful run, then the meaningful bits themselves.

Figure 2 lays out that decision tree. The costs are easy to count. A repeated value costs 1 bit. A value that fits the previous window costs 2 bits plus the window width. A value that needs a new window costs 2 + 5 + 6 = 13 bits plus the new width. The 5-bit leading-zero field caps the stored count at 31, so a value with more than 31 leading zeros is recorded as 31 and carries a few extra zero bits.

Figure 2: The XOR value decision tree. Cost grows from 1 bit for a repeat to 13 bits of overhead plus payload when the meaningful window changes.

The paper’s reported distribution on Facebook monitoring data is concrete: about 51% of values took the single-bit path, about 30% used the reuse-window case at an average of 26.6 bits, and about 19% used the new-window case at an average of 36.9 bits. (The article I fetched labels the two non-zero cases “10” and “11” while describing them as “0” and “1” after the leading control bit; the naming is inconsistent but the cost figures are not in doubt.) Those figures explain the headline of 1.37 bytes per point: when more than half your values are repeats and nearly all your timestamps are regular, the average collapses.

Why the format forces a block design

Because each value is coded relative to its predecessor, you cannot decode point 500 without decoding points 1 to 499 in the block. Gorilla accepts that: blocks are small (two hours), queries decode entire blocks, and in-memory scans are fast. The same property forces append-only writes, and it is why out-of-order samples are so troublesome, a point returned to in the failure-modes section. It also explains why compression is per-series: mixing series in one stream destroys the neighbour correlation the scheme relies on.

Deeper Analysis: A Bit-Level Walk-Through and Where the Ratio Comes From

Abstract descriptions hide the interesting cost structure, so here is a concrete run. I implemented the Gorilla-style encoder below (code appears in the next section) and encoded eight samples of an indoor temperature sensor with a 10-second period and a single 1-second jitter event. These numbers come from my own illustrative run, not from a benchmark suite.

Timestamps: 1000, 1010, 1020, 1030, 1041, 1051, 1061, 1071. Values: 21.5, 21.5, 21.5, 21.6, 21.6, 21.5, 21.5, 21.5.

Point	Timestamp cost	Value cost	Total bits	Why
1	64 raw	64 raw	128	Header values stored uncompressed
2	14 (first delta)	1	15	Value repeated, XOR is zero
3	1	1	2	Delta-of-delta 0, value repeated
4	1	57	58	Value moved 21.5 to 21.6, new window
5	9	1	10	Jitter of 1 s gives D = +1, so prefix `10` plus 7 bits
6	9	46	55	D = -1; value returns, window reused
7	1	1	2	Back to steady state
8	1	1	2	Steady state

The eight points cost 272 bits in total, or 34 bytes, against 128 bytes raw (1,024 bits): 3.76x. That small ratio is dominated by two things. The 128-bit header is amortised over only eight points, and the one value change costs 57 bits. Over a two-hour block of 720 points the header becomes noise. The value change is the expensive event, and it is worth seeing why.

Why 21.6 costs 57 bits

The decimal 21.5 is exactly representable in binary (it is 10101.1), so its double is 0x4035800000000000: the mantissa has only a few set bits. But 21.6 is not exactly representable. Its double is 0x403599999999999A, and the mantissa is the repeating pattern 1001 1001 and so on. XOR of the two is 0x19999999999A. That has 19 leading zeros and 1 trailing zero, leaving a 44-bit meaningful run. The cost is 1 (non-zero) + 1 (new window) + 5 (leading zeros) + 6 (length) + 44 (payload) = 57 bits. Gorilla did not save much here: the value moved by 0.1, and 0.1 in binary is a full-width mantissa.

This is the most under-appreciated fact about XOR floating point compression on IoT data. Sensors measure integers: ADC counts, or a decimal reading with fixed resolution such as 0.1 degrees. Converting to a float introduces the non-terminating binary fraction. A series that moves by 0.1 steps is a series of integer steps at heart, but in float form every step is a dense bit pattern. Compression lives or dies on whether the value is a “short” float (a small integer, or a dyadic fraction like 0.5 or 0.25) or a decimal that was rounded.

Illustrative results on synthetic signals

To get a feel for ratios, I ran the same encoder over three synthetic 1,000-point series with a 1 Hz regular clock (seeded random generator, so reproducible, but these are not real sensor traces and the numbers are illustrative only):

Signal	Bytes per point	Ratio vs 16 B raw	Comment
Temperature at 0.1 resolution, 8% chance of a 0.1 step per second, 11 distinct values	0.86	about 18.5x	Many repeats; window reuse works
Float signal: slow sine plus Gaussian noise, sigma 0.05, full precision	6.76	about 2.4x	Noise fills the mantissa; XOR finds little
Case 1 as scaled integers (value x 10), delta, zigzag, varint	1.00	about 16x	Byte-granular, cannot go below 1 byte

The comparison between rows one and three matters. Delta coding of scaled integers cannot beat one byte per sample with a varint, while bit-level XOR can fall under that for repetitive data. But when the data is noisy, as in row two, the XOR scheme’s advantage vanishes while a quantise-then-delta scheme would give you something close to the noise’s entropy. If you know the sensor’s true resolution, say a 12-bit ADC, rounding the noise away at the source is a lossless-with-respect-to-measurement operation, and it turns row two into a row-one problem. A 12-bit sample has at most 4,096 distinct levels, so a double carrying 64 bits for it is overbuilt by a factor of five before any codec runs.

Fixed-point and integer codecs: the other lane

TimescaleDB’s approach for integers shows the alternative path. It applies delta-of-delta, then packs the results with Simple-8b, a word-aligned scheme that stores as many same-width integers as fit into a 64-bit word and uses selector bits to record the width. It then run-length encodes repeated runs. In Timescale’s description, timestamps and integer-like types go through delta-of-delta plus Simple-8b with RLE; floats use the XOR scheme; low-cardinality columns use a dictionary; everything else falls back to an LZ-based array compression. The company also cites decode speeds “up to 40x faster than LZ-based compression” for the delta and XOR families. I treat that as a vendor claim, not an independent measurement.

Simple-8b is not bit-granular like Gorilla, but its word alignment allows SIMD-friendly decoding. For analytical scans that touch millions of values, decode throughput matters as much as ratio. The practical split in 2026 engines: row-oriented hot buffers use Gorilla-style bit-streams for tiny memory footprints, while columnar sealed segments use integer codecs plus vectorised float codecs.

Newer float codecs: Chimp, Elf and ALP

Gorilla’s XOR scheme has aged; several research codecs improved on it for specific data shapes.

Chimp (PVLDB 2022) analysed the properties of real time series and changed the control-bit layout. Its abstract says it needs, on average, around 50% of the space required by state-of-the-art streaming approaches and beats earlier techniques on compression and read time. The abstract does not name Gorilla specifically, so treat “half of Gorilla” as a likely, not confirmed, reading. See the Chimp paper page.

Elf (PVLDB 2023) erases the last few mantissa bits of values that were originally short decimals, so the XOR has many trailing zeros, then restores the exact original on decode. The authors claim better compression than nine competitors across 22 datasets and streaming operation with O(N) time and O(1) space. This directly attacks the 21.6 problem above. See the Elf paper page.

ALP (Afroozeh, Kuffo and Boncz, SIGMOD 2024) takes a different route: for doubles that began as decimals, it converts them to integers by multiplying by a power of ten, stores them with frame-of-reference and bit-packing, and keeps a separate path for doubles that are not decimal-origin. It is built for vectorised, auto-SIMD decoding rather than bit-serial streaming. The authors claim it beats Gorilla, Chimp128, Elf and Patas on speed and ratio, and DuckDB’s library page states that ALP is its default compression for floating-point numbers. See the DuckDB ALP page.

The unifying insight is the one from the 21.6 example: ALP is essentially the “scale to integers” trick applied automatically and in batches, while Elf achieves the same effect inside an XOR framework. I would not call these drop-in replacements for Gorilla in a write-heavy ingest path, since ALP works on vectors of values (batch-oriented) rather than one sample at a time. But for sealed, columnar segments they are the modern default.

Figure 3: A two-stage reduction. Deadband and swinging-door filters decide which points survive; a lossless Gorilla pass then packs the survivors.

Lossy Reduction: Deadband, Swinging Door and Their Interaction with Lossless Codecs

Lossless codecs hit a floor set by entropy. If a vibration sensor produces genuinely noisy samples, no bit-twiddling will give you 12x. Lossy methods change the question from “how do I encode every sample” to “which samples do I need to keep.”

Deadband (exception reporting)

The simplest rule: transmit or store a sample only if it differs from the last stored value by more than a threshold, the deadband. A temperature with a 0.2 degree deadband drops every reading within 0.2 of the last kept reading. The reconstruction rule is step-and-hold. It is cheap, needs one stored value of state, and fits a microcontroller with a few hundred bytes of RAM. The weakness is that it ignores trend: a slow ramp that never exceeds the deadband from the last kept value is dropped until it suddenly crosses, producing a late and steep jump on reconstruction. Process historians pair it with a maximum-time rule that forces a store after a set interval so that a flat line is distinguishable from a dead sensor.

Swinging-door trending (SDT) stores a piecewise-linear approximation. Take the last archived point as a pivot. Place an “upper door” hinged at the pivot offset by +dev and a “lower door” hinged at the pivot offset by -dev (dev is the compression deviation). As each new sample arrives, the doors swing: the upper door’s slope rises to stay below the point’s lower tolerance edge, and the lower door’s slope falls to stay above the point’s upper edge. While a straight line through the pivot can still pass inside every tolerance band seen so far, the doors stay open. When the doors cross, no single line fits, and the previous sample is archived as the new pivot. Figure 3 shows the control flow.

The Apache StreamPipes documentation describes the same idea as replacing a run of continuous points with a straight line defined by start and end points. Its parameters map onto the real-world historian configuration: compression deviation (the main knob), a minimum time interval that discards points arriving too soon after the last stored point, and a maximum time interval that forces a store. See the StreamPipes SDT processor docs. The PI System exposes a similar pair of settings; consult the AVEVA PI Server administration documentation for exact semantics, as I could not extract that page’s text in this run.

A subtlety I hit when implementing it: if you archive the actual last sample, the straight segment from the pivot to that sample can lie outside the door cone, because the endpoint’s own tolerance edges only constrain the doors, not the segment’s slope. In my first implementation, the maximum interpolation error reached 1.63 for a deviation of 1.0. The fix is to archive the projected value, the point on the mid-slope of the open doors at the last timestamp, rather than the raw reading. With that change the maximum error stayed at or below the deviation in every test I ran. Production historians differ on this detail; verify what yours stores, because “compression deviation” is not always the same as “worst-case error.”

Illustrative reduction numbers

On a synthetic 1,000-point signal (a sine of amplitude 10 around 50 with period about 251 samples, plus alternating noise of plus or minus 0.05), my swinging-door implementation kept 71 points at a deviation of 0.1 (about 14x), 26 points at 0.5 (about 38x) and 19 points at 1.0 (about 53x). These are illustrative and shape-dependent: a smooth sine is the best case.

A real-world data point comes from a Saudi Aramco paper on arXiv, “The Impact of Data Compression in Real-Time and Historical Data Acquisition Systems on the Accuracy of Analytical Solutions” (Faqehi et al.). Using swinging-door compression, they report temperature compressing by 96.8% but vibration by only 66.9%, because frequent fluctuations leave fewer collinear stretches. They also report that over-aggressive settings caused roughly 2.1% mean deviation, an 8% underestimate of standard deviation and 35% of transient anomalies missed. Their results come from a synthetic signal and two signal types, which the authors’ own caveat flags as limited. I include the figures because the direction is what matters: lossy compression is safest on smooth process variables and most dangerous on signals whose information is in the noise or the transients.

The interaction that surprises people

Stacking SDT in front of Gorilla looks like multiplication: 20x lossy times 10x lossless gives 200x. It does not work that way. SDT destroys the very regularity that Gorilla’s timestamp coding feeds on. After SDT the surviving timestamps are irregular, so delta-of-delta is rarely zero and the 1-bit timestamp path vanishes. Survivors also sit at turning points, so their values are less repetitive.

I measured this on the noisy float signal from the table above (illustrative, seeded synthetic). Lossless Gorilla alone gave about 6.76 bytes per point. SDT at a deviation of 0.1 kept 144 points, which Gorilla then encoded into about 1,116 bytes, or roughly 7.75 bytes per surviving point, a worse per-point cost than the lossless baseline. At deviation 0.3, 33 points cost about 274 bytes, about 8.3 bytes per survivor. The end-to-end ratio is still large (16,000 raw bytes down to about 1.1 KB and 0.27 KB, about 14x and 58x) because discarding points dominates. But the second stage contributes little, and it contributes least precisely where the first stage works best.

The lesson is to stop thinking of the stages as multiplicative. For the lossy stage, count the retained-point fraction. For the lossless stage, encode retained points with a codec that does not assume regular spacing: store explicit deltas of timestamps (varint) and scaled-integer values, or accept a per-point cost near 8 bytes and let a block-level compressor such as zstd mop up. Many historians do the sensible thing and simply archive timestamps and values in a plain columnar layout after exception filtering.

Implementation: Runnable Python Encoders

The snippets below are minimal, readable implementations. They are not production code: they use Python lists of bits for clarity, not a packed buffer.

import struct

class BitWriter:
    def __init__(self):
        self.bits = []
    def write(self, value, n):
        for i in range(n - 1, -1, -1):
            self.bits.append((value >> i) & 1)
    def __len__(self):
        return len(self.bits)

def f2b(x):                       # float -> 64-bit pattern
    return struct.unpack(">Q", struct.pack(">d", x))[0]

def lz(x): return 64 - x.bit_length()
def tz(x): return (x & -x).bit_length() - 1

def enc_ts(w, prev_delta, delta):
    d = delta - prev_delta        # delta-of-delta
    if d == 0:
        w.write(0, 1)
    elif -63 <= d <= 64:
        w.write(0b10, 2);   w.write(d + 63, 7)
    elif -255 <= d <= 256:
        w.write(0b110, 3);  w.write(d + 255, 9)
    elif -2047 <= d <= 2048:
        w.write(0b1110, 4); w.write(d + 2047, 12)
    else:
        w.write(0b1111, 4); w.write(d & 0xFFFFFFFF, 32)

def enc_val(w, prev, cur, st):
    x = f2b(prev) ^ f2b(cur)
    if x == 0:
        w.write(0, 1); return
    w.write(1, 1)
    l, t = min(lz(x), 31), tz(x)
    if st["l"] is not None and l >= st["l"] and t >= st["t"]:
        w.write(0, 1)             # reuse previous window
        w.write(x >> st["t"], 64 - st["l"] - st["t"])
    else:
        n = 64 - l - t
        w.write(1, 1); w.write(l, 5)
        w.write(n % 64, 6)        # 64 wraps to 0, as in the paper
        w.write(x >> t, n)
        st["l"], st["t"] = l, t

def compress(ts, vs):
    w = BitWriter()
    w.write(ts[0], 64); w.write(f2b(vs[0]), 64)
    prev_delta, st = 0, {"l": None, "t": None}
    for i in range(1, len(ts)):
        delta = ts[i] - ts[i - 1]
        if i == 1: w.write(delta, 14)
        else:      enc_ts(w, prev_delta, delta)
        prev_delta = delta
        enc_val(w, vs[i - 1], vs[i], st)
    return w

Check it against the walk-through: len(compress([1000,1010,1020,1030,1041,1051,1061,1071], [21.5,21.5,21.5,21.6,21.6,21.5,21.5,21.5])) returns 272 bits. A decoder mirrors the same branches; I leave it as an exercise, but any real implementation must test round-trips on adversarial inputs: NaN, negative zero, subnormals, and values whose XOR has 64 meaningful bits.

The swinging-door filter, with the projection fix described earlier:

def swinging_door(points, dev):
    """points: [(t, v)], t strictly increasing. Interpolation between
    returned points stays within dev of every input point."""
    if len(points) < 3:
        return list(points)
    t0, v0 = points[0]
    out = [(t0, v0)]
    hi, lo = float("-inf"), float("inf")
    prev_t = t0
    for t, v in points[1:]:
        dt = t - t0
        hi = max(hi, (v - dev - v0) / dt)
        lo = min(lo, (v + dev - v0) / dt)
        if hi > lo:                         # doors crossed
            slope = (prev_hi + prev_lo) / 2
            t0, v0 = prev_t, v0 + slope * (prev_t - t0)
            out.append((t0, v0))
            dt = t - t0
            hi = (v - dev - v0) / dt
            lo = (v + dev - v0) / dt
        prev_hi, prev_lo, prev_t = hi, lo, t
    slope = (prev_hi + prev_lo) / 2
    out.append((prev_t, v0 + slope * (prev_t - t0)))
    return out

Systems View: Where Each Codec Lives in an IoT Pipeline

The right question is not “which codec” but “which stage owns which transformation.” A typical industrial path has four places where bytes can be shrunk.

Figure 4: Where compression happens in an edge-to-database path. The gateway filters and batches; the database re-encodes into its own chunk format.

Stage 1: the sensor and microcontroller

Constrained devices have kilobytes of RAM, no floating-point unit in many cases and tight radio duty cycles. The right tool here is deadband and fixed-point scaling: keep the last reported value, send on change beyond a threshold or on a heartbeat, and transmit raw ADC counts or scaled integers instead of floats. The Gorilla bit-writer is feasible on a Cortex-M class chip, since the per-sample work is a few integer operations, but the real saving is in the radio: transmit time dominates energy cost. A rule of thumb to estimate: if your LPWAN or cellular payload has a fixed per-message overhead (headers, MQTT framing), batch size matters more than codec choice. Sixty 2-byte deltas in one message beat sixty messages of anything.

Stage 2: the edge gateway

A gateway usually has Linux, megabytes of memory and a view across many sensors. It is the right place for swinging door or deadband with per-tag parameters, for batching, and for store-and-forward. It must also keep a raw ring buffer until the upstream acknowledges, because lossy compression is irreversible and a rejected payload cannot be reconstructed from the compressed form. Keep per-tag compression parameters in configuration under version control; changing a deviation silently changes what your analytics see.

Stage 3: transport

MQTT and similar protocols already carry framing overhead, and TLS adds its own. Compressing a batch payload with a block compressor like zstd after the domain-specific codec is cheap and usually yields a modest extra gain on headers and metadata. Do not compress tiny single-sample messages; the overhead can exceed the payload.

Stage 4: the database

The database should own its chunk format. Prometheus seals XOR chunks in memory and persists them with a CRC-32C per chunk. TimescaleDB converts older chunks to a compressed columnar form per column type. InfluxDB’s TSM files compress per block by type. For your design, the key decision is whether to send compressed payloads from the edge and have the database decode and re-encode (simple, interoperable) or to push chunks in the database’s native format (fast, but couples the edge to a specific engine version). I favour decode and re-encode, because the cost is small relative to network savings and it keeps the edge codec independent. For sizing storage and retention policy around this, see our write-up on TimescaleDB hypertables, chunks, compression and continuous aggregates.

A worked storage estimate (illustrative)

Take 10,000 sensors at 1 Hz. Raw storage at 16 bytes per point is 10,000 x 86,400 x 16 = 13.8 GB per day. At the Gorilla paper’s 1.37 bytes per point (data-dependent; yours may be far worse) that is 1.18 GB per day, or about 432 GB per year. At my noisy-float figure of 6.76 bytes per point it is 5.8 GB a day, about 2.1 TB per year. The two estimates differ by a factor of five on the same sensor count, which is the real lesson: measure your own data. Run the encoder on a day of your actual traces before you size disks.

If a historian-style deadband removes 80% of samples first, the noisy case drops by the same factor, to about 1.2 GB a day, though the per-point cost of the survivors rises as discussed. These are planning numbers only.

Trade-offs, Gotchas, and What Goes Wrong

Out-of-order and late data. Delta-of-delta assumes monotonic, near-regular timestamps. A late point forces either a rewrite of the open block or a separate overflow structure. Gorilla-style engines typically reject or buffer out-of-order writes within a bounded window. If your devices buffer offline and replay hours later, expect either dropped samples or a heavier write path. Design a staging area that sorts before sealing a block.

Clock irregularity. Timestamps from devices with drifting clocks, or from event-driven reporting, do not give D = 0. The 1-bit path is replaced by 9, 12 or 36 bits. One practical trick is to quantise timestamps to the nominal period at ingest when the jitter is measurement noise and not information; another is to store the nominal period once per block and the small offsets separately. Be honest about whether the jitter carries meaning, since it can reveal scheduling problems.

Millisecond and nanosecond clocks. The Gorilla bucket sizes were tuned for second-resolution timestamps. A third-party Go implementation notes that millisecond timestamps cost more bits. With sub-second precision, jitter is larger in absolute terms and the small buckets are hit less often. Re-tune the bucket boundaries, or use a scheme with adaptive widths.

Noisy floats. As shown, full-precision noise defeats XOR. Do not buy a codec to fix a measurement problem. Round to the sensor’s resolution first.

NaN, infinity and missing data. NaN payload bits are rarely identical across samples, so a series that flickers between a real value and NaN costs full width each time. Use an explicit validity bitmap or a sentinel strategy rather than inline NaNs.

Random access and query cost. Bit-serial streams cannot be sliced. Reading one point costs decoding the block up to it. This is acceptable for dashboards that read ranges, and poor for point lookups or for parallel scans. Columnar vector codecs (ALP, Simple-8b with frame-of-reference) trade ratio for decodable positions.

Lossy compression and downstream analytics. The Aramco study’s numbers suggest the cost: understated variance and missed transients. Anomaly detection and predictive maintenance models depend on exactly the features lossy compression removes. Store raw data at least briefly, apply lossy reduction only to the long-term tier, and set deviation from the sensor’s accuracy specification rather than from a storage target.

Compression across tags is not free. Compressing each series independently loses cross-series redundancy: ten identical temperature probes in a vessel will move together. Few engines exploit that; if yours does not, a downsampled aggregate tier may serve better than a smarter codec.

Debuggability. A corrupt bit-stream block is nearly unreadable after the damage point, because every later value depends on its predecessor. Prometheus stores a CRC-32C per chunk for this reason. Always checksum sealed blocks and test your recovery path.

Practical Recommendations

Start from the signal, not the algorithm. Classify each tag as one of four types: slow-moving process variable (temperature, level, pressure), counter or cumulative total, high-rate vibration or waveform, and discrete state. Slow process variables suit deadband or swinging door at the edge plus Gorilla-class lossless coding in the database. Counters suit delta encoding, since the deltas are small non-negative integers. Waveforms are already near their entropy at full resolution; reduce them by sending features (RMS, FFT bands) at the edge or compress with a general-purpose codec, and do not apply swinging door. Discrete states suit run-length encoding and dictionaries.

Convert to scaled integers whenever the sensor resolution is known. That one decision often beats choosing a fancier float codec, and it makes the data friendlier to every downstream engine. Where you must keep floats, prefer an engine whose float codec understands decimal-origin values (ALP, or Elf-style erasing) for sealed segments.

Treat lossy parameters as data-governance decisions. Record the deviation per tag, tie it to the instrument accuracy, and keep a raw copy for a defined retention window. Test your analytics on compressed versus raw data before you commit, using the metrics the study above suggests: variance, peak values and anomaly recall.

Measure before you size. A day of real traces through the encoder in this post gives a defensible bytes-per-point figure in an hour.

[ ] Classify tags by signal type and pick a codec per class.
[ ] Round or scale values to sensor resolution before encoding.
[ ] Set swinging-door deviation from instrument accuracy, never from a storage target.
[ ] Keep a raw ring buffer at the gateway until the upstream acknowledges.
[ ] Checksum sealed blocks and test recovery from a corrupt block.
[ ] Define an out-of-order policy (window, staging, or reject) before go-live.
[ ] Benchmark your own data and record bytes per point per tag class.

Frequently Asked Questions

What is time series compression?

Time series compression reduces the storage and bandwidth of timestamped measurements by exploiting their regularity: timestamps are evenly spaced and consecutive values are similar. Lossless methods such as Gorilla’s delta-of-delta and XOR encoding reproduce every sample exactly. Lossy methods such as deadband and swinging-door trending discard samples that a straight line or step can reconstruct within a tolerance. Ratios depend heavily on the signal, from about 2x for noisy floats to over 10x for slow, repetitive data.

How does Gorilla compression work?

Gorilla splits each sample into a timestamp and a value. The timestamp is coded as the change in the gap between consecutive timestamps, which is zero for a regular clock and costs one bit. The value is XORed with the previous value; an identical value costs one bit, otherwise only the changed middle bits are stored along with a count of leading zeros and length. The original paper reported about 1.37 bytes per point on production monitoring data.

What is delta-of-delta encoding?

Delta-of-delta encoding applies differencing twice. The first difference is the gap between consecutive timestamps; the second difference is how much that gap changes. For a sensor reporting every 10 seconds, every gap is 10 and every second difference is 0, which can be stored in a single bit instead of 64. Jitter produces small non-zero values coded with short prefix codes. It requires near-regular, in-order timestamps to work well.

Is lossy compression safe for IoT sensor data?

It is safe only when the tolerance is set deliberately. Swinging-door trending bounds interpolation error by a chosen deviation, so smooth variables like temperature can be reduced heavily with little harm. Noisy or transient signals such as vibration compress less and lose more information. A Saudi Aramco study on arXiv found aggressive settings understated variance by about 8% and missed about 35% of transient anomalies, so keep raw data for analytics where transients matter.

What is the difference between Gorilla, Chimp and ALP?

Gorilla XORs each value with its predecessor in a bit-serial stream. Chimp, from a 2022 VLDB paper, refines the control-bit layout and reports around half the space of prior streaming approaches. ALP, from SIGMOD 2024, converts decimal-origin doubles to integers and packs them in vectors for fast, SIMD-friendly decoding, and is DuckDB’s default floating-point compression. Gorilla suits row-at-a-time ingestion; ALP suits sealed columnar segments.

Which compression do Prometheus, TimescaleDB and InfluxDB use?

Prometheus chunks use variable-width delta-of-delta timestamps and XOR values in the Gorilla style. TimescaleDB applies delta-of-delta with Simple-8b and run-length encoding to integers and timestamps, a Gorilla-style XOR scheme to floats, dictionary compression to low-cardinality columns, and LZ-based compression to other types. InfluxDB TSM uses delta encoding with Simple-8b or run-length encoding for timestamps, Gorilla’s float encoding for floats, bit-packing for booleans and Snappy for strings.

Comments

Leave a Reply Cancel reply

Tag Cloud

Categories