CBOR vs Protobuf vs MessagePack vs FlatBuffers: Encoding IoT Payloads on Constrained Devices (nanopb, CDDL, COSE) (2026)

A battery sensor that reports eight fields once a minute does not look like a serialization problem. Then you multiply it by fifty thousand devices on a metered NB-IoT or LTE-M plan, add a firmware that must fit in a few hundred kilobytes of flash, and discover that the format you picked on day one decides your radio airtime, your data bill, your signing story and how painfully you can evolve the schema in year three. The cbor vs protobuf question is really a question about where you want the cost to sit: on the wire, in the firmware, or in your governance process.

This post settles it with numbers you can reproduce. We encode one identical telemetry message in JSON, MessagePack, CBOR, Protobuf and FlatBuffers, measure the real byte counts with code, and then walk through the machinery that surrounds the encoding: schema evolution, CDDL, COSE signatures, deterministic CBOR, nanopb footprint, FlatBuffers zero-copy reads and zstd dictionaries.

You will leave with a decision procedure, a runnable Python script, and a clear sense of which format fits CoAP, MQTT and LwM2M.

What this covers: the format landscape, a byte-level worked example, schema evolution rules, CDDL and COSE, RFC 8949 determinism, nanopb and FlatBuffers resource costs, compression on top, transport fit, failure modes and a recommendation checklist.

Context and Background

Constrained devices rarely speak JSON for long. JSON is verbose, it needs a float parser, and it carries every key name in every message. The IETF’s answer for constrained environments is CBOR, the Concise Binary Object Representation, standardized in RFC 8949 (December 2020, which obsoleted RFC 7049). CBOR was designed explicitly for small code size, small message size and extensibility without a version negotiation. It shares JSON’s data model, so a map of text keys to numbers round-trips between the two, and it adds byte strings, tags and a defined set of floating point widths.

Protocol Buffers (Protobuf) come from a different tradition. Google designed it for internal RPC where both ends compile the same .proto schema. The wire format is not self-describing: you see field numbers and wire types, never field names. That is why Protobuf messages are tiny, and why a decoder without the schema can only guess at meaning. On microcontrollers the dominant implementation is nanopb, a plain-C, small-footprint implementation of the Protobuf wire format that supports static allocation.

MessagePack sits between the two: it is schema-less like CBOR, close to JSON in model, and widely implemented, but it is a community specification rather than an IETF standard. FlatBuffers, also from Google, takes a fourth path: the serialized buffer is laid out so that fields can be read in place with offset arithmetic, without any parse step.

These four are not interchangeable, and the IoT ecosystem has already made some choices for you. Lightweight M2M (LwM2M) 1.1 supports SenML CBOR and a CBOR content format alongside its own TLV; see our LwM2M device management architecture guide for how the object model maps onto payloads. The Constrained Application Protocol (CoAP) registers content formats for CBOR and SenML CBOR. If you are still deciding the transport itself, start with choosing your IoT protocol. The wider IETF constrained-node vocabulary (RFC 7228) is the useful frame for “constrained”: think tens of kilobytes of RAM and a hundred or so kilobytes of flash for the small classes.

One more piece of context matters for 2026. Payloads are no longer just telemetry; they are evidence. A reading that feeds a digital twin, a billing record or a safety interlock is increasingly expected to carry integrity and origin protection. That pushes encoding choices toward formats with a standard signing envelope, which is where COSE (CBOR Object Signing and Encryption, RFC 9052 and RFC 9053) enters the argument.

Four Formats, Four Philosophies

The quickest way to choose is to see each format as an answer to a different question. CBOR asks “how do we make JSON’s data model compact and standard for constrained nodes?” Protobuf asks “how do we make the smallest possible message when both sides share a compiled schema?” MessagePack asks “how do we make a faster, smaller JSON with minimal ceremony?” FlatBuffers asks “how do we avoid decoding at all?”

Figure 1: Decision flow for choosing an IoT payload encoding. The first fork is whether you need self-describing data; the second is whether signatures and IETF interoperability matter more than raw wire size.

The diagram compresses a longer argument. Self-description matters when the consumer of the data is not the producer: an operations engineer with a packet capture, a third-party integrator, a data lake that outlives the firmware that wrote into it. If any of those apply, a schema-less format (CBOR, MessagePack) saves you from a registry lookup just to read a byte stream. If both ends are under one team’s control and bandwidth is the scarcest resource, Protobuf’s numbered fields win.

CBOR in a paragraph

A CBOR data item starts with an initial byte whose top three bits are the major type and whose low five bits carry either the value itself (0 to 23) or a marker that 1, 2, 4 or 8 following bytes hold it. Major types cover unsigned integers, negative integers, byte strings, text strings, arrays, maps, tags and simple values or floats. Because small integers and short strings cost one header byte, a map of small integer keys to small values is very compact. Tags (major type 6) attach semantics, for example tag 1 for epoch time or tag 18 for a COSE_Sign1 structure. The format is extensible by design: a decoder that does not know a tag can still skip the item.

Protobuf in a paragraph

A Protobuf message is a sequence of key-value pairs. The key is a varint holding (field_number << 3) | wire_type, so fields numbered 1 to 15 cost a single tag byte. Wire type 0 is a varint, 1 is 64-bit fixed, 2 is length-delimited (strings, bytes, nested messages), and 5 is 32-bit fixed. Signed integers that are often negative should use sint32 or sint64, which apply ZigZag encoding so that -1 costs one byte rather than ten. In proto3, fields holding their default value are simply not emitted, which is a significant size win for sparse telemetry and a semantic trap we return to later.

MessagePack in a paragraph

MessagePack uses a first-byte type scheme similar in spirit to CBOR: “fixint” values for small integers, “fixstr” and “fixmap” for short strings and maps, and explicit width markers for larger values. It has an extension mechanism (ext types) and a standard timestamp extension. It has no registered tag semantics comparable to CBOR’s IANA registry, no half-precision float, and no defined deterministic encoding, which matters for signatures.

FlatBuffers in a paragraph

A FlatBuffer is a binary blob where a root offset points to a table; the table starts with a signed offset to its vtable, which lists the position of each field relative to the table start. A field absent from the vtable takes its schema default and costs nothing in the table body. Strings and vectors are referenced by offset. Because everything is addressed by offsets and naturally aligned, a reader can fetch one field from a mapped buffer in a handful of instructions without allocating memory. The price is padding, vtables and builder complexity on the write side.

Worked Example: One Message, Eight Encodings

Talk is cheap, so here is the message. It is the sort of record a battery-powered environmental sensor sends: a device identifier, an epoch timestamp, temperature, relative humidity, battery voltage in millivolts, signal strength, an alarm flag and a firmware string.

Field	Value	Notes
device_id	48213	unsigned
ts	1762300000	epoch seconds
temp_c	21.5	float
humidity	48.25	float
battery_mv	3012	unsigned
rssi	-87	signed
alarm	false	boolean
fw	“1.4.2”	text

The script below generates every number in this section. I ran it with Python 3.10, cbor2 6.2, msgpack 1.2.4, protobuf 7.x (for the generated class) and flatbuffers; the Protobuf schema is shown first. The sizes quoted here are the output of that run, but absolute sizes depend on library defaults, so treat the ordering and the mechanisms as the durable lesson rather than any single digit.

syntax = "proto3";
message Telemetry {
  uint32 device_id  = 1;
  uint32 ts         = 2;
  float  temp_c     = 3;
  float  humidity   = 4;
  uint32 battery_mv = 5;
  sint32 rssi       = 6;
  bool   alarm      = 7;
  string fw         = 8;
}

# measure_sizes.py  -- run after: protoc --python_out=. t.proto
# pip install cbor2 msgpack protobuf flatbuffers zstandard
import json, struct, cbor2, msgpack, flatbuffers
import t_pb2

msg = dict(device_id=48213, ts=1762300000, temp_c=21.5, humidity=48.25,
           battery_mv=3012, rssi=-87, alarm=False, fw="1.4.2")
keys = {k: i + 1 for i, k in enumerate(msg)}          # integer key map
intmap = {keys[k]: v for k, v in msg.items()}

def cbor_f32(m):                                       # force float32 (0xFA)
    out = b"\xa8"
    for k, v in m.items():
        out += cbor2.dumps(keys[k])
        out += b"\xfa" + struct.pack(">f", v) if isinstance(v, float) else cbor2.dumps(v)
    return out

pb = t_pb2.Telemetry(**msg).SerializeToString()

bld = flatbuffers.Builder(0)
fw = bld.CreateString("1.4.2")
bld.StartObject(8)
bld.PrependUint32Slot(0, 48213, 0); bld.PrependUint32Slot(1, 1762300000, 0)
bld.PrependFloat32Slot(2, 21.5, 0); bld.PrependFloat32Slot(3, 48.25, 0)
bld.PrependUint32Slot(4, 3012, 0);  bld.PrependInt32Slot(5, -87, 0)
bld.PrependBoolSlot(6, False, False)
bld.PrependUOffsetTRelativeSlot(7, fw, 0)
bld.Finish(bld.EndObject())
fb = bytes(bld.Output())

sizes = {
  "json":                    len(json.dumps(msg, separators=(",", ":")).encode()),
  "msgpack string keys":     len(msgpack.packb(msg)),
  "cbor string keys":        len(cbor2.dumps(msg, canonical=True)),
  "flatbuffers":             len(fb),
  "msgpack int keys":        len(msgpack.packb(intmap)),
  "cbor int keys":           len(cbor2.dumps(intmap, canonical=True)),
  "cbor int keys float32":   len(cbor_f32(msg)),
  "protobuf":                len(pb),
}
for name, n in sizes.items():
    print(f"{name:26s}{n:5d} bytes")

The run produced the following sizes. This is measured output from my run, not a vendor benchmark.

Encoding	Bytes	Why
JSON, compact	122	Key names and decimal digits in every message
MessagePack, string keys	93	Floats written as 8-byte doubles by default
CBOR, string keys	81	Preferred serialization shrinks floats to 2 bytes where exact
FlatBuffers	68	vtable, alignment padding, offsets
MessagePack, integer keys	47	Short keys, but doubles still cost 9 bytes each
CBOR, integer keys	35	Small keys plus half-precision floats
CBOR, integer keys, float32	39	Floats forced to 4 bytes plus a marker
Protobuf	33	Tags of one byte, no key names, default omitted
Fixed C struct (reference)	25	No framing at all, not self-describing

Figure 2: Measured size of the identical eight-field telemetry message in each encoding. The integer-key CBOR and Protobuf results are within two bytes of each other; the real separation is between keyed-by-name and keyed-by-number encodings.

The headline is that the gap between “CBOR vs Protobuf” mostly disappears once you stop using string keys. CBOR with small integer keys came in at 35 bytes against Protobuf’s 33. The ranking inside that pair flips with the data: Protobuf wins when many fields are zero, false or empty because proto3 omits them (our alarm=false cost nothing), while CBOR wins when floats happen to be exactly representable in half precision, as 21.5 and 48.25 are.

Reading the bytes

Hex dumps are where the mechanisms become visible. The CBOR integer-key encoding was a8 01 19 bc55 02 1a 690a9060 03 f9 4d60 04 f9 5208 05 19 0bc4 06 38 56 07 f4 08 65 312e342e32, 35 bytes. Walk it left to right. a8 is major type 5 (map) with eight pairs. 01 is the key 1. 19 bc55 is major type 0 with a two-byte argument: 48213. 1a 690a9060 is a four-byte unsigned integer, the timestamp. f9 4d60 is a half-precision float, which decodes to 21.5, and f9 5208 is 48.25. 38 56 is major type 1 (negative integer) with a one-byte argument, meaning -1 minus 86, which is -87. f4 is the simple value false. 65 is a five-byte text string, followed by its ASCII bytes.

The Protobuf encoding was 08 d5f802 10 e0a0aac806 1d 0000ac41 25 00004142 28 c417 30 ad01 42 05 312e342e32, 33 bytes. 08 is field 1 with wire type 0, then a three-byte varint for 48213. 10 is field 2, a five-byte varint. 1d is field 3 with wire type 5 (32-bit fixed), followed by little-endian 21.5. 30 ad01 is field 6, where ZigZag turned -87 into 173, which is a two-byte varint. Field 7 is absent because false is the default. 42 05 is field 8, length-delimited, five bytes.

Two lessons fall out. First, Protobuf pays for floats with four bytes always, while CBOR’s preferred serialization (RFC 8949 section 4.1) picks the shortest of half, single or double that preserves the value exactly. For sensor values that are quantized in hardware (a 0.25 degree step, say), half precision is exact surprisingly often; for a value such as 21.7 it is not, and the encoder falls back to four or eight bytes. Second, the varint and CBOR integer headers are both variable length, so large timestamps cost four to five bytes in either format. If you control both ends, sending the timestamp as an offset from a base time carried once per batch is a bigger saving than switching formats.

The MessagePack surprise

MessagePack with integer keys came in at 47 bytes, a dozen larger than CBOR, and the cause is instructive. The Python library packed the floats as 64-bit doubles (cb marker, 9 bytes each). MessagePack does define a 32-bit float (ca), and the library can emit it (use_single_float=True), which would cut four bytes per float, but the specification has no half-precision type, so it cannot match CBOR’s two-byte floats. On the messagepack vs protobuf axis the same lesson holds: with defaults, MessagePack is a good JSON replacement and a mediocre Protobuf replacement for float-heavy telemetry.

FlatBuffers: the bytes you pay for access speed

The 68-byte FlatBuffer holds the same data as the 33-byte Protobuf. The extra 35 bytes are the root offset, the vtable (a header plus one two-byte slot per field), the signed offset from table to vtable, alignment padding so that the 4-byte floats and integers sit on 4-byte boundaries, and a four-byte string length plus a null terminator. Notice that alarm=false and a default-valued field are still cheap: a field equal to its schema default can be omitted, and the vtable slot for it is zero. For one tiny message the overhead is brutal; for a large message with deep nesting it amortizes. FlatBuffers is a poor fit for a 30-byte uplink on a 51-byte LoRaWAN payload limit, and a good one for a gateway handing a 10 KB configuration blob to a task that reads three fields from it.

Schema Evolution: Where the Real Cost Lives

Wire size is a one-time engineering decision. Schema evolution is a decade-long operational one. Devices in the field run firmware from three generations ago, and your cloud must read them all without a flag day.

Protobuf rules

Protobuf’s evolution model is the most prescriptive. Field numbers are the identity of a field, so you never change or reuse a number; you reserve the number and name of a removed field. Adding a field is safe: old decoders skip unknown tags (and since Protobuf 3.5 unknown fields are retained and re-serialized rather than discarded). Changing a type is safe only between compatible wire representations, for example int32, uint32, int64 and bool share the varint wire type with caveats on truncation. The trap is the proto3 default-value rule: because 0, false and empty are not emitted, a receiver cannot distinguish “the sensor reported 0” from “the sensor did not report”. The remedy is the optional keyword (reinstated for explicit presence in proto3) or a wrapper message. On a nanopb target this costs a has_ boolean per field in the generated struct.

CBOR and MessagePack: evolution by convention

A schema-less format gives you evolution without ceremony and without safety. Adding a map key is trivially safe if readers ignore unknown keys. Removing or repurposing a key is dangerous precisely because nothing stops you. The discipline Protobuf enforces in tooling has to be recreated socially, which is the job of a schema language: CDDL for CBOR, covered next. A practical rule set: integer keys are never reused; negative integers are reserved for private or experimental keys (the COSE registries use that convention for labels); unknown keys are ignored, not rejected; and a payload carries its schema version either as a key or via the CoAP/MQTT content type.

FlatBuffers rules

FlatBuffers evolution is table-based and strict about ordering: new fields can only be added at the end of a table, and fields are deprecated, never removed, so that existing vtable slots keep their meaning. Reordering fields or inserting in the middle breaks old readers silently. Struct types, which are fixed layout, cannot evolve at all. The flatc compiler with a schema-compatibility checker in CI is the right safeguard.

CDDL: A Schema Language for Schema-less Formats

If CBOR has no built-in schema, how do you stop three teams from drifting? The Concise Data Definition Language, CDDL, is specified in RFC 8610. It describes the structure of CBOR (and JSON) data: maps, arrays, choices, ranges, sizes, and application-specific constraints. Crucially, a CDDL file is machine-readable, so you can generate validators and, with tools such as Nordic’s zcbor, generate C encoders and decoders for firmware directly from it. Treat the CDDL as the contract that plays the role of the .proto file.

Here is a CDDL definition of our telemetry message with integer keys. I have written it by hand and not run it through a validator, so treat it as untested.

; untested illustrative CDDL for the telemetry map
telemetry = {
  1 => uint .size 4,        ; device_id
  2 => uint .size 4,        ; ts, epoch seconds
  3 => float,               ; temp_c
  4 => float,               ; humidity
  5 => uint .le 6000,       ; battery_mv
  6 => int .ge -140 .le 0,  ; rssi dBm
  ? 7 => bool,              ; alarm, optional
  ? 8 => tstr .size (1..16),; fw version
  * int => any              ; unknown keys are tolerated
}

Three CDDL features deserve attention. The ? occurrence marks optional entries, giving you explicit presence, which Protobuf proto3 lacks by default. Control operators such as .size, .le and .ge encode range constraints that Protobuf cannot express at all, so a validator rejects an absurd battery voltage rather than passing it downstream. And the trailing * int => any wildcard encodes the “ignore unknown keys” evolution policy in the schema itself. The compatibility test you add to CI is mechanical: every sample from the previous release must still validate against the new CDDL.

The limitation is that CDDL is a validation language, not a code-generation contract with the guarantees of a compiled .proto. Generators exist, but their coverage of the full language varies, and you should pin the generator version and test round-trips on the exact subset you use.

CBOR Determinism: RFC 8949 and Why Signatures Need It

Two encoders can produce different bytes for the same logical value. An integer can be written in the shortest form or padded to eight bytes; a map’s pairs can appear in any order; a string can be definite or indefinite length; a float can be half, single or double. All are valid CBOR and all decode to equal data. If you compute a signature or a hash over the bytes, any variation breaks verification.

RFC 8949 section 4.2 defines deterministically encoded CBOR to remove that freedom. The Core Deterministic Encoding Requirements (section 4.2.1) are, in summary: use preferred serialization, meaning the shortest integer argument and the shortest float that preserves the value; use only definite-length encoding for strings, arrays and maps (no indefinite-length items); and sort map keys in bytewise lexicographic order of their deterministic encodings. Section 4.2.2 adds considerations for protocols that want more, such as how to treat floating point NaN and integers that could be floats, and section 4.2.3 describes the older length-first ordering inherited from RFC 7049, which some protocols (CTAP2 among them) still use. The point is that “canonical” is a protocol decision: pick one profile, write it down, and make every producer and verifier agree.

In our worked example the library’s canonical=True mode produced a map whose keys 1 to 8 are already in ascending order, which is also bytewise order for single-byte keys. Once keys cross 23 into two-byte encodings, bytewise ordering and numeric ordering still agree for non-negative integers, but negative keys (encoded under major type 1) sort after all non-negative keys in bytewise order. Many implementors learn this the hard way.

There is a practical escape hatch. COSE signs the exact bytes of the payload as transmitted. The verifier does not need to re-encode, so determinism is only required if two parties independently compute the same bytes, for example when you hash a logical record twice or sign a structure that the receiver re-serializes. Signing “the bytes you received” sidesteps the issue and is the recommended approach for telemetry. Keep this distinction clear in your design, because teams routinely pay for determinism they do not need and miss it where they do.

COSE: Signing and Authenticating CBOR Payloads

COSE, defined in RFC 9052 (structures and process) and RFC 9053 (initial algorithms), which together obsoleted RFC 8152, is to CBOR what JOSE is to JSON, but built with constrained nodes in mind. The single-signer message, COSE_Sign1, is a CBOR array of four elements wrapped in tag 18: a protected header (a byte string containing a CBOR map that is covered by the signature), an unprotected header map, the payload (embedded, or nil if detached) and the signature. Authenticated-only messages use COSE_Mac0 (tag 17) and authenticated encryption uses COSE_Encrypt0 (tag 16). Header labels are small integers: 1 is the algorithm (alg), 3 is content type and 4 is the key identifier (kid).

What actually gets signed is not the message but a canonical structure, the Sig_structure: the array ["Signature1", protected, external_aad, payload], serialized as CBOR. The context string prevents a signature made for one purpose being replayed in another; external_aad lets you bind application context (a device ID, a topic) that is not in the message itself, which is a cheap way to stop a valid message being replayed on the wrong channel.

Figure 3: COSE_Sign1 flow. The sensor signs the Sig_structure; the cloud rebuilds it from the received bytes, looks up the key by kid and validates the decoded payload against the CDDL contract.

The pairing is natural: CDDL and COSE are the two IETF pieces that turn “CBOR bytes” into “a governed, authenticated message”. The CDDL says what a valid payload looks like; COSE says who produced it and that it was not altered. The order of operations matters in the verifier: check the signature first, then decode and validate the payload. Parsing attacker-supplied structure before authentication is how decoders get exploited.

What signing costs on the wire

I built dummy COSE messages (zero-filled signatures, so the size is real but the message is not verifiable) around the 47-byte integer-key payload that the default cbor2 encoder produced in the batch experiment below. A COSE_Sign1 with a 64-byte signature (the size of Ed25519 or raw ES256 output) and a four-byte kid came to 128 bytes: 81 bytes of overhead, of which 64 is the signature itself. A COSE_Mac0 with an 8-byte truncated HMAC tag came to 71 bytes, an overhead of 24. The algorithm identifiers for these are registered in RFC 9053 (EdDSA is -8, ES256 is -7, and the truncated HMAC variants are in the HMAC family); verify the exact registry values against the IANA COSE registry before coding.

That asymmetry drives a real design choice. Public-key signatures give non-repudiation and let the cloud verify with a public key it cannot misuse, at 64 extra bytes per message. A MAC is far cheaper in bytes and CPU, but the key is shared, so any party that can verify can also forge. Many fleets sign a batch or a periodic digest with a public-key algorithm and MAC each individual reading. Another pattern is to rely on the transport: if DTLS or TLS already authenticates each device, per-message COSE is redundant for authentication and only earns its keep when messages traverse untrusted intermediaries such as a store-and-forward gateway or a broker you do not control. That end-to-end versus hop-by-hop distinction is the question to ask first.

nanopb: Protobuf on a Microcontroller

nanopb is a small C implementation of the Protobuf wire format, designed for 32-bit microcontrollers. The workflow is: a Python generator turns a .proto plus an optional .options file into a C header and source; the runtime is a few files (pb_encode.c, pb_decode.c, pb_common.c) with no dependency on malloc by default. The library’s documentation describes typical code size as a few kilobytes and RAM use as small and dominated by your message structs; I recall its documentation citing a range of roughly 2 to 10 kilobytes of code, but I could not re-verify the exact figure this run, so check the current docs for your build options before sizing flash.

Static allocation and the options file

The feature that makes nanopb viable on MCUs is that every variable-size field can be given a maximum in the .options file, which turns it into a fixed-size member of a plain C struct. For our message:

# telemetry.options  (nanopb generator options; untested)
Telemetry.fw  max_size:16

/* untested illustrative nanopb usage */
#include <pb_encode.h>
#include "telemetry.pb.h"

size_t encode_telemetry(uint8_t *buf, size_t cap, const SensorReading *r) {
    Telemetry m = Telemetry_init_zero;
    m.device_id  = r->id;
    m.ts         = r->epoch;
    m.temp_c     = r->temp;
    m.humidity   = r->rh;
    m.battery_mv = r->mv;
    m.rssi       = r->rssi;
    m.alarm      = r->alarm;
    strncpy(m.fw, FW_VERSION, sizeof(m.fw) - 1);

    pb_ostream_t s = pb_ostream_from_buffer(buf, cap);
    if (!pb_encode(&s, Telemetry_fields, &m)) return 0;   /* check PB_GET_ERROR(&s) */
    return s.bytes_written;
}

With max_size set, the struct has a fixed char fw[16], so the whole message lives on the stack or in a static buffer, and worst-case RAM is known at compile time. Without it, nanopb falls back to callbacks (you supply functions that stream the field) or to dynamic allocation if you enable it; callbacks are the right tool for large repeated fields such as a batch of readings, because you can emit them straight from a ring buffer without materializing the array.

What nanopb does not give you

nanopb implements a subset of Protobuf behavior by design, and the generated structs include presence booleans and counts for optional and repeated fields, which inflate RAM. It validates wire structure, not business ranges: a battery value of 60000 decodes happily. And the generated code is tied to the generator version; pin the generator in your build and treat .proto changes as reviewed API changes. Compare that with CBOR libraries such as QCBOR, TinyCBOR and zcbor, which expose a streaming encoder and decoder you drive by hand (or generate from CDDL in zcbor’s case). Hand-driven encoding gives you control over exactly which floats use which width, at the price of more application code and a greater chance of key typos.

FlatBuffers on Devices: Zero-Copy Reads and Their Limits

FlatBuffers’ selling point is that reading costs almost nothing. With generated accessors, reading->temp_c() is a bounds-assumed pointer add and load. There is no decode step, no heap, and no copy, so a gateway can memory-map a large buffer from flash or a file and touch only the fields it needs. That is a real advantage for configuration images, maps, firmware manifests and model metadata that are written rarely and read often.

For telemetry uplinks, the economics invert. Building a FlatBuffer requires the builder, which writes back-to-front and needs a growable or pre-sized buffer, and our measured 68 bytes against 33 for Protobuf is the cost of alignment and vtables. Zero-copy also does not mean zero-trust: a buffer from the network must be run through the generated verifier before any accessor is called, because the offsets inside it are attacker-controlled. Skipping verification on untrusted data is a classic memory-safety bug source, and verification itself walks the structure, which erodes the “no parse” claim for network inputs.

My assessment, and it is opinion, is that “flatbuffers iot” is a good choice on the gateway-and-edge-compute tier (Linux-class devices, shared memory between processes, large local datasets) and a rare choice on the sensor tier. If your question is “what do my battery nodes send”, FlatBuffers is seldom the answer; if it is “how do my edge services pass a large read-mostly model between processes”, it often is.

Compression on Top: zstd and Dictionaries

Compact encodings leave redundancy that general compression can remove, but small messages defeat general compressors. I measured this with the script below on synthetic data: 100 consecutive readings, 60 seconds apart, with a jittering temperature, a slowly draining battery and an RSSI that wobbles by one. The data is invented for the experiment, so treat the results as an illustration of mechanism, not a prediction for your fleet.

# compress_test.py -- pip install cbor2 zstandard
import cbor2, zstandard, random
keys = {'device_id':1,'ts':2,'temp_c':3,'humidity':4,'battery_mv':5,'rssi':6,'alarm':7,'fw':8}
random.seed(1)
def mk(i):
    return dict(device_id=48213, ts=1762300000+60*i,
                temp_c=round(21.5+random.uniform(-.3,.3),2), humidity=48.25,
                battery_mv=3012-i//10, rssi=-87+random.choice([-1,0,1]),
                alarm=False, fw="1.4.2")
enc = lambda m: cbor2.dumps({keys[k]: v for k, v in m.items()})
batch = [enc(mk(i)) for i in range(100)]
raw, one = b''.join(batch), batch[0]
c = zstandard.ZstdCompressor(level=3)
d = zstandard.train_dictionary(256, [enc(mk(i)) for i in range(200, 1200)])
zc = zstandard.ZstdCompressor(level=3, dict_data=d)
print(len(one), len(raw), len(c.compress(raw)), len(c.compress(one)),
      len(zc.compress(one)), len(zc.compress(raw)))

The printed results were: a single message of 47 bytes (the default encoder wrote the jittered floats as 8-byte doubles, so it is larger than the 35-byte canonical example); 100 messages concatenated at 4,700 bytes; that batch compressed with zstd level 3 to 810 bytes, about 8 bytes per reading; the single message compressed alone to 56 bytes, larger than the input because the frame header and entropy tables cost more than they save; and the single message compressed with a 256-byte trained dictionary to 37 bytes.

Three conclusions follow. Batching is the biggest lever: 100 readings compressed to roughly a sixth of their raw size, and an unusually regular stream does better still. Compressing one small message without a dictionary is a net loss. And a dictionary rescues small messages, because it supplies the shared structure (map header, keys, the firmware string) out of band. Zstandard, specified in RFC 8878, supports dictionaries natively; the dictionary ID is carried in the frame header.

The dictionary also had a limit that the numbers expose: applying the same dictionary to the 100-message batch gave 922 bytes, worse than the 810 bytes without it. A dictionary trained on single-message samples encodes assumptions that a long stream with its own internal repetition does not need. Train dictionaries for the exact framing you will use.

Operationally, a dictionary is a versioned schema artifact. Both ends must hold the same one, the dictionary ID must be checked, and a firmware update that changes the message shape needs a matching dictionary rollout. Also remember that compression and encryption interact: compress before encrypting, and be aware that compressing attacker-influenced data together with secrets enables length-oracle attacks such as CRIME and BREACH in some designs. For sensor telemetry that rarely mixes secrets with attacker input the risk is low, but it should be a conscious decision.

Transport Fit: CoAP, MQTT and LwM2M

Figure 4: End-to-end pipeline. A schema registry (CDDL files or .proto files) feeds both the device build and the ingest validator; zstd with a dictionary applies to batches before storage or backhaul.

Format and transport are separable, but each transport leans one way.

CoAP (RFC 7252) carries the payload as an opaque body and labels it with a Content-Format option. CBOR has a registered content format (60, as I recall; confirm in the IANA CoAP Content-Formats registry), and SenML in CBOR has its own (112). A COSE-wrapped payload uses a COSE content format. Protobuf has no IETF-registered content format, so you use application/octet-stream or a vendor-specific type, which weakens intermediaries’ ability to reason about the body. CoAP’s block-wise transfer (RFC 7959) handles larger payloads, and since CoAP messages commonly try to stay within a single datagram, every byte of overhead can push a message across a fragmentation boundary.

MQTT treats the payload as bytes. MQTT 5 adds a Content Type property and a Payload Format Indicator, but the indicator distinguishes only unspecified bytes from UTF-8 text. For binary formats, put the content type into the property or the topic convention, and do not assume subscribers can sniff the format. Because MQTT has no payload schema, the registry must be an out-of-band contract. For a broader MQTT comparison with other transports, see our IoT protocol selection guide.

LwM2M is the most opinionated. Its object model defines resources with fixed IDs and types, and version 1.1 added SenML CBOR and SenML JSON alongside the older TLV and a plain CBOR format for single resources. If you are on LwM2M, you do not choose Protobuf; the standard has already chosen the encoding and your freedom is in custom objects and their resource definitions. SenML adds its own per-record overhead (names, base time, units encoded with negative integer labels in the CBOR form), so a SenML CBOR record is bigger than our hand-tuned integer-key map. That is the price of an interoperable, self-describing format.

LoRaWAN and other tiny frames impose payload ceilings in the tens of bytes at low data rates. There, none of the self-describing formats is comfortable, and bit-packed custom frames or Protobuf with careful field ordering are the pragmatic choices, decoded in the network server through a codec. Because every byte there has airtime and duty-cycle consequences, run the arithmetic on your own regional limits rather than trusting a rule of thumb.

Trade-offs, Gotchas, and What Goes Wrong

Treating “fewer bytes” as the whole objective. The measured spread between integer-key CBOR (35) and Protobuf (33) is two bytes, well under the noise of your radio’s framing. Teams that migrate from CBOR to Protobuf for a 5% saving then lose self-description, standard signing and CoAP content-format support. Conversely, teams that keep string keys in CBOR pay more than double the integer-key size, which is the real waste.

Float width surprises. CBOR’s preferred serialization quietly changes a value’s wire width depending on its value. That is fine for decoding, but if your firmware parses a fixed offset, or your downstream system assumes a float32, you will see intermittent breakage when a reading crosses from half-precision-exact to not. Always decode through a library, never by offset. Likewise, half precision has about three decimal digits of precision; encoding a quantity through a lossy path is a data-quality decision, not just an encoding one. If exactness matters, force float32 or send scaled integers.

proto3 default ambiguity. The absent-zero problem produces real incidents: a sensor reporting 0 degrees looks identical to a missing reading. Use explicit presence for any field where zero is meaningful.

Reusing field numbers or keys. In Protobuf, reuse is silent data corruption: an old device’s field 4 decodes as a new field 4 of a different type. In CBOR integer-key maps it is the same hazard with no compiler to object. Keep a registry file with every number ever issued.

Non-deterministic encoding under signature. If anything between sign and verify re-encodes the payload (a gateway that decodes and re-emits, a JSON bridge), signatures break. Pass bytes through untouched, or define a deterministic profile per RFC 8949 section 4.2 and test it across implementations.

Decoder hardening. Self-describing formats let a malicious sender declare a four-gigabyte array or a deeply nested map in a few bytes. Set limits on nesting depth, item counts and total size in every decoder, authenticate before parsing where you can, and fuzz the decoder. FlatBuffers needs the verifier; nanopb needs max_size and max_count; CBOR libraries need nesting and length limits.

Clock and replay. A valid signed message can be replayed. Include a monotonic counter or timestamp inside the signed payload and track the last-seen value per device; COSE authenticates the message but does not by itself provide freshness.

Dictionary and schema skew. Fleet heterogeneity is the norm. Plan for a decoder that supports N previous schema versions and a registry that records which firmware emitted which version. Measure the share of the fleet on each version before deleting any decode path.

Practical Recommendations

For most constrained-device fleets in 2026, my recommendation is CBOR with integer map keys, a CDDL contract, and COSE where messages cross untrusted hops. It gets within a few bytes of Protobuf on typical telemetry, stays self-describing for operations and integrators, aligns with CoAP, SenML and LwM2M, and brings standard signing and encryption along. Use Protobuf with nanopb when you control both endpoints, bandwidth is dominant, you want compiler-enforced schema discipline, and you do not need standard signatures or content-type interoperability. Use MessagePack when you are replacing JSON in a software stack with good library support and no determinism or IETF alignment requirement. Use FlatBuffers for read-mostly data on Linux-class edge nodes, not for battery-sensor uplinks.

A short checklist to turn this into a decision:

Write the telemetry message once, with realistic values, and measure it in each candidate with a script like the one above before debating.
Decide self-description first; it removes half the candidates.
Fix the schema contract (CDDL or .proto) in a repository with CI checks for backward compatibility against stored sample messages from every shipped version.
Reserve keys and field numbers permanently; never reuse.
Choose float representation deliberately: half, single, scaled integer.
Decide whether signatures are end-to-end or hop-by-hop; use COSE_Mac0 or Sign1 accordingly, and sign received bytes rather than re-encoding.
Batch before you compress, and give small messages a versioned dictionary.
Set decoder limits and fuzz every parser on the cloud and device side.
Record the content type in the CoAP option, MQTT property or topic.

Frequently Asked Questions

Is CBOR smaller than Protobuf?

Usually not by much. In my measured example, integer-key CBOR was 35 bytes and Protobuf was 33. CBOR with string keys was 81 bytes. Protobuf tends to win when many fields are zero or empty, because proto3 omits defaults. CBOR can win when floats are exactly representable in half precision. The key choice, number or name, matters far more than the format.

Which is better for IoT, CBOR or Protobuf?

CBOR is the better default when you value self-describing data, IETF alignment with CoAP and LwM2M, and standard signing through COSE. Protobuf with nanopb is better when both ends share a compiled schema, strict evolution rules matter and you want the smallest practical messages. Neither is universally better; the question is where you want to pay for cost: bytes, tooling or governance.

What is nanopb and how big is it?

nanopb is a small plain-C implementation of the Protobuf wire format for microcontrollers. It supports static allocation through max_size and max_count options, so RAM use is known at compile time. Its documentation describes a footprint of a few kilobytes of code, but the exact figure depends on the options and compiler, so measure your own build rather than relying on a quoted number.

How do CDDL and COSE work together?

CDDL (RFC 8610) describes what a valid CBOR payload looks like, so a validator can reject malformed or out-of-range data. COSE (RFC 9052 and 9053) wraps the payload with a signature or MAC to prove origin and integrity. In practice you verify the COSE signature first, then validate the decoded payload against the CDDL, rejecting anything that fails either step.

Do I need deterministic CBOR to sign messages?

Often not. COSE signs the exact payload bytes you send, so a verifier checks those bytes without re-encoding. Deterministic encoding under RFC 8949 section 4.2 matters when two parties must independently produce identical bytes from the same logical data, such as hashing a record twice or re-serializing after a gateway. In that case fix one deterministic profile and test it across libraries.

Is FlatBuffers a good choice for IoT sensors?

Rarely for the sensor itself. Our 68-byte FlatBuffer compared with 33 bytes for Protobuf, and the builder adds complexity. It shines on gateways and edge Linux nodes that read large, mostly static data in place without parsing. Always run the verifier on untrusted buffers before using accessors, because offsets inside the buffer are attacker-controlled.