SAE J1939 Explained: PGN, SPN, Transport Protocol and a Telematics Pipeline

A heavy truck’s engine controller broadcasts its speed fifty times a second, and almost nobody outside the vehicle ever sees it. Plug a logger into the diagnostic connector and you get a firehose of 8-byte CAN frames with 29-bit identifiers that look like random hexadecimal. The vocabulary that turns that noise into “engine speed 2,167.75 rpm, coolant 87 degrees C, fault SPN 110 FMI 0” is SAE J1939, the higher-layer protocol family that sits on top of CAN for trucks, buses, tractors, construction machines, gensets and ships.

If you build heavy duty vehicle telematics, the cost of not understanding J1939 is concrete: mis-decoded fuel rates, silently dropped multi-frame fault messages, address collisions that take a retarder offline, and cloud bills inflated by streaming every frame at full rate. This post walks the protocol bottom-up, with runnable Python, and ends with a gateway design that gets data off the bus and into MQTT without losing the parts that matter.

What this covers: the 29-bit identifier, PDU1 versus PDU2, PGN and SPN decoding with worked scaling, address claiming and requests, the J1939 transport protocol (BAM and RTS/CTS), DM1 diagnostics, how J1939 relates to OBD-II and UDS, security, and a store-and-forward CAN-to-MQTT pipeline.

Context and Background

CAN, defined in ISO 11898, gives you a robust multi-master broadcast bus: arbitration by identifier, a 0 to 8 byte payload (classic CAN), CRC, and automatic retransmission. It deliberately says nothing about what the identifier means, who may send what, or how to move more than eight bytes. Every industry that adopted CAN had to invent that layer. In road vehicles, the Society of Automotive Engineers did it for heavy-duty equipment as the J1939 document set, which grew out of the earlier J1708/J1587 era and was adopted widely because diesel engine, transmission, brake and aftertreatment suppliers needed to interoperate.

J1939 is not one document. It is a family: J1939-21 defines the data link layer (identifier layout, transport protocol), J1939-71 the application layer (the actual parameter definitions), J1939-73 diagnostics, J1939-81 network management including address claiming, and J1939-13 the off-board diagnostic connector. The physical layer variants are J1939-11 and -15 (250 kbit/s) and J1939-14, which defines a 500 kbit/s layer; classic J1939 networks run at 250 kbit/s, and newer vehicles increasingly use 500 kbit/s. The authoritative catalogue of every PGN, SPN, scaling and enumeration is the J1939 Digital Annex (J1939DA), a paid, spreadsheet-style publication that is updated frequently. Public mirrors such as isobus.net expose partial lookups, and the agricultural ISOBUS standard (ISO 11783) reuses the same foundations, which is why a combine harvester and a truck share much of their vocabulary.

Two other ecosystems get confused with J1939. Passenger cars use OBD-II (SAE J1979 / ISO 15031) over ISO 15765-4 CAN, with request-response diagnostics on 11-bit identifiers. Modern ECUs of every kind also speak UDS (ISO 14229) over ISO 15765-2 transport for flashing and deep diagnostics. J1939 is different in character: it is broadcast-first. Most data is published periodically without anybody asking, and diagnostics are published too (DM1 is simply a periodic broadcast). That property is exactly what makes it a good telematics source and also what makes a passive gateway sufficient for most fleet use cases.

Compared with the industrial fieldbus world, J1939 sits near CANopen in intent but differs in philosophy. We compare the CAN family in CAN FD vs CAN XL vs CANopen for industrial fieldbus and robotics; the short version is that CANopen is object-dictionary and configuration oriented, while J1939 is parameter-group and broadcast oriented. If your vehicle is electrified, much of the same bus carries battery and inverter parameters, a topic covered in IoT electric vehicle architecture and battery telematics.

A caution on sourcing before we go further. The full J1939 documents are copyrighted and paid. Everything below about field layouts reflects the widely documented structure of the standard; where I state specific scaling or PGN assignments I have either cross-checked a public source (noted) or flagged that you must confirm against J1939DA for your revision and your OEM’s proprietary extensions.

The 29-bit Identifier and the Reference Architecture of J1939 Addressing

J1939 repurposes the 29-bit extended CAN identifier as six fields: a 3-bit priority, a 1-bit Extended Data Page, a 1-bit Data Page, an 8-bit PDU Format, an 8-bit PDU Specific and an 8-bit Source Address. The middle four fields form the 18-bit Parameter Group Number (PGN) that identifies what the message carries.

Figure 1: How a J1939 29-bit CAN identifier decomposes, and why the PDU Format value decides whether the PDU Specific byte is a destination or part of the PGN.

Figure 1 shows the decision that trips up most first-time decoders. Reading the identifier from most significant bit to least: bits 28 to 26 are priority (0 highest, 7 lowest), bit 25 is the Extended Data Page (EDP), bit 24 the Data Page (DP), bits 23 to 16 the PDU Format (PF), bits 15 to 8 the PDU Specific (PS), and bits 7 to 0 the Source Address (SA). Because CAN arbitration is bitwise with dominant zeros, a lower numeric identifier wins the bus, so putting priority in the top bits gives you a coarse priority scheme for free. Control-loop traffic such as torque commands typically sits at priority 3, while informational traffic uses 6.

PDU1 versus PDU2: when is the PS byte part of the PGN?

The PF byte decides. If PF is below 240 (0xF0), the message is PDU1, a destination-specific format: PS is the destination address and is not part of the PGN. A PDU1 PGN therefore has its low byte zeroed when you compute it. If PF is 240 or above, the message is PDU2, a broadcast format: PS becomes a “group extension” and is part of the PGN. This is why PDU2 PGNs span a large range (0xF000 to 0xFFFF on data page 0, i.e. 61440 to 65535), while PDU1 PGNs come in steps of 256 (for example 59904 = 0xEA00, 60416 = 0xEC00).

Take the identifier 0x0CF00400, one of the most common frames on any diesel vehicle. Priority is 3, EDP and DP are 0, PF is 0xF0 = 240, so it is PDU2; PS is 0x04 and SA is 0x00 (the engine). The PGN is 0xF004 = 61444, “Electronic Engine Controller 1” (EEC1). The code later in this post decodes exactly this.

Why the design is clever, and what it costs

Folding the destination into the identifier of point-to-point messages means a receiving ECU’s hardware acceptance filter can match “messages for me” with a single mask on bits 15 to 8. Folding the group extension into the identifier of broadcast messages gives 4,096 possible PDU2 groups on page 0 alone without any extra bytes. The cost is that the PGN alone does not tell you who sent a frame, and for PDU1 it does not tell you whom the frame was for unless you keep PS separately. A correct decoder must therefore keep three things per frame: the PGN, the source address, and, for PDU1, the destination.

Source addresses are one byte, with 0 to 253 assignable, 254 meaning “null/cannot claim” and 255 meaning “global” as a destination. By long convention engine #1 uses SA 0, transmission 3, brake system controller 11, instrument cluster 23, and so on, but these are preferred addresses from J1939-81’s industry-group tables, not guarantees. Addresses can move, which is why a telematics system should key on the J1939 NAME (discussed below) and not on SA alone.

The three PGN families you actually meet

Most traffic falls into three groups. Proprietary-A (PGN 61184, 0xEF00) is PDU1 and carries manufacturer-defined content addressed to a node. Proprietary-B (PGNs 65280 to 65535, 0xFF00 to 0xFFFF) is PDU2 broadcast with manufacturer-defined content, and it is where OEMs hide their most interesting signals. Everything else is standardized in J1939-71 and listed in J1939DA. For telematics, the practical consequence is that a decoder built only from public sources will show you a lot of standard PGNs and a lot of undecodable Proprietary-B frames; reading those requires the OEM’s documentation or a licensed database.

SPN Decoding: From Bits to Engineering Values

A Suspect Parameter Number (SPN) identifies one parameter inside a parameter group: its start bit, its length, its resolution (scale), its offset, its unit and its valid range. The PGN says “this frame is EEC1”; the SPN says “bytes 4 and 5, little-endian, 0.125 rpm per bit, is engine speed.”

Figure 2: The decode path for any J1939 signal: extract bits, reject reserved values, apply resolution and offset, then tag with source address and timestamp.

Worked example: engine speed in PGN 61444

SPN 190, Engine Speed, lives in PGN 61444 (EEC1). Per a public explainer I cross-checked (Wells Tsai’s J1939 introduction), it occupies 16 bits starting at bit 24 (that is, bytes 4 and 5 counting from byte 1 at bit 0), with a resolution of 0.125 rpm per bit and an offset of 0. A raw value of 17,214 therefore decodes to 17,214 x 0.125 = 2,151.75 rpm, and the maximum valid raw value of 64,255 (0xFAFF) is 8,031.875 rpm. J1939 payload fields are little-endian, so the low byte comes first on the wire: bytes 0xBE 0x43 form the raw value 0x43BE = 17,342, giving 2,167.75 rpm.

The same PGN carries other signals I am stating from the widely published EEC1 layout, so confirm against J1939DA for your revision: byte 1 low nibble is engine torque mode (SPN 899), byte 2 is driver’s demand engine percent torque (SPN 512), byte 3 is actual engine percent torque (SPN 513), bytes 4 and 5 are engine speed (SPN 190), byte 6 is the source address of the controlling device (SPN 1483), and byte 8 is engine demand percent torque (SPN 2432). The torque percentages use a 1 percent per bit scale with a -125 offset, so raw 0x96 = 150 decodes to 25 percent, and raw 125 is zero torque. The offset is why negative (engine braking) torque is representable in a single byte at all.

Reserved values: the part that breaks dashboards

An 8-bit parameter does not use the full 0 to 255 range for data. J1939 reserves the top of the range: for an 8-bit signal, 0xFF means “not available”, 0xFE means “error”, and 0xFB to 0xFD are reserved for future use; for a 16-bit signal the valid range ends at 0xFAFF, 0xFB00 to 0xFDFF is reserved, 0xFE00 to 0xFEFF means error and 0xFF00 to 0xFFFF means not available. A decoder that blindly multiplies 0xFFFF by 0.125 reports an engine running at 8,191.875 rpm. In my experience the most common telematics data-quality defect in the field is exactly this: “not available” decoded as a real value during key-on before a sensor warms up, creating spurious spikes that wreck averages and trip threshold alerts. Decode to a null, never to a number.

Signals narrower than a byte (status flags, 2-bit enumerations, 4-bit modes) need bit masks, and discrete values often have their own enumerations, such as 00 = off, 01 = on, 10 = error, 11 = not available for two-bit states. Treat “not available” as a first-class state in your schema.

The runnable decoder

The following Python decodes a 29-bit identifier, an EEC1 payload and a BAM reassembly. It is self-contained and was run to produce the output shown in the comments.

from dataclasses import dataclass

@dataclass
class J1939Id:
    priority: int; edp: int; dp: int; pf: int; ps: int; sa: int
    @property
    def pgn(self):
        if self.pf < 240:               # PDU1: PS is destination, not part of PGN
            return (self.edp << 17) | (self.dp << 16) | (self.pf << 8)
        return (self.edp << 17) | (self.dp << 16) | (self.pf << 8) | self.ps
    @property
    def dest(self):
        return self.ps if self.pf < 240 else 0xFF

def decode_id(can_id: int) -> J1939Id:
    return J1939Id((can_id >> 26) & 7, (can_id >> 25) & 1, (can_id >> 24) & 1,
                   (can_id >> 16) & 0xFF, (can_id >> 8) & 0xFF, can_id & 0xFF)

def decode_eec1(data: bytes) -> dict:
    assert len(data) == 8
    def pct(b): return None if b >= 0xFE else b - 125.0
    raw = data[3] | (data[4] << 8)
    rpm = None if raw >= 0xFB00 else raw * 0.125
    return {"engine_speed_rpm": rpm, "actual_torque_pct": pct(data[2]),
            "driver_demand_pct": pct(data[1]), "controlling_sa": data[5]}

def reassemble_bam(frames):
    """frames: list of (pgn, sa, data8). Returns {(sa, inner_pgn): payload}."""
    sessions, done = {}, {}
    for pgn, sa, d in frames:
        if pgn == 0xEC00 and d[0] == 0x20:       # TP.CM BAM
            size = d[1] | (d[2] << 8)
            inner = d[5] | (d[6] << 8) | (d[7] << 16)
            sessions[sa] = dict(size=size, n=d[3], pgn=inner, buf={})
        elif pgn == 0xEB00 and sa in sessions:   # TP.DT
            s = sessions[sa]; s["buf"][d[0]] = d[1:8]
            if len(s["buf"]) == s["n"]:
                data = b"".join(s["buf"][i] for i in range(1, s["n"] + 1))[:s["size"]]
                done[(sa, s["pgn"])] = data; del sessions[sa]
    return done

i = decode_id(0x0CF00400)
print(i, hex(i.pgn), i.pgn)
# J1939Id(priority=3, edp=0, dp=0, pf=240, ps=4, sa=0) 0xf004 61444
print(decode_eec1(bytes([0xF0, 0x7D, 0x96, 0xBE, 0x43, 0x00, 0xF0, 0x7D])))
# {'engine_speed_rpm': 2167.75, 'actual_torque_pct': 25.0, 'driver_demand_pct': 0.0, 'controlling_sa': 0}
payload = bytes([0x00, 0xFF, 0x6E, 0x00, 0x1F, 0x03, 0x9C, 0x00, 0x03, 0x05, 0x1A, 0x00, 0x07, 0x81])
cm  = bytes([0x20, len(payload), 0, 2, 0xFF, 0xCA, 0xFE, 0x00])      # BAM: 14 bytes, 2 packets, PGN 0xFECA
dts = [(0xEB00, 0x00, bytes([k + 1]) + (payload + b"\xff" * 7)[7 * k:7 * k + 7]) for k in range(2)]
out = reassemble_bam([(0xEC00, 0x00, cm)] + dts)
print(out[(0, 0xFECA)] == payload)   # True

Two details in the decoder deserve emphasis. First, the BAM session is keyed by source address, because two ECUs can run BAM sessions concurrently; for RTS/CTS you must key by the (source, destination) pair. Second, the last TP.DT frame is padded with 0xFF, so you must truncate to the announced size, which the code does with the slice.

Network Management: Address Claiming, Requests and the NAME

Unlike a static-ID bus, J1939 lets ECUs negotiate their source address at power-up, which is what allows two identical trailers’ ABS modules, or a second engine on a genset, to coexist. The mechanism is in J1939-81.

Address claiming with PGN 60928

Each ECU has a 64-bit NAME: an arbitrary-address-capable bit, an industry group, vehicle system and instance, function and function instance, ECU instance, a manufacturer code, and a 21-bit identity number. At start-up the ECU broadcasts an Address Claimed message (PGN 60928, 0xEE00) whose payload is its NAME and whose source address is the address it wants. If another ECU is already using that address, the one with the numerically lower NAME wins; the loser either claims a different address (if its arbitrary-address-capable bit is set) or sends Cannot Claim from SA 254 and stays silent on the bus. After a claim, the ECU is expected to wait a short settling interval (the standard cites 250 ms, which I recommend confirming in J1939-81 for your revision) before assuming ownership and transmitting.

For telematics this has two practical consequences. First, never hard-code “SA 0 is the engine” for fleets with mixed vehicles, retrofitted retarders, or multi-engine equipment; build a table that maps SA to NAME using the claims you observe, and re-key your signal metadata when a claim changes. Second, a passive gateway that never transmits cannot be the cause of address conflicts, so a listen-only design avoids the whole class of bus-off-and-collision failures that plague telematics boxes that actively poll.

Requests with PGN 59904

Most data is periodic, but some is on-demand. The Request PGN (59904, 0xEA00, PDU1) carries a three-byte payload: the requested PGN, least significant byte first. A node asks a specific destination (or the global address 255) to send the PGN once. A request for Address Claimed (60928) sent globally is the standard way to discover who is on the bus and learn all NAMEs. Responses may be a normal PGN broadcast or, if the target cannot comply, an Acknowledgment (PGN 59392, 0xE800) with a negative or “access denied” code. Note that polling is the active mode and has a bus-load cost; do it sparingly, on a schedule, for slow-moving data such as vehicle identification (PGN 65260) and software identification, and only if your gateway is permitted to transmit.

Transmission rates and bus load

Periodic PGNs have defined repetition rates: EEC1 is nominally every 20 ms per the J1939-71 value cited in the explainer above (some secondary sources quote 50 ms; treat the standard as authoritative), while slow signals like ambient conditions run at 1 s and many are on-change with a minimum gap. For capacity planning, a classic CAN frame with 8 data bytes is about 128 bits on the wire with a 29-bit identifier (rough upper bound with worst-case stuffing around 150). At 250 kbit/s that is about 2,000 frames per second if the bus were 100 percent loaded; J1939 networks are designed to stay well below roughly 30 to 40 percent steady-state load, which is on the order of 600 to 800 frames per second. Your gateway must keep up with that rate for sustained periods, which is trivial on a Linux SoC and not trivial on a poorly written 8-bit microcontroller with a single receive buffer.

Multi-Packet Messages: The J1939 Transport Protocol

Eight bytes is not enough for a VIN, a component identification string, or a diagnostic message listing more than one fault. J1939-21 therefore defines a transport protocol (TP) that carries up to 1,785 data bytes, which is 255 packets of seven data bytes each. The protocol is implemented with two PGNs: TP.CM (Connection Management, PGN 60416, 0xEC00) for control and TP.DT (Data Transfer, PGN 60160, 0xEB00) for payload. A separate extended transport protocol (J1939-21 / ISO 11783-6 ETP, with its own PGNs) exists for payloads up to roughly 117 MB, and it is mostly seen in software download and ISOBUS applications rather than in telematics.

Figure 3: Destination-specific J1939 transport protocol exchange with RTS, CTS windowing, data packets and the end-of-message acknowledgment. BAM omits the handshake.

BAM: broadcast, no flow control

The Broadcast Announce Message is used when the data goes to everyone. The sender transmits one TP.CM frame to the global address with control byte 0x20, the total message size (bytes 2 and 3, little-endian), the total packet count (byte 4), a reserved 0xFF in byte 5, and the 24-bit PGN of the message being carried (bytes 6 to 8). It then sends the TP.DT frames back to back, each with a sequence number from 1 to N in byte 1 and seven data bytes in bytes 2 to 8. There is no flow control and no acknowledgment, so receivers must be fast enough; the standard specifies minimum and maximum spacing between BAM data packets (on the order of 50 to 200 ms, which I have not re-verified against the current J1939-21 text) and a receive timeout beyond which an incomplete session is discarded. A public example from CSS Electronics’ decoder documentation shows 200A0002FFAAF000: control 0x20, 10 bytes, 2 packets, PGN 0xF0AA.

RTS/CTS: destination-specific with flow control

For a conversation with one node, the sender transmits Request To Send (control byte 0x10, same size, packet count, maximum packets per CTS, and PGN). The receiver replies with Clear To Send (control byte 0x11) granting a window: how many packets it will accept and the next sequence number to expect. The sender transmits that window of TP.DT frames; the receiver then grants another window, or, once all bytes have arrived, sends End of Message Acknowledgment (control byte 0x13). Either side can abort by sending a connection-abort message (control byte 0xFF) with a reason code. The same CSS Electronics example shows CTS granting a single packet at a time, which is legal and common in constrained receivers.

Why the protocol matters to a telematics designer

The most common multi-packet traffic you will care about is DM1 with more than one active fault, VIN (PGN 65260, a BAM), component identification (PGN 65259), and software identification (PGN 65242). If your decoder drops TP sessions, you will appear to work in the lab with single-fault vehicles and silently miss fault lists in the field, the worst possible failure mode for a maintenance product. The three implementation rules are: key sessions by (source, destination, inner PGN), enforce timeouts so a lost TP.DT does not leak state forever, and handle a new BAM from the same source replacing an older unfinished one. Linux SocketCAN has a kernel J1939 stack (CAN_J1939) that implements address claiming and TP transparently, which is the lowest-risk way to get this right if your gateway runs Linux; when you instead sniff raw frames for a passive logger, you reimplement the receive side, which is what the code above does.

Diagnostics: DM1, DM2 and the SPN-FMI Pair

J1939-73 defines the diagnostic messages, prefixed DM. The key ones are DM1 (active diagnostic trouble codes, PGN 65226 / 0xFECA, broadcast every second while any fault is active and on state change), DM2 (previously active codes, PGN 65227, on request), DM3 (clear previously active, PGN 65228) and DM11 (clear active). DM13 coordinates stopping broadcasts during service.

A DM1 payload starts with two bytes of lamp status: malfunction indicator, red stop, amber warning and protect lamps, each with a flash state. After them come one or more four-byte diagnostic trouble codes (DTC). Each DTC encodes the SPN in 19 bits (low 8 bits in byte 1, middle 8 bits in byte 2, the top 3 bits in the high bits of byte 3), a 5-bit Failure Mode Identifier (FMI) in the low bits of byte 3, a conversion-method bit and a 7-bit occurrence count in byte 4. The FMI says how the SPN is suspect: for example FMI 0 means data valid but above normal operating range at its most severe level, FMI 3 means voltage above normal or shorted high, FMI 4 voltage below normal or shorted low, FMI 31 condition exists. So “SPN 110 FMI 0” is engine coolant temperature (SPN 110) critically high; the SPN names the thing, the FMI names the failure.

The DM1 in our code sample is two lamp bytes plus three DTCs (14 bytes), so it cannot fit in one frame and travels as a BAM, which is the precise scenario in which a naive decoder loses data. With exactly one active fault the message is six bytes, fits in a single frame, and is sent without TP. Pad unused bytes with 0xFF; when no faults are active, the standard prescribes a DM1 whose DTC field reads SPN 0, FMI 0, occurrence 0 (or similar all-zero convention) with lamps off, rather than silence, so a monitor can distinguish “no faults” from “ECU dead”.

For fleet operations, what you derive from DM1 is a state machine per (NAME, SPN, FMI): first-seen timestamp, last-seen timestamp, occurrence count, severity from lamp state, and cleared when absent from DM1 for a few multiples of the transmission period. Alert on transitions, not on every one-second repeat, or you will bury operators and your broker under duplicates.

How J1939 Relates to OBD-II, UDS and ISO 15765

It is worth drawing the boundary, because heavy-duty vehicles speak all three. OBD-II (SAE J1979, ISO 15031) is the light-duty emissions diagnostic interface: a tester asks a service/PID question on 11-bit identifiers and the ECU answers, using ISO 15765-4 as the CAN mapping. Heavy-duty OBD regulations point to J1939-73 or to ISO 27145 (WWH-OBD, which uses UDS services) depending on jurisdiction and engine class, so a modern heavy truck may expose both a J1939 view and a UDS view of the same faults.

UDS (ISO 14229-1) provides request/response services such as read DTC information (0x19), read data by identifier (0x22), routine control and flash download, and it uses ISO 15765-2 (ISO-TP) to segment payloads with first frame, consecutive frame and flow control frames. ISO-TP and J1939-TP solve the same problem differently: ISO-TP lets a single frame carry six or seven data bytes plus a PCI byte and negotiates block size and separation time through flow control; J1939 TP uses seven data bytes per packet with a sequence number, a 1,785-byte ceiling, and the BAM broadcast option, which ISO-TP lacks. For a telematics gateway the rule of thumb is: use J1939 for the continuous, broadcast signal stream; use UDS only for the occasional controlled query or for OTA reflash orchestrated through a gateway ECU, and keep the two in separate code paths so a UDS session never competes with the passive monitor.

The J1939 digital annex is also the bridge to modern tooling. DBC files with J1939 attributes, produced from J1939DA, are what Vector, CSS Electronics, and open-source cantools consume; cantools and the Linux can-utils handle 29-bit identifiers and J1939 PGN extraction, and Scapy ships a J1939 contrib layer. Prefer a database-driven decoder over hand-written offsets; hand-written decoders drift when the OEM’s engine is revised.

A Heavy-Duty Vehicle Telematics Data Pipeline: CAN to MQTT

The decoding layer is half the problem. The other half is moving decoded data off a vehicle that spends part of its life in tunnels, quarries and depots with no coverage, without drowning the uplink or losing the events that matter.

Figure 4: A gateway-side CAN bus data pipeline: decode, edge-filter, buffer to flash, publish over MQTT with QoS 1, and feed a stream processor and time-series store.

Stage 1: capture and decode on the gateway

Run the CAN interface in listen-only mode unless you need to transmit. A listen-only controller does not ACK frames or drive the bus, so a faulty or misconfigured gateway cannot damage the vehicle network. With SocketCAN on Linux, open the interface with a hardware filter for the PGNs you care about; the kernel then discards everything else before your process wakes. Decode with a database-driven decoder, keep per-frame timestamps from the hardware or kernel (not from when your Python loop got scheduled), and carry (NAME, SA, PGN, SPN, value, unit, timestamp) as the canonical record. Mark values “not available” or “error” explicitly rather than dropping them, because the transition from not-available to valid after a cold start is itself diagnostic.

Stage 2: edge decimation and deduplication

Raw volume is the enemy. EEC1 at 50 Hz is 3,000 frames per minute for one PGN. A typical heavy truck presents several hundred SPNs, and a naive pipeline publishing every frame as JSON would generate gigabytes per vehicle per month. The remedy is per-signal deadband plus heartbeat filtering: emit a sample when the value moves by more than a threshold, or when a maximum interval has passed so the cloud can tell “unchanged” from “offline.” The snippet below, run on a synthetic 60-second engine-speed trace (a step from roughly 1,500 to 1,900 rpm halfway through), reduces 3,000 frames to 6 emitted samples with a 50 rpm deadband and a 10-second heartbeat.

class Decimator:
    """Per-signal edge filter: emit on deadband breach or heartbeat."""
    def __init__(self, deadband, heartbeat_s):
        self.db, self.hb, self.last = deadband, heartbeat_s, {}
    def offer(self, key, value, now):
        if value is None:
            return False
        prev = self.last.get(key)
        if prev is None or abs(value - prev[0]) >= self.db.get(key, 0) or now - prev[1] >= self.hb:
            self.last[key] = (value, now); return True
        return False
# 3000 synthetic frames in, 6 samples out (deadband 50 rpm, heartbeat 10 s)

That 500x reduction is an artifact of a clean synthetic trace; real signals with noise and driving dynamics will compress less, and the right deadband depends on the use case. Fuel-efficiency analytics wants fine speed and load profiles around transients; predictive maintenance wants slow trends and exceedance counts. A well-known compromise is to publish two streams: a low-rate summary (for example 1 Hz aggregates with min, max, mean per window) for all signals, and a high-rate burst captured into a ring buffer around trigger events such as a new DM1 fault, a hard-braking threshold or a coolant over-temperature, uploaded once the event is confirmed. Event-triggered capture is what turns a telemetry stream into a diagnostic tool.

Faults take a different path: DM1 changes, address-claim changes and TP-delivered identification data are low-volume and high-value, and should bypass decimation and be published immediately with their own topic and higher delivery guarantees.

Stage 3: store-and-forward

Assume the link is down more often than you hope. Write outbound messages to an append-only queue on flash (SQLite in WAL mode, or a simple segmented log), acknowledge deletion only after the broker acknowledges receipt, and bound the queue by bytes with an explicit drop policy. A defensible policy is to drop oldest high-rate samples first, then oldest summaries, and never drop fault events or state changes until the disk is truly full. Wear matters: a fleet gateway writing continuously to eMMC can exhaust its endurance in a few years, so batch writes (write every few seconds, not every message), use flash rated for the workload, and monitor wear indicators.

Stage 4: transport with MQTT

MQTT over TLS is the usual choice for the uplink because of its small framing, session persistence and QoS levels. Use QoS 1 (at least once) for telemetry and events and make the consumer idempotent with a message identifier of (vehicle id, sequence number), since QoS 1 can redeliver. Use a persistent session with a clean-start policy suited to your broker, set a Last Will to mark a vehicle offline, and use topic structure such as fleet/{tenant}/{vin}/telemetry, .../events and .../diag. Batch many samples into one payload (for example a compressed JSON array or a binary format such as Protocol Buffers or CBOR) rather than one message per sample; the per-message overhead of MQTT plus TLS records dwarfs a four-byte value. On cellular links, also consider the radio’s tail energy and billing granularity, which reward fewer, larger uploads.

Key material for TLS client certificates on a vehicle deserves care: a stolen gateway should not become a credential for the whole fleet. Per-device certificates, rotation and hardware-backed key storage follow the same patterns we cover in envelope encryption, KMS and HSM key management.

Stage 5: cloud-side processing

In the cloud, a stream processor validates, enriches (joins NAME to vehicle configuration and the signal database version), normalizes units and writes to a time-series store, while a separate path feeds alerting. Two modelling points save pain later. Store the decoder version with each record so a corrected scaling can be backfilled from retained raw frames; and keep a short-retention raw-frame archive for a sample of vehicles, because bugs in decoding are discovered months later and are only fixable if you kept the bytes. Partition time-series by vehicle and time, and downsample on a schedule with rollups, preserving min and max alongside the mean since averages hide the exceedances that cause failures.

Worked sizing example

Take 1,000 vehicles, 300 decoded signals each, and the two-stream design above. Summary stream: 300 signals at one sample per 10 seconds on average after decimation is 30 samples per second per vehicle; at roughly 12 bytes per sample in a compact binary encoding that is about 360 B/s, or roughly 31 MB per day per vehicle, about 31 GB per day across the fleet before compression. Naive full-rate publishing of just the 8-byte payloads at a conservative 600 frames per second, with 25-byte JSON overhead per frame at minimum, would be at least 15 KB/s per vehicle, over 1.2 GB per day per vehicle. These are back-of-envelope calculations from stated assumptions, not measurements, but they show why decimation is the difference between a viable and a ruinous cellular bill. Run the arithmetic against your own signal list and real data plan.

Security of J1939 Networks

J1939 was designed in an era when the vehicle bus was assumed to be a closed, trusted network. There is no authentication of the source address, no integrity protection, and no encryption. Any node that can write to the bus can claim any address, impersonate the engine, send a torque request, or flood the bus. Telematics gateways sit on that bus and also on a cellular network, which is exactly the bridge attackers want, and published research has repeatedly demonstrated vehicle-bus compromise through exposed diagnostic dongles.

SAE has been working on this. J1939-91C is the SAE network-security document for J1939. Descriptions in public trade material say it adds ECU authentication via digital certificates and a public key infrastructure, cryptographic authentication of selected messages, replay protection through freshness, and optional encryption, while leaving PGNs and CAN framing unchanged (Embitel’s overview). SAE Mobilus also lists technical papers evaluating vehicle-system performance of a J1939-91C implementation and a cybersecurity rating framework applied to it. What I could not verify from primary text is the document’s exact publication status, cryptographic algorithm choices and deployment prevalence in 2026, so treat any claim that a given fleet “supports J1939-91C” as something to confirm with the ECU supplier.

Regardless of standard adoption, a defensive posture for a telematics integrator looks like this:

Listen-only by default. If the gateway never transmits, an attacker who owns it cannot write to the bus through it. If it must transmit (for example to request VIN), use a hardware-enforced allow-list of PGNs.
Segment the network. Put the gateway behind a vehicle gateway ECU or a CAN firewall that filters what crosses from the diagnostic connector to powertrain buses.
Authenticate the cloud link properly. Mutual TLS with per-device identity, short-lived credentials, and remote disable.
Harden and update. Signed firmware, secure boot, and a tested over-the-air path. Treat the gateway as a networked computer, because it is one.
Treat inbound cloud commands as hostile. Remote diagnostics or parameter changes need authorization, auditing and rate limits.

Trade-offs, Gotchas, and What Goes Wrong

Proprietary signals. The richest data (exact fuel rate corrections, aftertreatment internals, OEM-specific counters) frequently sits in Proprietary-B PGNs or in manufacturer-specific SPNs. Public decoders will show them as hex. Budget for OEM documentation or licensing, and do not promise customers signal coverage you have not seen on a real vehicle of that make and model year.

Revision drift. J1939DA changes. A signal’s resolution, range or even meaning can differ across revisions and across engine families. Pin your database version per vehicle and validate decoded values against known physical ranges at ingestion, rejecting impossible numbers loudly.

Silent data loss in TP. As discussed, dropped TP sessions look like “no fault” in the lab. Test with a recorded multi-fault trace and assert that your decoder reproduces every DTC.

Bus loading and active polling. A gateway that sends frequent requests, or is badly wired and produces error frames, can raise error counters on other ECUs and in the worst case cause them to go error-passive or bus-off. Listen-only is not just a security feature; it is a reliability feature.

Timestamp quality. Frames are broadcast with no timestamp. If you timestamp in a user-space loop under load, jitter of tens of milliseconds appears, which ruins event ordering across signals. Use kernel or hardware timestamps and, for multi-vehicle analytics, GNSS-disciplined time.

Address changes and duplicates. Retrofit equipment and trailers can claim unexpected addresses. A decoder keyed on SA alone will mislabel signals after such an event. Key on NAME.

Data volume economics. Without decimation, bills and storage grow linearly with fleet size and, painfully, not with value. Conversely, over-aggressive decimation hides short transients, such as a brief pressure dip, that preceded a failure. Event-triggered high-rate capture is the compromise, and it needs careful trigger design to avoid missing the unknown unknowns.

Privacy and regulation. Location and driver behaviour data fall under data-protection law in many jurisdictions. This post is technical and not legal advice, but minimize what you collect, document retention, and give operators control over it.

Practical Recommendations

Start with the question you need answered, then work backwards to signals. A maintenance product needs DM1 with correct multi-frame handling, hours and distance counters, coolant, oil and aftertreatment state; a fuel product needs speed, load, fuel rate and idle time at transient-capturing resolution. Pick the minimum set and expand.

Build the decoding layer around a versioned signal database rather than hand-coded offsets, and make “not available” and “error” distinct states in your schema. Implement transport protocol reassembly early and test it with recorded multi-packet traces, including loss of a data frame mid-session. Run the gateway listen-only, key your metadata on NAME, and timestamp at the kernel.

Decimate at the edge with deadband plus heartbeat, add event-triggered high-rate capture, and treat faults as a separate high-priority path. Use a bounded store-and-forward queue with an explicit drop policy and idempotent consumers over MQTT QoS 1. Keep decoder version and raw-frame samples so decoding bugs can be fixed retroactively.

A short checklist before you go to production:

Gateway is listen-only unless a PGN allow-list justifies transmitting.
Decoder rejects 0xFF/0xFE-family values and maps them to null with state.
BAM and RTS/CTS sessions are keyed, time-limited and tested with multi-fault traces.
SA-to-NAME table is rebuilt from observed address claims.
Per-signal deadband and heartbeat are defined and reviewed with the product owner.
Offline queue is bounded, with a documented drop order and flash-wear monitoring.
Per-device TLS identities, rotation and remote revocation exist.
Signal database version is stored with every record, plus a raw-frame sample archive.

Frequently Asked Questions

What is SAE J1939 used for?

SAE J1939 is a family of standards that defines how electronic control units on heavy-duty vehicles and machinery communicate over CAN. It specifies the 29-bit identifier layout, the parameters that engines, transmissions and brakes broadcast, address management, multi-packet transfer and diagnostics. Trucks, buses, agricultural and construction machines, gensets and marine engines use it, and telematics systems read it to monitor performance, fuel use and faults without querying each ECU.

What is the difference between a PGN and an SPN in J1939?

A Parameter Group Number (PGN) identifies a message: an 18-bit value derived from the extended data page, data page, PDU Format and, for PDU2 messages, PDU Specific fields. A Suspect Parameter Number (SPN) identifies a single parameter within a message, such as engine speed (SPN 190) inside PGN 61444. The PGN says which message you have; the SPN defines the start bit, length, resolution, offset and unit of one value inside it.

How does the J1939 transport protocol send more than 8 bytes?

It splits the message into packets of seven data bytes, each tagged with a sequence number, carrying up to 1,785 bytes in total (255 packets). A Connection Management frame (PGN 60416) opens the session and a Data Transfer frame (PGN 60160) carries each packet. BAM broadcasts without acknowledgment, while RTS/CTS performs a handshake with flow control and ends with an end-of-message acknowledgment, which suits point-to-point exchange.

Is J1939 the same as OBD-II?

No. OBD-II is a request-response emissions diagnostic interface for light-duty vehicles, using 11-bit identifiers over ISO 15765-4. J1939 is a broadcast-oriented protocol for heavy-duty vehicles using 29-bit identifiers, where data is published periodically and diagnostics (DM1) are broadcast. Heavy-duty regulations may reference J1939-73 or UDS-based ISO 27145, so a truck can expose both, but the addressing, transport and data models differ.

How do you decode J1939 data into engineering values?

Get the 29-bit ID from the CAN frame and compute the PGN and source address. Look up the PGN in a J1939 database, extract the SPN bits by start position and length (little-endian), reject reserved values such as 0xFF or 0xFFFF-range codes, then apply the formula value = raw x resolution + offset. For engine speed, raw 17,342 x 0.125 = 2,167.75 rpm. Use a licensed J1939DA-derived DBC rather than hand-coded offsets.

What bit rate does J1939 use?

Classic J1939 networks run CAN 2.0B at 250 kbit/s, and newer installations use 500 kbit/s under the J1939-14 physical layer. Both use the same identifier and PGN structure above the data link, so decoding logic is unaffected. Confirm the bus speed before connecting, since a gateway configured at the wrong rate generates error frames that can disturb other ECUs, which is another reason to start in listen-only mode.