One post tagged with "binary-encoding"

JSON is killing your cellular data budget.

When your edge gateway publishes a single temperature reading as {"tag_id": 42, "value": 23.45, "type": "float", "status": 0, "ts": 1709312400}, that's 72 bytes of text to convey 10 bytes of actual information: a 2-byte tag ID, a 4-byte float, a 1-byte status code, and a 4-byte timestamp (which is shared across all tags in the same poll cycle anyway).

At 200 tags polled every 5 seconds, JSON payloads consume roughly 100 KB/minute — over 4 GB/month. On a $15/month cellular plan with a 1 GB cap, you've blown your data budget by day 8.

Binary encoding solves this. By designing a compact wire format purpose-built for industrial telemetry, you can reduce per-tag overhead from ~70 bytes to ~7 bytes — a 10x reduction that makes cellular and satellite IIoT deployments economically viable.

This article covers the engineering of binary payload formats for industrial MQTT, from byte-level encoding decisions to the buffering and delivery systems that ensure data integrity.

Why JSON Falls Short for Industrial Telemetry​

JSON became the default payload format for MQTT in the IIoT world because it's human-readable, self-describing, and every platform can parse it. These are real advantages during development and debugging. But they come at a cost that compounds brutally at scale.

The Overhead Tax​

Let's dissect a typical JSON telemetry message:

{
  "device_type": 1010,
  "serial": 1106550353,
  "ts": 1709312400,
  "tags": [
    {"id": 1, "status": 0, "type": "uint16", "values": [4200]},
    {"id": 2, "status": 0, "type": "float", "values": [23.45]},
    {"id": 3, "status": 0, "type": "bool", "values": [1]}
  ]
}

This payload is approximately 250 bytes. The actual data content:

• Device type: 2 bytes
• Serial number: 4 bytes
• Timestamp: 4 bytes
• 3 tag values: 2 + 4 + 1 = 7 bytes
• 3 tag IDs: 6 bytes
• 3 status codes: 3 bytes

Total useful data: 26 bytes. The other 224 bytes are structural overhead — curly braces, square brackets, quotation marks, colons, commas, key names, and redundant type strings.

That's an overhead ratio of 9.6x. For every byte of machine data, you're transmitting nearly 10 bytes of JSON syntax.

CPU Cost on Embedded Gateways​

JSON serialization isn't free on embedded hardware. Constructing JSON objects, converting numbers to strings, escaping special characters, and computing string lengths all consume CPU cycles that could be spent polling more tags or running edge analytics.

On an ARM Cortex-A7 gateway (common in industrial routers), JSON serialization of a 200-tag batch takes 2–5ms. The equivalent binary encoding takes 200–500μs — an order of magnitude faster. When you're polling Modbus every second and need to leave CPU headroom for other tasks, this matters.

Designing a Binary Telemetry Format​

A practical binary format for industrial MQTT must balance compactness with extensibility. Here's a proven structure used in production industrial gateways.

Message Structure​

┌─────────────────────────────────────────┐
│ Header                                  │
│  ├─ Timestamp (4 bytes, uint32)         │
│  ├─ Device Type (2 bytes, uint16)       │
│  └─ Serial Number (4 bytes, uint32)     │
├─────────────────────────────────────────┤
│ Tag Group                               │
│  ├─ Tag Count (2 bytes, uint16)         │
│  ├─ Tag Record 1                        │
│  │   ├─ Tag ID (2 bytes, uint16)        │
│  │   ├─ Status (1 byte, uint8)          │
│  │   ├─ Type (1 byte, uint8)            │
│  │   ├─ Value Count (1 byte, uint8)     │
│  │   └─ Values (variable)              │
│  ├─ Tag Record 2                        │
│  │   └─ ...                             │
│  └─ Tag Record N                        │
└─────────────────────────────────────────┘

Type Encoding​

Use a single byte to encode the value type, which also determines the byte width of each value:

This type system covers every data type you'll encounter in Modbus and EtherNet/IP PLCs. The decoder uses the type code to determine exactly how many bytes to read for each value — no parsing ambiguity, no delimiter scanning.

Size Comparison​

For the same 3-tag example above:

Binary encoding:

• Header: 10 bytes (timestamp + device type + serial)
• Tag count: 2 bytes
• Tag 1 (uint16): 2 + 1 + 1 + 1 + 2 = 7 bytes
• Tag 2 (float32): 2 + 1 + 1 + 1 + 4 = 9 bytes
• Tag 3 (bool): 2 + 1 + 1 + 1 + 1 = 6 bytes

Total: 34 bytes vs. 250 bytes for JSON. That's a 7.3x reduction.

The savings compound as tag count increases. At 100 tags (a typical mid-size PLC), a JSON batch runs 6–8 KB; the binary equivalent is 700–900 bytes. At 200 tags, JSON hits 12–16 KB while binary stays under 2 KB.

Data Grouping: Batches and Groups​

Individual tag values shouldn't be published as individual MQTT messages. The MQTT protocol itself adds overhead: a PUBLISH packet includes a fixed header (2 bytes minimum), topic string (20–50 bytes for a typical industrial topic), and packet identifier (2 bytes for QoS 1). Publishing 200 individual messages means 200× this overhead.

Timestamp-Grouped Batches​

The most effective grouping strategy collects all tag values from a single poll cycle into one batch, sharing a single timestamp:

[Batch Start: timestamp=1709312400]
  Tag 1: id=1, status=0, type=uint16, value=4200
  Tag 2: id=2, status=0, type=float,  value=23.45
  Tag 3: id=3, status=0, type=bool,   value=1
  ...
[Batch End]

The timestamp in the batch header applies to all contained tags. This eliminates per-tag timestamp overhead — a savings of 4 bytes per tag, or 800 bytes across 200 tags.

Batch Size Limits​

MQTT brokers and clients have maximum message size limits. Azure IoT Hub limits messages to 256 KB. AWS IoT Core allows 128 KB. Most on-premise Mosquitto deployments default to 256 MB but should be configured lower for production use.

More importantly, your edge gateway's memory and processing constraints impose practical limits. A 4 KB batch size works well for most deployments:

• Large enough to hold 200+ tags in binary format
• Small enough to fit in constrained gateway memory
• Fast enough to serialize without impacting the poll loop

When a batch exceeds the configured size, close it and start a new one. The cloud decoder handles multiple batches with the same timestamp gracefully.

Change-of-Value Filtering Before Batching​

Apply change-of-value (COV) filtering before adding values to the batch, not after. If a tag's value hasn't changed since the last report and COV is enabled for that tag, skip it entirely. This reduces batch sizes further during steady-state operation — when 80% of tags are unchanged, your binary batch shrinks proportionally.

However, implement a periodic full-refresh: every hour (or configurable interval), reset all COV baselines and include every tag in the next batch. This ensures the cloud always has a complete snapshot, even if individual change events were lost during a brief disconnection.

The Page Buffer: Store-and-Forward in Fixed Memory​

Binary encoding solves the bandwidth problem. But you still need to handle MQTT disconnections without losing data. The page-based ring buffer is the industrial standard for store-and-forward in embedded systems.

Architecture​

Pre-allocate a contiguous memory region at startup and divide it into fixed-size pages:

┌────────────────────────────────────────────────┐
│ Buffer Memory (e.g., 512 KB)                   │
│                                                │
│ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ │
│ │Page 0│ │Page 1│ │Page 2│ │Page 3│ │Page 4│ │
│ │      │ │      │ │      │ │      │ │      │ │
│ └──────┘ └──────┘ └──────┘ └──────┘ └──────┘ │
└────────────────────────────────────────────────┘

Pages cycle through three states:

1. Free — empty, available for writing
2. Work — currently being written to by the Modbus polling thread
3. Used — full, waiting for MQTT delivery

Page Layout​

Each page contains multiple messages, packed sequentially:

┌─────────────────────────────────────┐
│ Page Header (struct, ~16 bytes)     │
├─────────────────────────────────────┤
│ Message 1:                          │
│  ├─ Message ID (4 bytes)            │
│  ├─ Message Size (4 bytes)          │
│  └─ Message Body (variable)         │
├─────────────────────────────────────┤
│ Message 2:                          │
│  ├─ Message ID (4 bytes)            │
│  ├─ Message Size (4 bytes)          │
│  └─ Message Body (variable)         │
├─────────────────────────────────────┤
│ ... (more messages)                 │
├─────────────────────────────────────┤
│ Free space                          │
└─────────────────────────────────────┘

The 4-byte message ID field is filled by the MQTT library when the message is published (at QoS 1). The gateway uses this ID to match publish acknowledgments to specific messages.

Write Path​

1. Check if the current work page has enough space for the new message (message size + 8 bytes for ID and size fields).
2. If yes: write the message, advance the write pointer.
3. If no: move the work page to the "used" queue, grab a free page as the new work page, and write there.
4. If no free pages exist: grab the oldest used page (overflow condition). Log a warning — you're losing the oldest buffered data, but preserving the newest.

This overflow strategy is deliberately biased toward fresh data. In industrial monitoring, a temperature reading from 5 minutes ago is far more valuable than one from 3 days ago that was buffered during an outage.

Delivery Path​

1. Take the first page from the "used" queue.
2. Read the next undelivered message (tracked by a per-page read pointer).
3. Publish via MQTT at QoS 1.
4. Wait for PUBACK — don't advance the read pointer until the broker confirms receipt.
5. On PUBACK: advance the read pointer. If the page is fully delivered, move it back to "free."
6. On disconnect: stop sending, keep writing. The buffer absorbs the outage.

The wait-for-PUBACK step is critical. Without it, you're fire-and-forgetting into a potentially disconnected socket, and data silently disappears.

Thread Safety​

The write path (Modbus polling thread) and delivery path (MQTT thread) operate concurrently on the same buffer. A mutex protects all page state transitions:

• Moving pages between free/work/used queues
• Checking available space
• Advancing read/write pointers
• Processing delivery acknowledgments

Keep the critical section as small as possible — lock, update pointers, unlock. Never hold the mutex during a Modbus read or MQTT publish; those operations can block for seconds.

Delivery Tracking and Watchdogs​

In production, "the MQTT connection is up" doesn't mean data is flowing. The connection can be technically alive (TCP socket open, keepalives passing) while messages silently fail to publish or acknowledge.

Delivery Timestamp Tracking​

Track the timestamp of the last successfully delivered message (confirmed by PUBACK). If this timestamp falls more than N minutes behind the current time, something is wrong:

• The broker may be rejecting messages (payload too large, topic permission denied)
• The network may be passing keepalives but dropping data packets
• The MQTT library may be stuck in an internal error state

When the delivery watchdog fires, tear down the entire MQTT connection and reinitialize. It's a heavy-handed recovery, but it's reliable. In industrial systems, a clean restart beats a subtle degradation every time.

Status Telemetry​

The gateway should periodically publish its own status message containing:

• Daemon uptime — how long since last restart
• System uptime — how long since last boot
• Buffer state — pages free/used/work, current fill level
• PLC link state — is the Modbus connection healthy
• Firmware version — for remote fleet management
• Token expiration — time remaining on the MQTT auth token

This status message can use JSON even if data messages use binary — it's infrequent (every 30–60 seconds) and readability matters more than compactness for diagnostics.

Bandwidth Math: Real-World Numbers​

Let's calculate the actual savings for a typical deployment:

Scenario: 150 tags, polled every 5 seconds, 50% change rate with COV enabled, cellular connection.

JSON Format​

• Average tag JSON: ~60 bytes
• Tags per poll (with 50% COV): 75
• Batch overhead: ~50 bytes
• Total per poll: 75 × 60 + 50 = 4,550 bytes
• Per minute (12 polls): 54.6 KB
• Per day: 78.6 MB
• Per month: 2.36 GB

Binary Format​

• Average tag binary: ~7 bytes
• Header per batch: 12 bytes
• Total per poll: 75 × 7 + 12 = 537 bytes
• Per minute (12 polls): 6.4 KB
• Per day: 9.3 MB
• Per month: 279 MB

Savings: 88% reduction — from 2.36 GB to 279 MB. On a $20/month cellular plan with 500 MB included, JSON doesn't fit. Binary does, with headroom.

Add MQTT overhead (topic strings, packet headers) and TLS overhead (~40 bytes per record), and real-world savings are slightly less dramatic but still consistently in the 8–10x range.

Decoding on the Cloud Side​

Binary encoding shifts complexity from the edge to the cloud. The decoder must:

1. Parse the header to extract timestamp, device type, and serial number.
2. Iterate tag records using the type code to determine value byte widths.
3. Reconstruct typed values — particularly IEEE 754 floats from their 4-byte binary representation.
4. Handle partial messages — if a batch was truncated due to buffer overflow, the decoder must fail gracefully on the last incomplete record without losing the valid records before it.

Most cloud platforms (Azure IoT Hub, AWS IoT Core) support custom message decoders that transform binary payloads to JSON for downstream processing. Write the decoder once, and the rest of your analytics pipeline sees standard JSON.

How machineCDN Implements Binary Telemetry​

machineCDN's edge daemon uses binary encoding by default for all data telemetry. The implementation includes:

• Compact binary batching with shared timestamps per group, reducing per-tag overhead to 5–9 bytes depending on data type.
• Page-based ring buffer with pre-allocated memory, zero runtime allocation, and deliberate overflow behavior that preserves fresh data.
• Per-message PUBACK tracking with delivery watchdog and automatic connection recycling.
• Parallel JSON status messages for gateway diagnostics, published on a separate topic at lower frequency.
• Automatic format negotiation — the cloud ingestion layer detects binary vs. JSON based on the first byte of the payload and routes to the appropriate decoder.

The result: machineCDN gateways routinely operate on 500 MB/month cellular plans, monitoring 200+ tags at 5-second intervals, with full store-and-forward resilience during connectivity outages.

When to Use Binary vs. JSON​

Binary encoding isn't always the right choice. Use this decision framework:

For most production industrial deployments — where gateways connect hundreds of tags over cellular and reliability trumps developer convenience — binary encoding is the clear winner. Save JSON for your status messages and the debugging serial port.

Getting Started​

If you're designing a binary telemetry format for your own gateway:

1. Start with the type system. Define your type codes and byte widths. Match them to your PLC's native data types.
2. Design the header. Include version, device identity, and a shared timestamp. Add a format version byte so you can evolve the format without breaking old decoders.
3. Build the buffer first. Get store-and-forward working before optimizing the encoding. Data integrity matters more than data compactness.
4. Write the decoder alongside the encoder. Test with known values. Verify float encoding especially — IEEE 754 byte ordering bugs are silent and devastating.
5. Measure real bandwidth. Deploy both JSON and binary formats on the same gateway for a week and compare actual data consumption. The numbers will sell the approach to stakeholders who question the added complexity.

Binary encoding is a solved problem in industrial telemetry. The patterns are well-established, the savings are dramatic, and the complexity cost is paid once at design time and amortized across every byte your fleet ever transmits.