3 posts tagged with "store-and-forward"

Here's the scenario every IIoT engineer dreads: your edge gateway is collecting temperature, pressure, and vibration data from 200 tags across 15 PLCs. The cellular modem on the factory roof drops its connection — maybe for 30 seconds during a handover, maybe for 4 hours because a backhoe hit a fiber line. When connectivity returns, what happens to the data?

If your answer is "it's gone," you have a buffer management problem. And fixing it properly requires understanding paged ring buffers — the unsung hero of reliable industrial telemetry.

Why Naive Buffering Fails​

The simplest approach — queue MQTT messages in memory and retry on reconnect — has three fatal flaws:

1. Memory exhaustion: A gateway reading 200 tags at 1-second intervals generates ~12,000 readings per minute. At ~100 bytes per JSON reading, that's 1.2 MB/minute. A 4-hour outage accumulates ~288 MB. Your 256 MB embedded gateway just died.

2. No delivery confirmation: MQTT QoS 1 guarantees "at least once" delivery, but the Mosquitto client library's in-flight message queue is finite. If you publish 50,000 messages into a disconnected client, most will be silently dropped by the client library's internal buffer long before the broker sees them.

3. Thundering herd on reconnect: When connectivity returns, dumping 288 MB of queued messages simultaneously will choke the cellular uplink (typically 1–5 Mbps), cause broker-side backpressure, and likely trigger another disconnect.

The Paged Ring Buffer Architecture​

The solution is a fixed-size, page-based circular buffer that sits between the data collection layer and the MQTT client. Here's how it works:

Memory Layout​

The buffer is allocated as a single contiguous block — typically 2 MB on an embedded gateway. This block is divided into equal-sized pages, where each page can hold one complete MQTT payload.

┌─────────────────────────────────────────────────┐
│                 2 MB Buffer Memory              │
├────────┬────────┬────────┬────────┬────────┬────┤
│ Page 0 │ Page 1 │ Page 2 │ Page 3 │ Page 4 │ ...│
│ 4 KB   │ 4 KB   │ 4 KB   │ 4 KB   │ 4 KB   │    │
└────────┴────────┴────────┴────────┴────────┴────┘

With a 4 KB page size and 2 MB total buffer, you get approximately 500 pages. Each page holds multiple MQTT messages packed sequentially.

Page States​

Every page exists in exactly one of three states:

• Free: Available for new data. Part of a singly-linked free list.
• Work: Currently being filled with incoming data. Only one work page exists at a time.
• Used: Full of data, waiting to be transmitted. Part of a singly-linked FIFO queue.

Free Pages → [P5] → [P6] → [P7] → null
Work Page  → [P3] (currently filling)
Used Pages → [P0] → [P1] → [P2] → null
              ↑ sending        waiting →

The Write Path​

When a batch of PLC tag values arrives from the data collection layer:

1. Check the work page: If there's no current work page, pop one from the free list. If the free list is empty, steal the oldest used page (overflow — we're losing old data to make room for new data, which is the correct trade-off for operational monitoring).

2. Calculate fit: Each message is packed as: [4-byte message ID] [4-byte message size] [message payload]. Check if the current work page has enough remaining space for this overhead plus the payload.

3. If it fits: Write the message ID (initially zero — will be filled by the MQTT client), the size, and the payload. Advance the write pointer.

4. If it doesn't fit: Move the current work page to the tail of the used queue. Pop a new page from the free list (or steal from used queue). Write into the new page.

Page Internal Layout:
┌──────────┬──────────┬─────────────┬──────────┬──────────┬─────────────┐
│ msg_id_1 │ msg_sz_1 │ payload_1   │ msg_id_2 │ msg_sz_2 │ payload_2   │
│ (4 bytes) │ (4 bytes) │ (N bytes)  │ (4 bytes) │ (4 bytes) │ (M bytes)  │
└──────────┴──────────┴─────────────┴──────────┴──────────┴─────────────┘
                                     ↑ write_p (current position)

The Send Path​

The MQTT send logic runs after every write operation and follows strict rules:

1. Check prerequisites: Connection must be up (connected == 1) AND no packet currently in-flight (packet_sent == 0). If either fails, do nothing — the data is safely buffered.

2. Select the send source: If there are used pages, send from the first one in the FIFO. If no used pages exist but the work page has data, promote the work page to used and send from it.

3. Read the next message from the current page's read pointer: extract the size, get the data pointer, and call mosquitto_publish() with QoS 1.

4. Mark packet as in-flight: Set packet_sent = 1. This is critical — only one message can be in-flight at a time. This prevents the thundering herd problem and ensures ordered delivery.

5. Wait for acknowledgment: The MQTT client library calls the publish callback when the broker confirms receipt (PUBACK for QoS 1). Only then do we advance the read pointer and send the next message.

The Acknowledgment Path​

When the Mosquitto library fires the on_publish callback with a packet ID:

1. Verify the ID matches the in-flight message on the current used page
2. Advance the read pointer past the delivered message (skip message ID + size + payload bytes)
3. Check if page is fully delivered: If read_p >= write_p, move the page back to the free list
4. Clear the in-flight flag: Set packet_sent = 0
5. Immediately attempt to send the next message — this creates a natural flow control where messages are delivered as fast as the broker can acknowledge them

Delivery Flow:
                     publish()
  [Used Page] ──────────────────→ [MQTT Broker]
       ↑                               │
       │              PUBACK            │
       └────────────────────────────────┘
       advance read_p, try next

Thread Safety: The Mutex Dance​

In a real gateway, data collection and MQTT delivery run on different threads. The PLC polling loop writes data every second, while the Mosquitto client library fires callbacks from its own network thread. Every buffer operation — add, send, acknowledge, connect, disconnect — must be wrapped in a mutex:

// Data collection thread:
mutex_lock(buffer)
  add_data(payload)
  try_send_next()  // opportunistic send
mutex_unlock(buffer)

// MQTT callback thread:
mutex_lock(buffer)
  mark_delivered(packet_id)
  try_send_next()  // chain next send
mutex_unlock(buffer)

The key insight is that try_send_next() is called from both threads — after every write (in case we're connected and idle) and after every acknowledgment (to chain the next message). This ensures maximum throughput without busy-waiting.

Handling Disconnects Gracefully​

When the MQTT connection drops, two things happen:

1. The disconnect callback fires: Set connected = 0 and packet_sent = 0. The in-flight message is NOT lost — it's still in the page at the current read pointer. When connectivity returns, it will be re-sent.

2. Data keeps flowing in: The PLC polling loop doesn't stop. New data continues to fill pages. The used queue grows. If it fills all available pages, new pages will steal from the oldest used pages — but this only happens under extreme sustained outages.

When the connection re-establishes:

1. The connect callback fires: Set connected = 1 and trigger try_send_next()
2. Buffered data starts flowing: Messages are delivered in FIFO order, one at a time, with acknowledgment pacing

This means the broker receives data in chronological order, with timestamps embedded in each batch. Analytics systems downstream can seamlessly handle the gap — they see a burst of historical data followed by real-time data, all correctly timestamped.

The Cloud Watchdog: Detecting Silent Failures​

There's a subtle failure mode: the MQTT connection appears healthy (no disconnect callback), but data isn't actually being delivered. This can happen with certain TLS middlebox issues, stale TCP connections that haven't timed out, or Azure IoT Hub token expirations.

The solution is a delivery watchdog:

1. Track the timestamp of the last successful packet delivery
2. On a periodic check (every 120 seconds), compare the current time against the last delivery timestamp
3. If no data has been delivered in 120 seconds AND the connection claims to be up, force a reconnection:
   • Reset the MQTT configuration timestamp (triggers config reload)
   • Clear the watchdog timer
   • The main loop will detect the stale configuration and restart the MQTT client

if (now - last_delivery_time > 120s) AND (connected) {
    log("No data delivered in 120s — forcing MQTT reconnect")
    force_mqtt_restart()
}

This catches the "zombie connection" problem that plagues many IIoT deployments — the gateway thinks it's sending, but nothing is actually arriving at the cloud.

Binary vs. JSON: The Bandwidth Trade-off​

The paged buffer doesn't care about the payload format — it stores raw bytes. But the choice between JSON and binary encoding has massive implications for buffer utilization:

JSON payload for one tag reading:

{"id":42,"values":[23.7],"ts":1709337600}

~45 bytes per reading.

Binary payload for the same reading:

Tag ID:    2 bytes (uint16)
Status:    1 byte
Value Cnt: 1 byte  
Value Sz:  1 byte
Value:     4 bytes (float32)
─────────────────────
Total:     9 bytes per reading

That's a 5x reduction. With batching (multiple readings per batch header), the per-reading overhead drops further because the timestamp and device identity are shared across a group of values.

On a cellular connection billing per megabyte, this isn't academic — it's the difference between $15/month and $75/month per gateway. On satellite connections (Iridium, Starlink maritime), it can be $50 vs. $250.

Binary Batch Wire Format​

A binary batch on the wire follows this structure:

[0xF7]                          — 1 byte, magic/version marker
[num_groups]                    — 4 bytes, big-endian uint32
For each group:
  [timestamp]                   — 4 bytes, big-endian time_t
  [device_type]                 — 2 bytes, big-endian uint16
  [serial_number]               — 4 bytes, big-endian uint32
  [num_values]                  — 4 bytes, big-endian uint32
  For each value:
    [tag_id]                    — 2 bytes, big-endian uint16
    [status]                    — 1 byte (0 = OK, else error code)
    If status == 0:
      [values_count]            — 1 byte
      [value_size]              — 1 byte (1, 2, or 4)
      [values...]               — values_count × value_size bytes

A batch of 50 tag readings fits in ~600 bytes binary versus ~3,000 bytes JSON. Over a 4-hour outage with 200 tags at 60-second intervals, that's the difference between buffering ~4.8 MB (binary) versus ~24 MB (JSON) — within or far exceeding a typical gateway's buffer.

Sizing Your Buffer: The Math​

For a given deployment, calculate your buffer needs:

Tags: 200
Read interval: 60 seconds
Binary payload per reading: ~9 bytes
Readings per minute: 200
Bytes per minute: 200 × 9 = 1,800 bytes
With batch overhead (~15 bytes per group): ~1,815 bytes/min

Buffer size: 2 MB = 2,097,152 bytes
Retention: 2,097,152 / 1,815 = ~1,155 minutes = ~19.2 hours

So a 2 MB buffer can hold approximately 19 hours of data for 200 tags at 60-second intervals using binary encoding. With JSON, that drops to ~3.8 hours. Size your buffer accordingly.

What machineCDN Does Differently​

machineCDN's edge gateway implements this paged ring buffer architecture natively. Every gateway shipped includes:

• Fixed 2 MB paged buffer with configurable page sizes matching the MQTT broker's maximum packet size
• Automatic binary encoding for all telemetry — 5x bandwidth reduction over JSON
• Single-message flow control with QoS 1 acknowledgment tracking — no thundering herd on reconnect
• 120-second delivery watchdog that detects zombie connections and forces reconnect
• Graceful overflow handling — when buffer fills, oldest data is recycled (not newest), preserving the most recent operational state

For plant engineers, this means deploying a gateway on a cellular connection and knowing that a connectivity outage — whether 30 seconds or 12 hours — won't result in lost data. The buffer holds, the watchdog monitors, and data flows in order when the link comes back.

Key Takeaways​

1. Never use unbounded queues for industrial telemetry buffering — use fixed-size paged buffers that degrade gracefully under memory pressure
2. One message in-flight at a time prevents the thundering herd problem and ensures ordered delivery
3. Always track delivery acknowledgments — don't just publish and forget; verify the broker received each packet before advancing
4. Implement a delivery watchdog — silent MQTT failures are harder to detect than disconnects
5. Use binary encoding — 5x bandwidth reduction means 5x longer buffer retention on the same memory
6. Size for your worst outage — calculate how much buffer you need based on tag count, interval, and the longest connectivity gap you expect
7. Thread safety is non-negotiable — data collection and MQTT delivery run concurrently; every buffer operation needs mutex protection

The paged ring buffer isn't exotic computer science — it's a practical engineering pattern that's been battle-tested in thousands of industrial deployments. The difference between a prototype IIoT system and a production one often comes down to exactly this kind of infrastructure.

Your edge gateway reads 200 tags from a PLC every second. The MQTT connection to your cloud broker drops for 3 minutes because someone bumped the cellular antenna. What happens to the 36,000 data points collected during the outage?

If your answer is "they're gone," you have a toy system, not an industrial one.

Reliable telemetry delivery is the hardest unsolved problem in most IIoT architectures. Everyone focuses on the protocol layer — Modbus reads, EtherNet/IP connections, OPC-UA subscriptions — but the real engineering is in what happens between reading a value and confirming it reached the cloud. This article breaks down the buffer architecture that makes zero-data-loss telemetry possible on resource-constrained edge hardware.

The Problem: Three Asynchronous Timelines​

In any edge-to-cloud telemetry system, you're managing three independent timelines:

1. PLC read cycle — Tags are read at fixed intervals (1s, 60s, etc.). This never stops. The PLC doesn't care if your cloud connection is down.

2. Batch collection — Raw tag values are grouped into batches by timestamp and device. Batches accumulate until they hit a size limit or a timeout.

3. MQTT delivery — Batches are published to the broker. The broker acknowledges receipt. At QoS 1, the MQTT library handles retransmission, but only if you give it data in the right form.

These three timelines run independently. The PLC read loop runs on a tight 1-second cycle. Batch finalization might happen every 30–60 seconds. MQTT delivery depends on network availability. If any one of these stalls, the others must keep running without data loss.

This is fundamentally a producer-consumer problem with a twist: the consumer (MQTT) can disappear for minutes at a time, and the producer (PLC reads) cannot slow down.

The Batch Layer: Grouping Values for Efficient Transport​

Raw tag values are tiny — a temperature reading is 4 bytes, a boolean is 1 byte. Sending each value as an individual MQTT message would be absurdly wasteful. Instead, values are collected into batches — structured payloads that contain multiple timestamped readings from one or more devices.

Batch Structure​

A batch is organized as a series of groups, where each group represents one polling cycle (one timestamp, one device):

Batch
├── Group 0: { timestamp: 1709284800, device_type: 5000, serial: 12345 }
│   ├── Value: { id: 2, values: [72.4] }          // Delivery Temp
│   ├── Value: { id: 3, values: [68.1] }          // Mold Temp
│   └── Value: { id: 5, values: [12.6] }          // Flow Value
├── Group 1: { timestamp: 1709284860, device_type: 5000, serial: 12345 }
│   ├── Value: { id: 2, values: [72.8] }
│   ├── Value: { id: 3, values: [68.3] }
│   └── Value: { id: 5, values: [12.4] }
└── ...

Dual-Format Encoding: JSON vs Binary​

Production edge daemons typically support two encoding formats for batches, and the choice has massive implications for bandwidth:

JSON format:

{
  "groups": [
    {
      "ts": 1709284800,
      "device_type": 5000,
      "serial_number": 12345,
      "values": [
        {"id": 2, "values": [72.4]},
        {"id": 3, "values": [68.1]}
      ]
    }
  ]
}

Binary format (same data):

Header:  F7                           (1 byte - magic)
Groups:  00 00 00 01                  (4 bytes - group count)
Group 0: 65 E5 A0 00                  (4 bytes - timestamp)
         13 88                        (2 bytes - device type: 5000)
         00 00 30 39                  (4 bytes - serial number)
         00 00 00 02                  (4 bytes - value count)
Value 0: 00 02                        (2 bytes - tag id)
         00                           (1 byte  - status: OK)
         01                           (1 byte  - values count)
         04                           (1 byte  - element size: 4 bytes)
         42 90 CC CD                  (4 bytes - float 72.4)
Value 1: 00 03
         00
         01
         04
         42 88 33 33                  (4 bytes - float 68.1)

The JSON version of this payload: ~120 bytes. The binary version: ~38 bytes. That's a 3.2x reduction — and on a metered cellular connection at $0.01/MB, that savings compounds quickly when you're transmitting every 30 seconds 24/7.

The binary format uses a simple TLV-like structure: magic byte, group count (big-endian uint32), then for each group: timestamp (uint32), device type (uint16), serial number (uint32), value count (uint32), then for each value: tag ID (uint16), status byte, value count, element size, and raw value bytes. No field names, no delimiters, no escaping — just packed binary data.

Batch Finalization Triggers​

A batch should be finalized (sealed and queued for delivery) when either condition is met:

1. Size limit exceeded — When the accumulated batch size exceeds a configured maximum (e.g., 500KB for JSON, or when the binary buffer is 90%+ full). The 90% threshold for binary avoids the edge case where the next value would overflow the buffer.

2. Collection timeout expired — When elapsed time since the batch started exceeds a configured maximum (e.g., 60 seconds). This ensures data flows even during quiet periods with few value changes.

if (elapsed_seconds > max_collection_time) → finalize
if (batch_size > max_batch_size)          → finalize

Both checks happen after every group is closed (after every polling cycle). This means finalization granularity is tied to your polling interval — if you poll every 1 second and your batch timeout is 60 seconds, each batch will contain roughly 60 groups.

The "Do Not Batch" Exception​

Some values are too important to wait for batch finalization. Equipment alarms, pump state changes, emergency stops — these need to reach the cloud immediately. These tags are flagged as "do not batch" in the configuration.

When a do-not-batch tag changes value, it bypasses the normal batch pipeline entirely. A mini-batch is created on the spot — containing just that single value — and pushed directly to the outgoing buffer. This ensures sub-second cloud visibility for critical state changes, while bulk telemetry still benefits from batch efficiency.

Tag: "Pump Status"     interval: 1s    do_not_batch: true
Tag: "Heater Status"   interval: 1s    do_not_batch: true
Tag: "Delivery Temp"   interval: 60s   do_not_batch: false  ← normal batching

The Buffer Layer: Surviving Disconnections​

This is where most IIoT implementations fail. The batch layer produces data. The MQTT layer consumes it. But what sits between them? If it's just an in-memory queue, you'll lose everything on disconnect.

Page-Based Ring Buffer Architecture​

The production-grade answer is a page-based ring buffer — a fixed-size memory region divided into equal-sized pages that cycle through three states:

States:
  FREE  → Available for writing
  WORK  → Currently being filled with batch data
  USED  → Filled, waiting for MQTT delivery

Lifecycle:
  FREE → WORK (when first data is added)
  WORK → USED (when page is full or batch is finalized)
  USED → transmit → delivery ACK → FREE (recycled)

Here's how it works:

Memory layout: At startup, a contiguous block of memory is allocated (e.g., 2MB). This block is divided into pages of a configured size (matching the MQTT max packet size, typically matching the batch size). Each page has a small header tracking its state and a data area.

┌──────────────────────────────────────────────┐
│ [Page 0: USED] [Page 1: USED] [Page 2: WORK]│
│ [Page 3: FREE] [Page 4: FREE] [Page 5: FREE]│
│ [Page 6: FREE] ... [Page N: FREE]            │
└──────────────────────────────────────────────┘

Writing data: When a batch is finalized, its serialized bytes are written to the current WORK page. Each message gets a small header: a 4-byte message ID slot (filled later by the MQTT library) and a 4-byte size field. If the current page can't fit the next message, it transitions to USED and a fresh FREE page becomes the new WORK page.

Overflow handling: When all FREE pages are exhausted, the buffer reclaims the oldest USED page — the one that's been waiting for delivery the longest. This means you lose old data rather than new data, which is the right trade-off: the most recent readings are the most valuable. An overflow warning is logged so operators know the buffer is under pressure.

Delivery: When the MQTT connection is active, the buffer walks through USED pages and publishes their contents. Each publish gets a packet ID from the MQTT library. When the broker ACKs the packet (via the PUBACK callback for QoS 1), the corresponding page is recycled to FREE.

Disconnection recovery: When the MQTT connection drops:

1. The disconnect callback fires
2. The buffer marks itself as disconnected
3. Data continues accumulating in pages (WORK → USED)
4. When reconnected, the buffer immediately starts draining USED pages

No data is lost unless the buffer physically overflows. With 2MB of buffer and 500KB page size, you get 4 pages of headroom — enough to survive several minutes of disconnection at typical telemetry rates.

Thread Safety​

The PLC read loop and the MQTT event loop run on different threads. The buffer must be thread-safe. Every buffer operation acquires a mutex:

• buffer_add_data() — called from the PLC read thread after batch finalization
• buffer_process_data_delivered() — called from the MQTT callback thread on PUBACK
• buffer_process_connect() / buffer_process_disconnect() — called from MQTT lifecycle callbacks

Without proper locking, you'll see corrupted pages, double-free crashes, and mysterious data loss under load. This is non-negotiable.

Sizing the Buffer​

Buffer sizing depends on three variables:

1. Data rate: How many bytes per second does your polling loop produce?
2. Expected outage duration: How long do you need to survive without MQTT?
3. Available memory: Edge devices (especially industrial routers) have limited RAM

Example calculation:

• 200 tags, average 6 bytes each (including binary overhead) = 1,200 bytes/group
• Polling every 1 second = 1,200 bytes/second = 72KB/minute
• Target: survive 30-minute outage = 2.16MB buffer
• With 500KB pages = 5 pages minimum (round up for safety)

In practice, 2–4MB covers most scenarios. On a 32MB industrial router, that's well within budget.

The MQTT Layer: QoS, Reconnection, and Watchdogs​

QoS 1: At-Least-Once Delivery​

For industrial telemetry, QoS 1 is the right choice:

• QoS 0 (fire and forget): No delivery guarantee. Unacceptable for production data.
• QoS 1 (at least once): Broker ACKs every message. Duplicates possible but data loss prevented. Good trade-off.
• QoS 2 (exactly once): Eliminates duplicates but doubles the handshake overhead. Rarely worth it for telemetry.

The page buffer's recycling logic depends on QoS 1: pages are only freed when the PUBACK arrives. If the ACK never comes (connection drops mid-transmission), the page stays in USED state and will be retransmitted after reconnection.

Connection Watchdog​

MQTT connections can enter a zombie state — the TCP socket is open, the MQTT loop is running, but no data is actually flowing. This happens when network routing changes, firewalls silently drop the connection, or the broker becomes unresponsive.

The fix: a watchdog timer that monitors delivery acknowledgments. If no PUBACK has been received within a timeout window (e.g., 120 seconds) and data has been queued for transmission, force a reconnect:

if (now - last_delivered_packet_time > 120s) {
  if (has_pending_data) {
    // Force MQTT reconnection
    reset_mqtt_client();
  }
}

This catches the edge case where the MQTT library thinks it's connected but the network is actually dead. Without this watchdog, your edge daemon could silently accumulate hours of undelivered data in the buffer, eventually overflowing and losing it all.

Asynchronous Connection​

MQTT connection establishment (DNS resolution, TLS handshake, CONNACK) can take several seconds, especially over cellular links. This must not block the PLC read loop. The connection should happen on a separate thread:

1. Main thread detects connection is needed
2. Connection thread starts connect_async()
3. Main thread continues reading PLCs
4. On successful connect, the callback fires and buffer delivery begins

If the connection thread is still working when a new connection attempt is needed, skip it — don't queue multiple connection attempts or you'll thrash the network stack.

TLS for Production​

Any MQTT connection leaving your plant network must use TLS. Period. Industrial telemetry data — temperatures, pressures, equipment states, alarm conditions — is operationally sensitive. On the wire without encryption, anyone on the network path can see (and potentially modify) your readings.

For cloud brokers like Azure IoT Hub, TLS is mandatory. The edge daemon should:

• Load the CA certificate from a PEM file
• Use MQTT v3.1.1 protocol (widely supported, well-tested)
• Monitor the SAS token expiration timestamp and alert before it expires
• Automatically reinitialize the MQTT client when the certificate or connection string changes (file modification detected via stat())

Daemon Status Reporting​

A well-designed edge daemon reports its own health back through the same MQTT channel it uses for telemetry. A periodic status message should include:

• System uptime and daemon uptime — detect restarts
• PLC link state — is the PLC connection healthy?
• Buffer state — how full is the outgoing buffer?
• MQTT state — connected/disconnected, last ACK time
• SAS token expiration — days until credentials expire
• Software version — for remote fleet management

An extended status format can include per-tag state: last read time, last delivery time, current value, and error count. This is invaluable for remote troubleshooting — you can see from the cloud exactly which tags are stale and why.

Value Comparison and Change Detection​

Not all values need to be sent every polling cycle. A temperature that's been 72.4°F for the last hour doesn't need to be transmitted 3,600 times. Change detection — comparing the current value to the last sent value — can dramatically reduce bandwidth.

The implementation: each tag stores its last transmitted value. After reading, compare:

if (tag.compare_enabled && tag.has_been_read_once) {
  if (current_value == tag.last_value) {
    skip_this_value();  // Don't add to batch
  }
}

Important caveats:

• Not all tags should use comparison. Continuous process variables (temperatures, flows) should always send, even if unchanged — the recipient needs the full time series to calculate trends and detect flatlines (a stuck sensor reads the same value forever, which is itself a fault condition).
• Discrete state tags (booleans, enums) are ideal for comparison — they change rarely and each change is significant.
• Floating-point comparison should use an epsilon threshold, not exact equality, to avoid sending noise from ADC jitter.

Putting It All Together: The Main Loop​

The complete edge daemon main loop ties all these layers together:

1. Parse configuration (device addresses, tag lists, MQTT credentials)
2. Allocate memory (PLC config pool + output buffer)
3. Format output buffer into pages
4. Start MQTT connection thread
5. Detect PLC device (probe address, determine type/protocol)
6. Load device-specific tag configuration

MAIN LOOP (runs every 1 second):
  a. Check for config file changes → restart if changed
  b. Read PLC tags (coalesced Modbus/EtherNet/IP)
  c. Add values to batch (with comparison filtering)
  d. Check batch finalization triggers (size/timeout)
  e. Process incoming commands (config updates, force reads)
  f. Check MQTT connection watchdog
  g. Sleep 1 second

Every component — polling, batching, buffering, delivery — operates within this single loop iteration, keeping the system deterministic and debuggable.

How machineCDN Implements This​

The machineCDN edge runtime implements this full stack natively on resource-constrained industrial routers. The page-based ring buffer runs in pre-allocated memory (no dynamic allocation after startup), the MQTT layer handles Azure IoT Hub and local broker configurations interchangeably, and the batch layer supports both JSON and binary encoding selectable per-device.

On a Teltonika RUT9xx router with 256MB RAM, the daemon typically uses under 4MB total — including 2MB of buffer space that can store 20+ minutes of telemetry during a connectivity outage. Tags are automatically sorted, coalesced, and dispatched with zero configuration beyond listing the tag names and addresses.

The result: edge gateways that have been running continuously for years in production environments, surviving cellular dropouts, network reconfigurations, and even firmware updates without losing a single data point.

Conclusion​

Reliable telemetry delivery isn't about the protocol — it's about the pipeline. Modbus reads are the easy part. The hard engineering is in the layers between: batching values efficiently, buffering them through disconnections, and confirming delivery before recycling memory.

The key design principles:

1. Never block the read loop — PLC polling is sacred
2. Buffer with finite, pre-allocated memory — dynamic allocation on embedded systems is asking for trouble
3. Reclaim oldest data first — in overflow, recent values matter more
4. Acknowledge before recycling — a page stays USED until the broker confirms receipt
5. Watch for zombie connections — a connected socket doesn't mean data is flowing

Get these right, and your edge infrastructure becomes invisible — which is exactly what production IIoT should be.

Here's an uncomfortable truth about industrial IoT: your cloud platform is only as reliable as the worst cellular connection on your factory floor.

And in manufacturing environments — where concrete walls, metal enclosures, and electrical noise are the norm — that connection can drop for minutes, hours, or days. If your edge architecture doesn't account for this, you're not building an IIoT system. You're building a fair-weather dashboard that goes dark exactly when you need it most.

This guide covers the architecture patterns that separate production-grade edge gateways from science projects: store-and-forward buffering, intelligent batch processing, binary serialization, and the MQTT reliability patterns that actually work when deployed on a $200 industrial router with 256MB of RAM.