2 posts tagged with "watchdog"

Here's a scenario every IIoT engineer dreads: it's 2 AM on a Saturday, your edge gateway in a plastics manufacturing plant has lost its MQTT connection to the cloud, and nobody notices until Monday morning. Forty-eight hours of production data — temperatures, pressures, cycle counts, alarms — gone. The maintenance team wanted to correlate a quality defect with process data from Saturday afternoon. They can't.

This is a reliability problem, and it's solvable. The patterns that separate a production-grade edge gateway from a prototype are: configuration hot-reload (change settings without restarting), connection watchdogs (detect and recover from silent failures), and graceful resource management (handle reconnections without memory leaks).

This guide covers the architecture behind each of these patterns, with practical design decisions drawn from real industrial deployments.

The Problem: Why Edge Gateways Fail Silently​

Industrial edge gateways operate in hostile environments: temperature swings, electrical noise, intermittent network connectivity, and 24/7 uptime requirements. The failure modes are rarely dramatic — they're insidious:

• MQTT connection drops silently. The broker stops responding, but the client library doesn't fire a disconnect callback because the TCP connection is still half-open.
• Configuration drift. An engineer updates tag definitions on the management server, but the gateway is still running the old configuration.
• Memory exhaustion. Each reconnection allocates new buffers without properly freeing the old ones. After enough reconnections, the gateway runs out of memory and crashes.
• PLC link flapping. The PLC reboots or loses power briefly. The gateway keeps polling, getting errors, but never properly re-detects or reconnects.

Solving these requires three interlocking systems: hot-reload for configuration, watchdogs for connections, and disciplined resource management.

Pattern 1: Configuration File Hot-Reload​

The simplest and most robust approach to configuration hot-reload is file-based with stat polling. The gateway periodically checks if its configuration file has been modified (using the file's modification timestamp), and if so, reloads and applies the new configuration.

Design: stat() Polling vs. inotify​

You have two options for detecting file changes:

stat() polling — Check the file's st_mtime on every main loop iteration:

on_each_cycle():
    current_stat = stat(config_file)
    if current_stat.mtime != last_known_mtime:
        reload_configuration()
        last_known_mtime = current_stat.mtime

inotify (Linux) — Register for kernel-level file change notifications:

fd = inotify_add_watch(config_file, IN_MODIFY)
poll(fd)  // blocks until file changes
reload_configuration()

For industrial edge gateways, stat() polling wins. Here's why:

1. It's simpler. No file descriptor management, no edge cases with inotify watches being silently dropped.
2. It works across filesystems. inotify doesn't work on NFS, CIFS, or some embedded filesystems. stat() works everywhere.
3. The cost is negligible. A single stat() call takes ~1 microsecond. Even at 1 Hz, it's invisible.
4. It naturally integrates with the main loop. Industrial gateways already run a polling loop for PLC reads. Adding a stat() check is one line.

Graceful Reload: The Teardown-Rebuild Cycle​

When a configuration change is detected, the gateway must:

1. Stop active PLC connections. For EtherNet/IP, destroy all tag handles. For Modbus, close the serial port or TCP connection.
2. Free allocated memory. Tag definitions, batch buffers, connection contexts — all of it.
3. Re-read and validate the new configuration.
4. Re-detect the PLC and re-establish connections with the new tag map.
5. Resume data collection with a forced initial read of all tags.

The critical detail is step 2. Industrial gateways often use a pool allocator instead of individual malloc/free calls. All configuration-related memory is allocated from a single large buffer. On reload, you simply reset the allocator's pointer to the beginning of the buffer:

// Pseudo-code: pool allocator reset
config_memory.write_pointer = config_memory.base_address
config_memory.used_bytes = 0
config_memory.free_bytes = config_memory.total_size

This eliminates the risk of memory leaks during reconfiguration. No matter how many times you reload, memory usage stays constant.

Multi-File Configuration​

Production gateways often have multiple configuration files:

• Daemon config — Network settings, serial port parameters, batch sizes, timeouts
• Device configs — Per-device-type tag maps (one JSON file per machine model)
• Connection config — MQTT broker address, TLS certificates, authentication tokens

Each file should be watched independently. If only the daemon config changes (e.g., someone adjusts the batch timeout), you don't need to re-detect the PLC — just update the runtime parameter. If a device config changes (e.g., someone adds a new tag), you need to rebuild the tag chain.

A practical approach: when the daemon config changes, set a flag to force a status report on the next MQTT cycle. When a device config changes, trigger a full teardown-rebuild of that device's tag chain.

Pattern 2: Connection Watchdogs​

The most dangerous failure mode in MQTT-based telemetry is the silent disconnect. The TCP connection appears alive (no RST received), but the broker has stopped processing messages. The client's publish calls succeed (they're just writing to a local socket buffer), but data never reaches the cloud.

The MQTT Delivery Confirmation Watchdog​

The robust solution uses MQTT QoS 1 delivery confirmations as a heartbeat:

// Track the timestamp of the last confirmed delivery
last_delivery_timestamp = 0

on_publish_confirmed(packet_id):
    last_delivery_timestamp = now()

on_watchdog_check():  // runs every N seconds
    if last_delivery_timestamp == 0:
        return  // no data sent yet, nothing to check

    elapsed = now() - last_delivery_timestamp
    if elapsed > WATCHDOG_TIMEOUT:
        trigger_reconnect()

With MQTT QoS 1, the broker sends a PUBACK for every published message. If you haven't received a PUBACK in, say, 120 seconds, but you've been publishing data, something is wrong.

The key insight is that you're not watching the connection state — you're watching the delivery pipeline. A connection can appear healthy (no disconnect callback fired) while the delivery pipeline is stalled.

Reconnection Strategy: Async with Backoff​

When the watchdog triggers, the reconnection must be:

1. Asynchronous — Don't block the PLC polling loop. Data collection should continue even while MQTT is reconnecting. Collected data gets buffered locally.
2. Non-destructive — The MQTT loop thread must be stopped before destroying the client. Stopping the loop with force=true ensures no callbacks fire during teardown.
3. Complete — Disconnect, destroy the client, reinitialize the library, create a new client, set callbacks, start the loop, then connect. Half-measures (just calling reconnect) often leave stale state.

A dedicated reconnection thread works well:

reconnect_thread():
    while true:
        wait_for_signal()  // semaphore blocks until watchdog triggers

        log("Starting MQTT reconnection")
        stop_mqtt_loop(force=true)
        disconnect()
        destroy_client()
        cleanup_library()

        // Re-initialize from scratch
        init_library()
        create_client(device_id)
        set_credentials(username, password)
        set_tls(certificate_path)
        set_protocol(MQTT_3_1_1)
        set_callbacks(on_connect, on_disconnect, on_message, on_publish)
        start_loop()
        set_reconnect_delay(5, 5, no_exponential)
        connect_async(host, port, keepalive=60)

        signal_complete()  // release semaphore

Why a separate thread? The connect_async call can block for up to 60 seconds on DNS resolution or TCP handshake. If this runs on the main thread, PLC polling stops. Industrial processes don't wait for your network issues.

PLC Connection Watchdog​

MQTT isn't the only connection that needs watching. PLC connections — both EtherNet/IP and Modbus TCP — can also fail silently.

For Modbus TCP, the watchdog logic is simpler because each read returns an explicit error code:

on_modbus_read_error(error_code):
    if error_code in [ETIMEDOUT, ECONNRESET, ECONNREFUSED, EPIPE, EBADF]:
        close_modbus_connection()
        set_link_state(DOWN)
        // Will reconnect on next polling cycle

For EtherNet/IP via libraries like libplctag, a return code of -32 (connection failed) should trigger:

1. Setting the link state to DOWN
2. Destroying the tag handles
3. Attempting re-detection on the next cycle

A critical detail: track consecutive errors, not individual ones. A single timeout might be a transient hiccup. Three consecutive timeouts (error_count >= 3) indicate a real problem. Break the polling cycle early to avoid hammering a dead connection.

Link State Telemetry​

The gateway should treat the connection state itself as a telemetry point. When the PLC link goes up or down, immediately publish a link state tag — a boolean value with do_not_batch: true:

link_state_changed(device, new_state):
    publish_immediately(
        tag_id=LINK_STATE_TAG,
        value=new_state,  // true=up, false=down
        timestamp=now()
    )

This gives operators cloud-side visibility into gateway connectivity. A dashboard can show "Device offline since 2:47 AM" instead of just "no data" — which is ambiguous (was the device off, or was the gateway offline?).

Pattern 3: Store-and-Forward Buffering​

When MQTT is disconnected, you can't just drop data. A production gateway needs a paged ring buffer that accumulates data during disconnections and drains it when connectivity returns.

Paged Buffer Architecture​

The buffer divides a fixed-size memory region into pages of equal size:

Total buffer: 2 MB
Page size: ~4 KB (derived from max batch size)
Pages: ~500

Page states:
  FREE → Available for writing
  WORK → Currently being written to
  USED → Full, queued for delivery

The lifecycle:

1. Writing: Data is appended to the WORK page. When it's full, WORK moves to the USED queue, and a FREE page becomes the new WORK page.
2. Sending: When MQTT is connected, the first USED page is sent. On PUBACK confirmation, the page moves to FREE.
3. Overflow: If all pages are USED (buffer full, MQTT down for too long), the oldest USED page is recycled as the new WORK page. This loses the oldest data to preserve the newest — the right tradeoff for most industrial applications.

Thread safety is critical. The PLC polling thread writes to the buffer, the MQTT thread reads from it, and the PUBACK callback advances the read pointer. A mutex protects all buffer operations:

buffer_add_data(data, size):
    lock(mutex)
    append_to_work_page(data, size)
    if work_page_full():
        move_work_to_used()
    try_send_next()
    unlock(mutex)

on_puback(packet_id):
    lock(mutex)
    advance_read_pointer()
    if page_fully_delivered():
        move_page_to_free()
    try_send_next()
    unlock(mutex)

on_disconnect():
    lock(mutex)
    connected = false
    packet_in_flight = false  // reset delivery state
    unlock(mutex)

Sizing the Buffer​

Buffer sizing depends on your data rate and your maximum acceptable offline duration:

buffer_size = data_rate_bytes_per_second × max_offline_seconds

For a typical deployment:

• 50 tags × 4 bytes average × 1 read/second = 200 bytes/second
• With binary encoding overhead: ~300 bytes/second
• Maximum offline duration: 2 hours (7,200 seconds)
• Buffer needed: 300 × 7,200 = ~2.1 MB

A 2 MB buffer with 4 KB pages gives you ~500 pages — more than enough for 2 hours of offline operation.

The Minimum Three-Page Rule​

The buffer needs at minimum 3 pages to function:

1. One WORK page (currently being written to)
2. One USED page (queued for delivery)
3. One page in transition (being delivered, not yet confirmed)

If you can't fit 3 pages in the buffer, the page size is too large relative to the buffer. Validate this at initialization time and reject invalid configurations rather than failing at runtime.

Pattern 4: Periodic Forced Reads​

Even with change-detection enabled (the compare flag), a production gateway should periodically force-read all tags and transmit their values regardless of whether they changed. This serves several purposes:

1. Proof of life. Downstream systems can distinguish "the value hasn't changed" from "the gateway is dead."
2. State synchronization. If the cloud-side database lost data (a rare but real scenario), periodic full-state updates resynchronize it.
3. Clock drift correction. Over time, individual tag timers can drift. A periodic full reset realigns all tags.

A practical approach: reset all tags on the hour boundary. Check the system clock, and when the hour rolls over, clear all "previously read" flags. Every tag will be read and transmitted on its next polling cycle, regardless of change detection:

on_each_read_cycle():
    current_hour = localtime(now()).hour
    previous_hour = localtime(last_read_time).hour

    if current_hour != previous_hour:
        reset_all_tags()  // clear read-once flags
        log("Hourly forced read: all tags will be re-read")

This adds at most one extra transmission per tag per hour — a negligible bandwidth cost for significant reliability improvement.

Pattern 5: SAS Token and Certificate Expiry Monitoring​

If your MQTT connection uses time-limited credentials (like Azure IoT Hub SAS tokens or short-lived TLS certificates), the gateway must monitor expiry and refresh proactively.

For SAS tokens, extract the se (expiry) parameter from the connection string and compare it against the current system time:

on_config_load(sas_token):
    expiry_timestamp = extract_se_parameter(sas_token)

    if current_time > expiry_timestamp:
        log_warning("Token has expired!")
        // Still attempt connection — the broker will reject it,
        // but the error path will trigger a config reload
    else:
        time_remaining = expiry_timestamp - current_time
        log("Token valid for %d hours", time_remaining / 3600)

Don't silently fail. If the token is expired, log a prominent warning. The gateway should still attempt to connect (the broker rejection will be informative), but operations teams need visibility into credential lifecycle.

For TLS certificates, monitor both the certificate file's modification time (has a new cert been deployed?) and the certificate's validity period (is it about to expire?).

How machineCDN Implements These Patterns​

machineCDN's edge gateway — deployed on OpenWRT-based industrial routers in plastics manufacturing plants — implements all five patterns:

• Configuration hot-reload using stat() polling on the main loop, with pool-allocated memory for zero-leak teardown/rebuild cycles
• Dual watchdogs for MQTT delivery confirmation (120-second timeout) and PLC link state (3 consecutive errors trigger reconnection)
• Paged ring buffer with 2 MB capacity, supporting both JSON and binary encoding, with automatic overflow handling that preserves newest data
• Hourly forced reads that ensure complete state synchronization regardless of change detection
• SAS token monitoring with proactive expiry warnings

These patterns enable 99.9%+ data capture rates even in plants with intermittent cellular connectivity — because the gateway collects data continuously and back-fills when connectivity returns.

Implementation Checklist​

If you're building or evaluating an edge gateway for industrial IoT, verify that it supports:

Conclusion​

Reliability in industrial IoT isn't about preventing failures — it's about recovering from them automatically. Networks will drop. PLCs will reboot. Certificates will expire. The question is whether your edge gateway handles these events gracefully or silently loses data.

The patterns in this guide — hot-reload, watchdogs, store-and-forward, forced reads, and credential monitoring — are the difference between a gateway that works in the lab and one that works at 3 AM on a holiday weekend in a plant with spotty cellular coverage.

Build for the 3 AM scenario. Your operations team will thank you.

In industrial IoT, the MQTT connection between an edge gateway and the cloud isn't just another network link — it's the lifeline that carries every sensor reading, every alarm event, and every machine heartbeat from the factory floor to the platform where decisions get made. When that connection fails (and it will), the difference between losing data and delivering it reliably comes down to how well you've designed your resilience patterns.

This guide covers the engineering patterns that make MQTT connections production-hardened for industrial telemetry — the kind of patterns that emerge only after years of operating edge devices in factories with unreliable cellular connections, expired certificates, and firmware updates that reboot network interfaces at 2 AM.

The Industrial MQTT Reliability Challenge​

Enterprise MQTT (monitoring dashboards, chat apps, consumer IoT) can tolerate occasional message loss. Industrial MQTT cannot. Here's why:

• A single missed alarm could mean a $200,000 compressor failure goes undetected
• Regulatory compliance may require continuous data records with no gaps
• Production analytics (OEE, downtime tracking) become meaningless with data holes
• Edge gateways operate unattended for months or years — there's nobody to restart the process

The standard MQTT client libraries provide reconnection, but reconnection alone isn't resilience. True resilience means:

1. Data generated during disconnection is preserved
2. Reconnection happens without blocking data acquisition
3. Authentication tokens are refreshed before they expire
4. The system detects and recovers from "zombie connections" (TCP says connected, but no data flows)
5. All of this works on devices with 32MB of RAM running on cellular networks

Asynchronous Connection Architecture​

The first and most important pattern: never let MQTT connection attempts block your data acquisition loop.

The Problem with Synchronous Connect​

A synchronous mqtt_connect() call blocks until it either succeeds or times out. On a cellular network with DNS issues, this can take 30–60 seconds. During that time, your edge device isn't reading any PLCs, which means:

• Lost data points during the connection attempt
• Stale data in the PLC's scan buffer
• Potential PLC communication timeouts if you miss polling windows

The Async Pattern​

The production-proven pattern separates the connection lifecycle into its own thread:

Main Thread:                    Connection Thread:
┌─────────────┐                ┌──────────────────┐
│ Read PLCs   │                │ Wait for signal   │
│ Batch data  │──signal───────>│ Connect async     │
│ Buffer data │                │ Set callbacks      │
│ Continue... │<──callback─────│ Report status      │
└─────────────┘                └──────────────────┘

Key design decisions:

1. Use a semaphore pair to coordinate: one "job ready" semaphore and one "thread idle" semaphore. The main thread only signals a new connection attempt if the connection thread is idle (try-wait on the idle semaphore).

2. Connection thread is long-lived — it starts at boot and runs forever, waiting for connection signals. Don't create/destroy threads for each connection attempt; the overhead on embedded Linux systems is significant.

3. Never block the main thread waiting for connection. If the connection thread is busy with a previous attempt, skip and try again on the next cycle.

// Pseudocode for async connection pattern
void connection_thread() {
    while (true) {
        wait(job_semaphore);         // Block until signaled
        
        result = mqtt_connect_async(host, port, keepalive=60);
        if (result != SUCCESS) {
            log("Connection attempt failed: %d", result);
        }
        
        post(idle_semaphore);        // Signal that we're done
    }
}

void main_loop() {
    while (true) {
        read_plc_data();
        batch_and_buffer_data();
        
        if (!mqtt_connected && try_wait(idle_semaphore)) {
            // Connection thread is idle — kick off new attempt
            post(job_semaphore);
        }
    }
}

Reconnection Delay​

After a disconnection, don't immediately hammer the broker with reconnection attempts:

• Fixed delay: 5 seconds between attempts works well for most industrial scenarios
• Don't use exponential backoff for industrial MQTT — unlike consumer apps where millions of clients might storm a broker simultaneously, your edge gateway is one device connecting to one endpoint. A constant 5-second retry gets you reconnected faster than exponential backoff without creating meaningful load.
• Disable jitter — again, you're not protecting against thundering herd. Get connected as fast as reliably possible.

Page-Based Output Buffering​

The output buffer is where resilience lives. When MQTT is disconnected, data keeps flowing from PLCs. Without proper buffering, that data is lost.

Buffer Architecture​

The most robust pattern for embedded systems uses a page-based ring buffer:

┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│  Page 0  │ │  Page 1  │ │  Page 2  │ │  Page 3  │
│ [filled] │ │ [filling]│ │  [free]  │ │  [free]  │
│  sent ✓  │ │  ← write │ │          │ │          │
└──────────┘ └──────────┘ └──────────┘ └──────────┘
     ↑ read

Three page states:

• Free pages: Available for new data
• Work page: Currently being written to by the data acquisition loop
• Used pages: Filled with data, waiting to be sent

How it flows:

1. Data arrives from the batch layer → written to the current work page
2. When the work page is full → moved to the used pages queue
3. When MQTT is connected → first used page begins transmission
4. When MQTT confirms delivery (via PUBACK for QoS 1) → page moves back to free pool
5. When the connection drops → stop sending, but keep accepting data

The Critical Overflow Case​

What happens when all pages are full and new data arrives? You have two choices:

1. Drop new data (preserve old data) — generally wrong for industrial monitoring, where the most recent data is most valuable
2. Overwrite oldest data (preserve new data) — correct for most IIoT scenarios

The practical implementation: when no free pages are available, extract the oldest used page (which hasn't been sent yet), reuse it for new data, and log a buffer overflow warning. This means you lose the oldest unsent data, but you always have the most recent readings.

Page Size Tuning​

Page size creates a trade-off:

Practical recommendation: For industrial telemetry, 16–32KB pages work well. With a 500KB total buffer, that gives you 16–32 pages. At typical telemetry rates (1KB every 10 seconds), this provides 3–5 minutes of offline buffering — enough to ride through most network glitches.

Minimum page count: You need at least 3 pages for the system to function: one being written, one being sent, and one free for rotation. Validate this at initialization.

Thread Safety​

The buffer must be thread-safe because it's accessed from:

• The data acquisition thread (writes)
• The MQTT publish callback (marks pages as delivered)
• The connection/disconnection callbacks (enable/disable sending)

Use a single mutex protecting all buffer operations. Don't use multiple fine-grained locks — the complexity isn't worth it for the throughput levels of industrial telemetry (kilobytes per second, not gigabytes).

MQTT Delivery Pipeline: One Packet at a Time​

For QoS 1 delivery (the minimum for industrial data), the edge gateway must track delivery acknowledgments. The pattern that works in production:

Stop-and-Wait Protocol​

Rather than flooding the broker with multiple in-flight publishes, use a strict one-at-a-time delivery:

1. Send one message from the head of the buffer
2. Set a "packet sent" flag — no more sends until this clears
3. Wait for PUBACK via the publish callback
4. On PUBACK: Clear the flag, advance the read pointer, send the next message
5. On disconnect: Clear the flag (the retransmission will happen after reconnection)

// MQTT publish callback (called by network thread)
void on_publish(int packet_id) {
    lock(buffer_mutex);
    
    // Verify the acknowledged ID matches our sent packet
    if (current_page->read_pointer->message_id == packet_id) {
        // Advance read pointer past this message
        advance_read_pointer(current_page);
        
        // If page fully delivered, move to free pool
        if (read_pointer >= write_pointer) {
            move_page_to_free(current_page);
        }
        
        // Allow next send
        packet_in_flight = false;
        
        // Immediately try to send next message
        try_send_next();
    }
    
    unlock(buffer_mutex);
}

Why one at a time? Industrial edge devices have limited RAM. Maintaining a window of multiple in-flight messages requires tracking each one for retransmission. The throughput difference is negligible because industrial telemetry data rates are low (typically <100 messages per minute), and the round-trip to a cloud MQTT broker is 50–200ms. One-at-a-time gives you ~5–20 messages per second — more than enough.

Watchdog Patterns​

Reconnection handles obvious disconnections. Watchdogs handle the subtle ones.

The Zombie Connection Problem​

TCP connections can enter a state where:

• The local TCP stack believes the connection is active
• The remote broker has timed out and dropped the session
• No PINGREQ/PINGRESP is exchanged because the network path is black-holed (packets leave but never arrive)
• The MQTT library's internal keep-alive timer hasn't fired yet

During a zombie connection, your edge device is silently discarding data — it thinks it's publishing, but nothing reaches the broker.

MQTT Delivery Watchdog​

Monitor the time since the last successfully delivered packet (confirmed by PUBACK):

// Record delivery time on every PUBACK
void on_publish(int packet_id) {
    clock_gettime(CLOCK_MONOTONIC, &last_delivered_timestamp);
    // ... rest of delivery handling
}

// In your main loop (every 60 seconds)
void check_mqtt_watchdog() {
    if (!mqtt_connected)
        return;
    
    elapsed = now - last_delivered_timestamp;
    
    if (has_pending_data && elapsed > WATCHDOG_TIMEOUT) {
        log("MQTT watchdog: no delivery in %d seconds, forcing reconnect", elapsed);
        mqtt_disconnect();
        // Reconnection thread will handle the rest
    }
}

Watchdog timeout: Set this to 2–3× your keep-alive interval. If your MQTT keep-alive is 60 seconds, set the watchdog to 120–180 seconds. This gives the MQTT library's built-in keep-alive mechanism time to detect the problem first, with the watchdog as a safety net.

Upstream Token/Certificate Watchdog​

Cloud IoT platforms (Azure IoT Hub, AWS IoT Core, Google Cloud IoT) use time-limited authentication:

• Azure IoT Hub: Shared Access Signature (SAS) tokens with expiry timestamps
• AWS IoT Core: X.509 certificates with expiry dates
• Google Cloud IoT: JWT tokens (typically 1–24 hour lifetime)

When a token expires, the broker closes the connection. If your edge device doesn't handle this gracefully, it enters a reconnection loop that burns battery (for cellular devices) and creates connection storm load on the broker.

The pattern:

1. Parse the token expiry at startup — extract the se= (signature expiry) timestamp from SAS tokens
2. Log a warning when the token is approaching expiry (e.g., within 1 week)
3. Compare against system time — if the token is expired, log a critical alert but continue trying to connect (the token might be refreshable via a management API)
4. If the system clock is wrong (common on embedded devices without RTC), the token check will fail spuriously — log this case separately

// SAS token expiry check
time_t se_timestamp = parse_sas_expiry(token);
time_t now = time(NULL);

if (now > se_timestamp) {
    log(WARNING, "SAS token expired! Token valid until: %s", ctime(&se_timestamp));
    log(WARNING, "Current time: %s — ensure NTP is running", ctime(&now));
    // Continue anyway — reconnection will fail with auth error
} else {
    time_t remaining = se_timestamp - now;
    if (remaining < 604800) {  // Less than 1 week
        log(WARNING, "SAS token expires in %d days", remaining / 86400);
    }
}

System Uptime Reporting​

Include system and daemon uptime in your status messages. This helps diagnose issues remotely:

• System uptime tells you if the device rebooted (power outage, watchdog reset, kernel panic)
• Daemon uptime tells you if just the software restarted (crash, OOM kill, manual restart)
• Azure/MQTT uptime tells you how long the current connection has been active

When you see a pattern of short MQTT uptimes with long system uptimes, you know it's a connectivity or authentication issue, not a hardware problem.

Status Reporting Over MQTT​

Edge gateways should periodically publish their own health status, not just telemetry data. A well-designed status message includes:

{
  "cmd": "status",
  "ts": 1709391600,
  "version": {
    "sdk": "2.1.0",
    "firmware": "5.22",
    "revision": "a3f8c2d"
  },
  "system_uptime": 864000,
  "daemon_uptime": 72000,
  "sas_expiry": 1712070000,
  "plc": {
    "type": 1017,
    "link_state": 1,
    "config_version": "v3.2",
    "serial_number": 196612
  },
  "buffer": {
    "free_pages": 12,
    "used_pages": 3,
    "overflow_count": 0
  }
}

Publish status on two occasions:

1. Immediately after connecting — so the cloud knows the device is alive and what version it's running
2. Periodically (every 5–15 minutes) — for ongoing health monitoring

Extended status (including full tag listings and values) should only be sent on-demand (via cloud-to-device command) to avoid wasting bandwidth.

Protocol Version and QoS Selection​

MQTT Protocol Version​

Use MQTT 3.1.1 for industrial deployments in 2026. While MQTT 5.0 offers useful features (topic aliases, flow control, shared subscriptions), the library support on embedded Linux systems is less mature, and many cloud IoT brokers still have edge cases with v5 features.

MQTT 3.1.1 does everything an edge gateway needs:

• QoS 0/1/2
• Retained messages
• Last Will and Testament
• Keep-alive

QoS Level Selection​

Why not QoS 2? The exactly-once guarantee of QoS 2 requires a 4-step handshake (PUBLISH → PUBREC → PUBREL → PUBCOMP), doubling the round-trips. For industrial telemetry, occasional duplicates are easily handled by the cloud platform (deduplicate by timestamp + device serial), and the reduced latency of QoS 1 is worth it.

Why not QoS 0? Fire-and-forget has no delivery guarantee. For a consumer temperature sensor, losing one reading per hour is acceptable. For a $2M injection molding machine, losing the reading that showed the barrel temperature exceeded safe limits is not.

Cloud-to-Device Commands​

Resilient MQTT isn't just about outbound telemetry. Edge gateways need to receive commands from the cloud:

• Configuration updates — new tag definitions, changed polling intervals, updated batch sizes
• Force read — immediately read and transmit all tag values
• Status request — request a full status report including all tag values
• Link state — report whether each connected PLC is reachable

Subscribe on Connect​

Subscribe to the command topic immediately in the on-connect callback, before doing anything else:

void on_connect(status) {
    if (status == 0) {  // Connection successful
        mqtt_subscribe(command_topic, QoS=1);
        send_status(full=false);
        buffer_process_connect();  // Enable data transmission
    }
}

Topic structure for Azure IoT Hub:

Publish: devices/{device_id}/messages/events/
Subscribe: devices/{device_id}/messages/devicebound/#

The # wildcard on the subscribe topic captures all cloud-to-device messages regardless of their property bags.

TLS Configuration for Industrial MQTT​

Virtually all cloud MQTT brokers require TLS. The configuration is straightforward but has operational pitfalls:

Certificate Management​

• Store the CA certificate file on the device filesystem
• Monitor the file modification time — if the cert file is updated, reinitialize the MQTT client
• Don't embed certificates in firmware — they expire, and firmware updates in factories are expensive

Common TLS Failures​

Putting It All Together: The Resilient Edge Stack​

The complete architecture for a production-hardened IIoT edge gateway:

┌──────────────────────────────────────────────┐
│                   Cloud                       │
│   ┌──────────────────────────────────┐       │
│   │  MQTT Broker (Azure/AWS/GCP)     │       │
│   └──────────────┬───────────────────┘       │
└──────────────────┼───────────────────────────┘
                   │ TLS + QoS 1
┌──────────────────┼───────────────────────────┐
│  Edge Gateway    │                            │
│   ┌──────────────┴───────────────────┐       │
│   │  MQTT Client (async connect)      │       │
│   │  - Reconnect thread               │       │
│   │  - Delivery watchdog              │       │
│   │  - Token expiry monitor           │       │
│   └──────────────┬───────────────────┘       │
│   ┌──────────────┴───────────────────┐       │
│   │  Page-Based Output Buffer         │       │
│   │  - Ring buffer with overflow      │       │
│   │  - Thread-safe page management    │       │
│   │  - Stop-and-wait delivery         │       │
│   └──────────────┬───────────────────┘       │
│   ┌──────────────┴───────────────────┐       │
│   │  Data Batch Layer                 │       │
│   │  - JSON or binary encoding        │       │
│   │  - Size-based finalization        │       │
│   │  - Timeout-based finalization     │       │
│   └──────────────┬───────────────────┘       │
│   ┌──────────────┴───────────────────┐       │
│   │  PLC Communication Layer          │       │
│   │  - Modbus TCP / RTU              │       │
│   │  - EtherNet/IP                    │       │
│   │  - Link state tracking            │       │
│   └──────────────────────────────────┘       │
└──────────────────────────────────────────────┘

Platforms like machineCDN implement this complete stack, handling the complexity of reliable MQTT delivery so that plant engineers can focus on what matters: understanding their machine data, not debugging network connections.

Key Takeaways​

1. Never block PLC reads for MQTT connections — use asynchronous connection in a separate thread
2. Buffer everything — page-based ring buffers survive disconnections and minimize memory fragmentation
3. Deliver one message at a time with QoS 1 — simple, reliable, and sufficient for industrial data rates
4. Implement watchdogs — delivery watchdog for zombie connections, token expiry watchdog for authentication lifecycle
5. Report status — edge device health telemetry is as important as machine telemetry
6. Monitor file changes — detect certificate and configuration updates without restarting
7. Use MQTT 3.1.1 with QoS 1 — mature, well-supported, and sufficient for all industrial use cases
8. Design for unattended operation — the gateway must recover from any failure without human intervention

Building resilient MQTT connections isn't about handling the happy path — it's about handling every way the network, the broker, the certificates, and the device itself can fail, and ensuring that when everything comes back online, every data point makes it to the cloud.