One post tagged with "fault-tolerance"

In any industrial IIoT deployment, the connection between your edge gateway and the PLC is the most critical — and most fragile — link in the data pipeline. Ethernet cables get unplugged during maintenance. Serial lines pick up noise from VFDs. PLCs go into fault mode and stop responding. Network switches reboot.

If your edge software can't detect these failures, recover gracefully, and continue collecting data once the link comes back, you don't have a monitoring system — you have a monitoring hope.

This guide covers the real-world engineering patterns for building resilient PLC connections, drawn from years of deploying gateways on factory floors where "the network just works" is a fantasy.

Why Connection Resilience Isn't Optional​

Consider what happens when a Modbus TCP connection silently drops:

• No timeout configured? Your gateway hangs on a blocking read forever.
• No reconnection logic? You lose all telemetry until someone manually restarts the service.
• No link-state tracking? Your cloud dashboard shows stale data as if the machine is still running — potentially masking a safety-critical failure.

In a 2024 survey of manufacturing downtime causes, 17% of IIoT data gaps were attributed to gateway-to-PLC communication failures that weren't detected for hours. The machines were fine. The monitoring was blind.

The Link-State Model​

The foundation of connection resilience is treating the PLC connection as a state machine with explicit transitions:

┌──────────┐     connect()      ┌───────────┐
│           │ ─────────────────► │           │
│ DISCONNECTED │               │ CONNECTED   │
│  (state=0) │ ◄───────────────── │ (state=1)   │
│           │   error detected  │           │
└──────────┘                    └───────────┘

Every time the link state changes, the gateway should:

1. Log the transition with a precise timestamp
2. Deliver a special link-state tag upstream so the cloud platform knows the device is offline
3. Suppress stale data delivery — never send old values as if they're fresh
4. Trigger reconnection logic appropriate to the protocol

Link-State as a Virtual Tag​

One of the most powerful patterns is treating link state as a virtual tag with its own ID — distinct from any physical PLC tag. When the connection drops, the gateway immediately publishes:

{
  "tag_id": "0x8001",
  "type": "bool",
  "value": false,
  "timestamp": 1709395200
}

When it recovers:

{
  "tag_id": "0x8001",
  "type": "bool",
  "value": true,
  "timestamp": 1709395260
}

This gives the cloud platform (and downstream analytics) an unambiguous signal. Dashboards can show a "Link Down" banner. Alert rules can fire. Downtime calculations can account for monitoring gaps vs. actual machine downtime.

The link-state tag should be delivered outside the normal batch — immediately, with QoS 1 — so it arrives even if the regular telemetry buffer is full.

Protocol-Specific Failure Detection​

Modbus TCP​

Modbus TCP connections fail in predictable ways. The key errors that indicate a lost connection:

When any of these occur, the correct sequence is:

1. Call flush() to clear any pending data in the socket buffer
2. Close the Modbus context
3. Set the link state to disconnected
4. Deliver the link-state tag
5. Wait before reconnecting (back-off strategy)
6. Re-create the TCP context and reconnect

Critical detail: After a connection failure, you should flush the serial/TCP buffer before attempting reads. Stale bytes in the buffer will cause desynchronization — the gateway reads the response to a previous request and interprets it as the current one, producing garbage data.

# Pseudocode — Modbus TCP recovery sequence
on_read_error(errno):
    modbus_flush(context)
    modbus_close(context)
    link_state = DISCONNECTED
    deliver_link_state(0)
    
    # Don't reconnect immediately — the PLC might be rebooting
    sleep(5 seconds)
    
    result = modbus_connect(context, ip, port)
    if result == OK:
        link_state = CONNECTED
        deliver_link_state(1)
        force_read_all_tags()  # Re-read everything to establish baseline

Modbus RTU (Serial)​

Serial connections have additional failure modes that TCP doesn't:

• Baud rate mismatch after PLC firmware update
• Parity errors from electrical noise (especially near VFDs or welding equipment)
• Silence on the line — device powered off or address conflict

For Modbus RTU, timeout tuning is critical:

• Byte timeout: How long to wait between characters within a frame (typically 50ms)
• Response timeout: How long to wait for the complete response after sending a request (typically 400ms for serial, can go lower for TCP)

If the response timeout is too short, you'll get false disconnections on slow PLCs. Too long, and a genuine failure takes forever to detect. For most industrial environments:

Byte timeout: 50ms (adjust for baud rates below 9600)
Response timeout: 400ms for RTU, 2000ms for TCP

After any RTU failure, flush the serial buffer. Serial buffers accumulate noise bytes during disconnections, and these will corrupt the first valid response after reconnection.

EtherNet/IP (CIP)​

EtherNet/IP connections through the CIP protocol have a different failure signature. The libplctag library (commonly used for Allen-Bradley Micro800 and CompactLogix PLCs) returns specific error codes:

• Error -32: Gateway cannot reach the PLC. This is the most common failure — it means the TCP connection to the gateway succeeded, but the CIP path to the PLC is broken.
• Negative tag handle on create: The tag path is wrong, or the PLC program was downloaded with different tag names.

For EtherNet/IP, a smart approach is to count consecutive -32 errors and break the reading cycle after a threshold (typically 3 attempts):

# Stop hammering a dead connection
if consecutive_error_32_count >= MAX_ATTEMPTS:
    set_link_state(DISCONNECTED)
    break_reading_cycle()
    wait_and_retry()

This prevents the gateway from spending its entire polling cycle sending requests to a PLC that clearly isn't responding, which would delay reads from other devices on the same gateway.

Contiguous Read Failure Handling​

When reading multiple Modbus registers in a contiguous block, a single failure takes out the entire block. The gateway should:

1. Attempt up to 3 retries for the same register block before declaring failure
2. Report failure status per-tag — each tag in the block gets an error status, not just the block head
3. Only deliver error status on state change — if a tag was already in error, don't spam the cloud with repeated error messages

# Retry logic for contiguous Modbus reads
read_count = 3
do:
    result = modbus_read_registers(start_addr, count, buffer)
    read_count -= 1
while (result != count) AND (read_count > 0)

if result != count:
    # All retries failed — mark entire block as error
    for each tag in block:
        if tag.last_status != ERROR:
            deliver_error(tag)
            tag.last_status = ERROR

The Hourly Reset Pattern​

Here's a pattern that might seem counterintuitive: force-read all tags every hour, regardless of whether values changed.

Why? Because in long-running deployments, subtle drift accumulates:

• A tag value might change during a brief disconnection and the change is missed
• The PLC program might be updated with new initial values
• Clock drift between the gateway and cloud can create gaps in time-series data

The hourly reset works by comparing the current system hour to the hour of the last reading. When the hour changes, all tags have their "read once" flag reset, forcing a complete re-read:

current_hour = localtime(now).hour
previous_hour = localtime(last_reading_time).hour

if current_hour != previous_hour:
    reset_all_tags()  # Clear "readed_once" flag
    log("Force reading all tags — hourly reset")

This creates natural "checkpoints" in your time-series data. If you ever need to verify that the gateway was functioning correctly at a given time, you can look for these hourly full-read batches.

Buffered Delivery: Surviving MQTT Disconnections​

The PLC connection is only half the story. The other critical link is between the gateway and the cloud (typically over MQTT). When this link drops — cellular blackout, broker maintenance, DNS failure — you need to buffer data locally.

A well-designed telemetry buffer uses a page-based architecture:

┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐
│ Free   │ │ Work   │ │ Used   │ │ Used   │
│ Page   │ │ Page   │ │ Page 1 │ │ Page 2 │
│        │ │ (writing) │ │ (queued) │ │ (sending)│
└────────┘ └────────┘ └────────┘ └────────┘

• Work page: Currently being written to by the tag reader
• Used pages: Full pages queued for MQTT delivery
• Free pages: Delivered pages recycled for reuse
• Overflow: When free pages run out, the oldest used page is sacrificed (data loss, but the system keeps running)

Each page tracks the MQTT packet ID assigned by the broker. When the broker confirms delivery (PUBACK for QoS 1), the page is moved to the free list. If the connection drops mid-delivery, the packet_sent flag is cleared, and delivery resumes from the same position when the connection recovers.

Buffer sizing rule of thumb: At least 3 pages, each sized to hold 60 seconds of telemetry data. For a typical 50-tag device polling every second, that's roughly 4KB per page. A 64KB buffer gives you ~16 pages — enough to survive a 15-minute connectivity gap.

Practical Deployment Checklist​

Before deploying a gateway to the factory floor:

• Test cable disconnection: Unplug the Ethernet cable. Does the gateway detect it within 10 seconds? Does it reconnect automatically?
• Test PLC power cycle: Turn off the PLC. Does the gateway show "Link Down"? Turn it back on. Does data resume without manual intervention?
• Test MQTT broker outage: Kill the broker. Does local buffering engage? Restart the broker. Does buffered data arrive in order?
• Test serial noise (for RTU): Introduce a ground loop or VFD near the RS-485 cable. Does the gateway detect errors without crashing?
• Test hourly reset: Wait for the hour boundary. Do all tags get re-read?
• Monitor link-state transitions: Over 24 hours, how many disconnections occur? More than 2/hour indicates a cabling or electrical issue.

How machineCDN Handles This​

machineCDN's edge gateway software implements all of these patterns natively. The daemon tracks link state as a first-class virtual tag, buffers telemetry through MQTT disconnections using page-based memory management, and automatically recovers connections across Modbus TCP, Modbus RTU, and EtherNet/IP — with protocol-specific retry logic tuned from thousands of deployments in plastics manufacturing, auxiliary equipment, and temperature control systems.

When you connect a machine through machineCDN, the platform knows the difference between "the machine stopped" and "the gateway lost connection" — a distinction that most IIoT platforms can't make.

Conclusion​

Connection resilience isn't a feature you add later. It's an architectural decision that determines whether your IIoT deployment survives its first month on the factory floor. The core principles:

1. Track link state explicitly — as a deliverable tag, not just a log message
2. Handle each protocol's failure modes — Modbus TCP, RTU, and EtherNet/IP all fail differently
3. Buffer through MQTT outages — page-based buffers with delivery confirmation
4. Force-read periodically — hourly resets prevent drift and create verification checkpoints
5. Retry intelligently — back off after consecutive failures instead of hammering dead connections

Build these patterns into your gateway from day one, and your monitoring system will be as reliable as the machines it's watching.