I recently led a project where the requirement felt impossible at first: build a vendor‑agnostic, hot‑standby controller pair that can fail over in under 200 ms without buying any extra hardware modules or vendor‑specific redundancy kits. After several iterations and field tests, I arrived at a practical pattern that works with commodity Ethernet, standard protocols, and only the controllers and network gear you already have on the shop floor. Below I walk through the architecture, key techniques, and implementation details so you can reproduce it in your plant.

Why 200 ms matters and what “no extra hardware” means

Two reasons plants ask for <200 ms failover:

  • Process continuity: fast‑moving lines (e.g., high‑speed packaging, pick‑and‑place) cannot tolerate longer gaps without compensation.
  • Operator and safety interactions: short outages avoid nuisance trips and complicated state recovery.

    • By “no extra hardware” I mean: no vendor‑specific redundancy cards, no dedicated redundancy switches or black‑box failover appliances. You can use dual NICs, existing managed switches, and the controllers’ standard networking features.

    Core concept — synchronized dual controllers + stateless I/O arbitration

    The pattern that delivers sub‑200 ms failover is simple in concept:

  • Run two controllers (Primary and Secondary) in lock‑step at the application layer: both execute the control logic simultaneously, but only one is authoritative for outputs.
  • Keep both controllers tightly time‑synchronized (PTP/IEEE 1588) so they produce deterministically consistent commands.
  • Mirror commands from both controllers to I/O using a first‑arrival / sequence‑number arbitration at the I/O gateway. The I/O accepts the first valid command for a given cycle and rejects duplicates.
  • Use a very fast failure detection (sub‑100 ms) via multicast heartbeats and BFD‑style logic to switch the authority flag quickly.

    • This avoids hardware redundancy modules: outputs are arbitrated by simple logic that can run on a commodity industrial PC, an edge I/O gateway, or even on the I/O device if it supports custom logic or user code.

    Required building blocks (all standard, vendor‑agnostic)

    Dual NICs or network pathsManaged Ethernet switches (existing)PTP (IEEE 1588) time syncLightweight heartbeat (UDP multicast) + fast state machineI/O gateway with simple arbitration logic
    ItemRole
    Two controllers (same program)Run identical control logic; produce deterministic outputs and sequence metadata
    Optional but recommended for network path diversity
    VLANs, IGMP snooping, multicast prioritization
    Ensure cycle alignment and timestamp reliability
    Detect and declare controller health quickly
    Accept first valid command per cycle; enforce sequence/timestamp

    Detailed mechanics — messages, sequences, and arbitration

    Design your controllers to emit two outputs per control cycle:

  • Control packet: includes timestamp (PTP time), cycle number, sequence number, the output vector, and a signature/checksum. Use UDP multicast to minimize latency and avoid TCP head‑of‑line blocking.
  • Heartbeat packet: tiny UDP multicast frame at a high frequency (e.g., 50–100 Hz) that shows liveness and role (Primary/Secondary).

    • Arbitration rules implemented in the I/O gateway (or a soft I/O driver):

  • Maintain the latest accepted sequence number per output channel.
  • When a control packet arrives, validate timestamp/sequence and checksum.
  • If its sequence is greater than the stored value and the timestamp is within an acceptable window, accept and apply outputs immediately.
  • If duplicate sequence arrives from the other controller, discard it.
  • If heartbeats from Primary stop for X ms (configurable, e.g., 100 ms) and Secondary heartbeats persist, promote Secondary’s packets as authoritative.

    • This approach guarantees outputs switch over with the first valid packet from Secondary once Primary is declared dead. If packet transmission and processing are tuned, you can easily keep the observable gap under 200 ms.

    Time synchronization — why PTP is essential

    Without tight time sync, the two controllers’ sequence numbers and timestamps can drift, complicating arbitration and requiring larger safety windows. I recommend IEEE 1588 PTP v2 for sub‑microsecond alignment on modern industrial switches. Some PLC vendors bundle PTP clients; otherwise use a small NTP/PTP appliance on the network (this is not controller redundancy hardware — it's a synchronization service).

    Fast failure detection — heartbeat and BFD principles

    Heartbeat frequency and the switching algorithm determine your detection time. I typically implement:

  • Heartbeat at 50–200 Hz (5–20 ms between packets).
  • A failure declared after 4–8 missed heartbeats (20–160 ms), tuned for network jitter and packet loss environment.
  • Complementary BFD‑like checks between controllers if supported by network gear.

    • This combination lets the I/O gateway detect loss quickly and accept the secondary’s first control packet almost immediately after the failure window.

    Handling in‑flight commands and I/O safety

    In‑flight commands are addressed by the sequence/timestamp and the “first arrival wins” rule. But safety is crucial:

  • For critical actuators, implement a short output hold or ramp on state changes to avoid abrupt transitions during failover.
  • Use watchdog timers in both controllers and on I/O devices to force safe states if communications are ambiguous.
  • Log all accepted sequence transitions to a circular buffer for post‑event reconciliation and root‑cause analysis.

    • Implementation notes and vendor examples

    You can implement this pattern with many modern platforms:

  • On open soft‑PLC stacks (TwinCAT, Codesys, Ignition), run the same IEC 61131 program on two hosts and publish UDP multicast packets using built‑in UDP libraries.
  • On vendor PLCs (Siemens, Rockwell), leverage their Ethernet sockets or publish via OPC UA PubSub over UDP if available. OPC UA PubSub provides sequence numbers and timestamps out of the box.
  • Edge gateways (industrial PCs running Linux) can host the arbitration logic as a small C/Go/Node process; keep it lean to minimize latency.

    • Testing, KPIs and tuning

    Key metrics to measure:

  • Detection time — time between Primary failure and detection by gateway.
  • Application gap — time between last valid Primary output and first applied Secondary output.
  • Packet loss and jitter statistics on the multicast group.

    • Test plan essentials:

  • Repeatable fault injection (soft kill of Primary app, hard power off) and measure end‑to‑end gap with instrumentation.
  • Network load tests to ensure multicast packets maintain priority under congestion (use QoS on switches).
  • Failback behavior — ensure Primary can rejoin as Secondary without causing output thrash. Implement hysteresis and explicit role negotiation.

    • Common pitfalls

  • Relying on TCP or polling: TCP adds latency. UDP multicast + sequence numbers is far simpler and faster.
  • Insufficient PTP accuracy: causes confusing arbitration windows and forced longer timeouts.
  • Ignoring I/O device limits: some legacy remote I/O cannot safely accept duplicate commands; verify device behavior.
  • Lack of deterministic scheduling: ensure both controllers run the application with identical cycle times and priorities.

    • This pattern trades vendor‑supplied black boxes for a deterministic, standards‑based software approach. In my experience, it delivers sub‑200 ms failover on commodity networks when properly tuned. If you’d like, I can share a reference implementation (UDP packet format, simple arbitration pseudocode, and test harness) tailored to your controller family and network layout.