How Industrial IoT Devices Recover From Failed OTA Updates

Industrial Linux devices deployed in factories, remote sites, and equipment cabinets face a critical challenge: how do you update software remotely without risking a bricked device that requires an expensive service call?

Unlike development boards that can be easily accessed and reflashed, industrial IoT hardware must survive failed updates, unexpected power loss, and network interruptions while maintaining continuous operation. A robust over-the-air (OTA) update system needs to do more than transfer files—it must install software safely, verify functionality, and recover automatically when something goes wrong.

Why it matters

The EU Cyber Resilience Act, which entered force in December 2024 with main obligations starting December 2027, establishes requirements for vulnerability handling and security maintenance throughout a product's lifecycle. While the CRA doesn't mandate specific OTA architectures, manufacturers need reliable update infrastructure to maintain connected products and respond to security vulnerabilities without costly field visits.

The A/B update foundation

The core principle behind resilient OTA updates is straightforward: never replace the running operating system in place. Instead, maintain two complete bootable system instances—commonly labeled A and B.

While system A runs, the update installs into system B. The device then reboots and attempts to start B. If B boots successfully and passes health checks, it becomes active. If it fails, the device automatically reverts to A. This approach eliminates the biggest risk of in-place updates: a power failure during installation that leaves the system unusable.

However, not all A/B implementations provide equal protection. Two partitions on a single disk offer software redundancy, but two physically independent storage devices add another layer of resilience.

Hardware-assisted redundancy

Sfera Labs' Strato Pi Max industrial platform, based on Raspberry Pi Compute Module, demonstrates how hardware design supports robust updates. The platform offers multiple storage configurations, including a dual-SD setup where two complete systems reside on independent SD cards. The onboard microcontroller controls storage routing and can switch between cards, meaning each A/B side lives on separate physical media.

This hardware redundancy pairs with Silitics' Rugix software stack, which provides reproducible Linux images, update management, and rollback control. The open-source Rugix example for Strato Pi Max includes a "stratopi-dual-sd" configuration that combines software A/B updates with physical storage redundancy.

The update and recovery process

When an update arrives, the system writes it to the inactive SD card while the active system continues running. A watchdog-based rollback mechanism arms before reboot. The system then restarts while switching cards. If the updated system boots and reports healthy operation, it remains active. If it fails repeatedly, the system automatically switches back to the previous SD card.

The watchdog component is critical here. A successful Linux boot doesn't necessarily mean the industrial application is operational. Production systems should configure the watchdog heartbeat to reflect actual application health, not just that the operating system started.

Security and traceability

Production update bundles should be cryptographically signed to establish a chain of trust between the release process and deployed devices. Additionally, generating a Software Bill of Materials (SBOM) for each release provides visibility into included components and dependencies. When vulnerabilities are discovered, SBOMs help identify which products and versions are affected.

The Rugix Bakery build system can generate both signed update bundles and SBOMs alongside system images, providing traceability throughout the product lifecycle.

Production considerations

Adapting this architecture for production systems requires replacing demonstration credentials with proper authentication, establishing production signing infrastructure, making the watchdog reflect application health, and integrating with fleet-management platforms. The approach doesn't require specific backend infrastructure—updates can be delivered through custom platforms or operator-controlled mechanisms.

These details were first reported by Automation World in an article by Giampiero Baggiani of Sfera Labs and Maximilian Köhl of Silitics.