Building OTA Updates Taught Me Everything Except OTA
When I started building over-the-air (OTA) updates for my ESP32 fleet, I assumed the problem would be fairly straightforward. A device checks for a new firmware version, downloads the binary over HTTPS, verifies it, writes it to flash, and reboots. It sounded like a self-contained feature that would live entirely inside the firmware. I couldn’t have been more wrong. The firmware update itself turned out to be the easiest part. Everything around it testing, networking, security, provisioning, and developer experience became the real engineering challenge.
The first problem wasn’t security or reliability. It was testing. Every firmware change followed the exact same routine: build the binary, connect an ESP32 over USB, flash the board, wait for it to boot, trigger an OTA update, and watch the logs. There was nothing technically difficult about this workflow, but it was painfully repetitive. More importantly, every experiment depended on having a physical development board connected to my machine. Progress was limited by how quickly I could flash hardware and manually verify the results.
After enough iterations, I realized I had accidentally become my own CI pipeline. Every commit required the same manual steps, and every regression depended on me noticing it. That wasn’t a firmware problem it was a tooling problem. If OTA updates were going to be a core feature of the platform, they deserved the same level of automation as the backend services they interacted with.
That realization led me to build a native Linux Hardware Abstraction Layer for my ESP32 SDK. The goal wasn’t to emulate the ESP32 instruction set or simulate peripherals. Instead, I abstracted the hardware-specific interfaces so that the application logic could execute as a normal Linux process while preserving the same OTA workflow. Once that layer existed, GitHub Actions could spin up the backend, start MQTT and HTTPS services, run the SDK natively, perform an OTA update, and verify the result automatically on every commit. A process that once required plugging hardware into my desk became a single git push.
Ironically, automating the workflow uncovered problems I had never seen while testing manually. OTA updates began failing in seemingly random ways. Sometimes Mosquitto couldn’t communicate with the backend. Sometimes MinIO appeared unreachable. Other times the services looked perfectly healthy while devices quietly failed to download firmware. The failures didn’t follow a pattern, and every log suggested something different. I spent hours inspecting application code, convinced I had introduced some subtle bug into the OTA implementation.
The code wasn’t the problem. Firewall rules were silently blocking communication between services. Nothing crashed, nothing threw obvious errors, and every component believed it was functioning correctly. Packets simply weren’t reaching their destination. It was one of those bugs that forces you to stop thinking like an application developer and start thinking about the system as a whole. That experience taught me that distributed systems don’t really care how correct your code is if the network underneath isn’t behaving the way you expect.
Once the OTA pipeline became reliable, another question started bothering me. If I could remotely install firmware onto every device in a fleet, what prevented someone else from doing exactly the same thing? Until that point, I had been thinking about OTA as a mechanism for distributing software. Suddenly it became obvious that it was really a problem of trust. Downloading firmware isn’t difficult. Determining whether that firmware should be trusted is where the actual engineering begins.
That realization fundamentally changed the implementation. OTA evolved from a simple download-and-install feature into a layered verification pipeline. Devices pin the server’s TLS certificate to ensure they’re communicating with the correct endpoint. Every downloaded firmware image is validated using SHA-256 integrity checks to detect corruption. Ed25519 signatures prove that the firmware was produced by the expected signing key rather than an attacker. Key IDs make it possible to rotate signing keys in the future without breaking already deployed devices. Individually, none of these mechanisms are particularly complex. Together, they establish a chain of trust that makes remote firmware updates something I can actually rely on.
The next challenge was provisioning. Early prototypes shared credentials because it was convenient during development, but that approach falls apart the moment you move beyond a handful of boards. If every device shares the same identity, you can’t revoke a compromised device without affecting every other device. You can’t rotate credentials individually, trace telemetry back to a specific board, or confidently answer a simple question like, “Which device actually sent this data?” The solution was to stop treating devices as anonymous clients. Every device now receives its own UUID, API key, certificates, and configuration during provisioning, giving each board a unique identity throughout its lifetime.
Looking back, the interesting part of this project wasn’t OTA itself. Every solved problem revealed another layer underneath it. Solving testing led to CI/CD. CI/CD exposed networking issues that manual testing had hidden. Reliable networking raised security questions. Security naturally led to provisioning and device identity. Each improvement uncovered another assumption that worked during development but wouldn’t survive in production.
I started this project expecting to write firmware.
Instead, I ended up learning how much infrastructure exists around firmware. The code running on the ESP32 is only one piece of a production IoT system. Everything surrounding it, the automation, networking, trust model, provisioning pipeline, and developer experience is what ultimately determines whether the system is something you can confidently deploy and maintain.
The complete implementation is open source if you’d like to explore it: