Shipping firmware fixes to a fleet with no rollback

The field units ran an older release line. The development line had moved ahead by 233 non-merge commits: watchdogs, recovery ladders, diagnostics and a redesigned status light.

Two facts shaped every decision about moving that work to the field. The field firmware had no OTA rollback, so a bad update meant a site visit. And the update trigger compared version labels as plain text, so the only remote recovery was to point a unit's twin back at the previous image.

I had to decide which changes go to the field, in what order, and how to prove each release before it touched a production unit. The client had to own the priorities.

Each OTA costs about 1.2 MB per unit on cellular and has no undo. Builds cost nothing, so commits, pull requests and tagged bench builds could be as small as I liked. Field steps had to be few, and each one safe.

A catalogue with stable IDs

I wrote every change into a catalogue of 150 entries in lettered groups. Each entry has an ID that the release plan, the test plan, the client deck and the release notes all share. The catalogue grew to 166 entries within two weeks.

An assessment rule that holds up

My first pass rejected 99 changes for the field line. A re-test against the field source found 41 of those rejections wrong. The first pass had asked whether a fault had been seen in the field. The better test asks whether the defect is provable in the source and inert without its trigger. I then attacked my own promotions twice and found defects in 9 of them.

The biggest saving needed no release

Payload compression was supported, but its only caller hard-coded it off. At 500 tags an upload is 26.4 KB raw and 4.9 KB compressed, which per device comes to 38 MB a day against 7 MB. That saving repays the data cost of the whole OTA campaign in about 15 hours. Turning it on needed one thing: the client confirming its ingest accepts compressed bodies.

Releases as a menu

Each release got a fixed kind (patch, minor, major, or joint work with the client) and stated dependencies. The client set the order, and I recommended patches first for small early wins. Version numbers came last, once the order was agreed.

Fourteen theme-sized slots then collapsed into four releases to cut the number of QA cycles. Anything that needed the client, such as a confirmation, a decision or backend work, went into its own final step, so no release waited on someone else's calendar.

Gates before production

1. CandidatesNumbered catalogue entries; the client sets scope and order
2. BenchTest plan cases, measurements inside their thresholds, a rollback rehearsal
3. QA unitsEvery test row passes in QA
4. Production canaryThe canary rows pass again
5. FleetRollout once the gate workbook is complete

With no rollback, every release had to pass a bench rollback rehearsal. A gate workbook decided promotion: every test row passes in QA, the canary rows pass again on the canary, and every bench measurement stays inside its threshold. The test plans used Given/When/Then cases, positive and negative.

The gate removed scope as well as adding it. A downgrade guard on the candidate refused the twin rollback to the previous release, the one remote recovery lever the test plan rehearses. I reverted it before QA and later dropped it from the plan.

QA rollout

The QA image went to four QA units. The production image and the release tag stayed untouched until the bench pass. The bench earned its place on the first day: a number typed into a twin field that the firmware expected as a string crash-looped the unit every 45 seconds. The fix reached the QA units the same day, and no production unit ever ran that build.

Rollout also exposed a deployment trap. A unit already running a label ignores a rebuilt image with the same label, so each QA rebuild meant bouncing units through the previous release and back.

Release notes that describe the release

I numbered the next release's candidates 1 to 30, later 1 to 35, and used the same numbers on the client deck and in the notes, so the client could record a decision against each number.

The notes follow one rule: they say what the release does. Each item states the wrong behavior, its cost and the change. Plan history, past decisions and open investigations go in the deck. The notes list only rows that are built or proven.

The field release passed its QA gate with 42 of 42 test cases in October 2026 and runs on the QA units. Production rollout was still pending at the time of writing, so I have no production result to report yet.

• Put a build number in every label. The same-label skip cost a bounce through the previous release on every QA rebuild. A unique label per build removes the trap.
• Fix rollback before features. The missing OTA rollback drove the rehearsal gate and the small number of field steps. A bootloader that can roll back makes every later release cheaper to ship.
• Write the notes rule down on day one. One draft of the notes carried plan and decision history and needed a rewrite.
• Target the canary by twin tag. IoT Hub automatic device configurations already target devices by twin tag and roll back by changing the condition. The missing piece is a tag scheme.