Gateway or device: placing an upload failure

I lead the firmware for a cellular RFID gateway that an enterprise client runs at its customers' sites. Each unit is an ESP32 with an LTE modem and Wi-Fi, an Ethernet link to an RFID reader, and an Azure IoT Hub device twin that carries its configuration and status. Every 60 seconds it posts the tags it has read to the client's REST API over mutual TLS.

When an upload gets no answer, the firmware waits out a 20-second read, writes a status of -1 to the twin, and tries again on the next cycle. In September 2026 every unit on the client's QA channel showed -1. In October the production units showed it too. The older production firmware restarts after twelve failed uploads, so the production units that were online rebooted every four to six minutes.

A -1 says the reply never arrived. It says nothing about why. The cause could sit in the device (memory, the TLS stack, a bad body), in the network path (some guest Wi-Fi networks inspect TLS), in the credentials, or in the API itself, and each of those has a different owner.

Guess wrong and you pay for it with a firmware release that fixes nothing, or a site visit to a unit that was fine. I wanted evidence that put the fault on one side of the wire before anyone changed code.

Take the device out

From a laptop I replayed the exact request the firmware sends, with the same client certificate. The QA API completed the mutual-TLS handshake and then sent nothing for 30 seconds. On production, one well-formed request finished its handshake in 472 ms and then also got no reply in 30 seconds, while a request with no credentials drew a 401 in about a second. The route was up and checking credentials, and authenticated requests stalled behind that check. A laptop with no firmware on it failed the same way, which put the stall in the gateway.

Read the credentials

Once the QA gateway answered again, the same request with the current client certificate returned 201 in about 300 ms. An expired certificate drew a 403 and no certificate drew a 401. With those three responses mapped, I could read a unit's credential state from its status field alone.

The QA gateway recovered on its own in October 2026. The next boot of a QA unit uploaded with HTTP 201 on unchanged firmware.

Time the failure

One unit kept failing after the gateway recovered. Twelve uploads failed in under three minutes, about 15 seconds per cycle. A unit stuck behind a silent gateway spends the full 20-second read on every attempt. This one failed faster than that, so the fault sat on the device.

My first suspect was internal heap. The unit's largest free block was 32,756 bytes, against 63,476 bytes on a unit whose upload succeeded. I logged it as unproven, and the next piece of evidence replaced it.

Follow the bytes

The unit failed fast on every cycle, so I looked at what the device does between opening the connection and reading the reply. I read the TLS client in the Arduino-ESP32 core that the field firmware builds on. Its write() sends one TLS record, which holds at most 16,384 bytes, and the print path above it does not loop. A body over 16 KB went out cut short, and the API sat waiting for the rest.

A replay settled it. A 20,985-byte body cut at 16,384 bytes got nothing back in 60 seconds, and the full body got a 201 in half a second. Bodies that size are routine on a busy shelf: at 500 tags an uncompressed upload runs about 26 KB.

Test	Result	Points to
Laptop replay, same certificate, during the stall	Handshake completes, no reply in 30 s	gateway
Request with no credentials	401 in about 1 s	route is up
Current, expired and missing certificate	201, 403, 401	credentials readable
One unit's failure timing after recovery	About 15 s per failed cycle, under the 20 s read	device
Replay of a body cut at 16,384 bytes	No reply in 60 s; full body 201 in 0.5 s	device write path

• A write loop that sends the body in 4 KB chunks, handles a stalled or dropped socket, and feeds the watchdog between chunks. The firmware now counts any 2xx as accepted. Before, it matched a sentence in the 201 body.
• TLS buffers of 4 KB and up moved from internal RAM to PSRAM, which leaves the internal heap for everything else.
• The device self-test reports the TLS error code and labels allocation failures as low memory, so the twin names the next failure of this kind.

I tested the extracted write helper on the host first, where all 7 cases passed. Then I ported both changes to the development line, where all 18 native test suites passed.

On the fixed build the failing unit uploaded with a 201 every cycle, and its largest free internal block rose to 86 to 94 KB, up from 32,756 bytes.

The QA stall belonged to the client, and it cleared without a firmware change. The production stall was still open with the client when I wrote this in October 2026.

• Report status on a timer. On this firmware the status record rides on a successful upload, and its counters freeze at the last success. A unit whose uploads fail goes quiet in the twin at the moment you most need to hear from it.
• Reproduce payload-size failures on a bench emulator first. A local server that logs the bytes it receives shows a cut-short body in one run, and it puts large-body cases into the test plan before any build reaches a unit.
• Build the replay harness first. Replaying the firmware's exact request from a laptop answered in minutes what the twin could not answer at all.