When MQTT payloads quietly drift

An HVAC alumni fleet in Greater Bangkok shipped a gateway patch on a Thursday. The patch wrote valve_open as the strings "true" and "false" instead of JSON booleans. The ingestion job coerced unknown types to null. The ops tile still showed “98% valves reporting” because it counted messages, not parseable fields. Product thought usage had collapsed. Firmware thought nothing had changed. Three weeks of stand-ups were spent on the wrong mystery.

Drift is usually a type, a name, or a unit

We group MQTT surprises into three drawers. Type: boolean to string, int to float, missing to zero. Name: valveOpen appearing beside valve_open after a contractor weekend. Unit: Celsius in one SKU, tenths of a degree in another, both plotted on the same axis. IoT App Analytics dies in the third drawer more often than people admit.

The Monday check

Before the weekly product meeting, pull a 200-message sample from each active topic. Diff field names and inferred types against last week’s contract snapshot — a JSON file in git is enough; you do not need a schema registry on day one. If a field changed type or vanished above a tiny threshold, the dashboard is frozen until firmware or ingestion owns the break.

This sounds bureaucratic. It takes twelve minutes when the contract exists. It takes three weeks when it does not. Device Signal Studio spends most of module five on that snapshot, which several students have said moves too quickly. They are not wrong; we still refuse to skip it.

What we do not do

We do not recommend a particular broker. We do not claim protobuf will save you if nobody versions the message. We do not treat “optional” as a synonym for “we will decide later.” Optional fields that product later treats as required are how HVAC charts go quietly false.

If your topics already frighten you, a private instrumentation audit is usually cheaper than another month of green tiles.