9.2 The MQTT Protocol in Depth
9.2.1 Core Mechanisms of MQTT
MQTT (Message Queuing Telemetry Transport) owes its standing among IoT protocols to two early design decisions: it changed the message-routing model from point-to-point to publish/subscribe, and it lifted the reliability guarantee from the transport layer up to the application layer. These two choices determined that it would later become one of the most widely used protocols for remote monitoring and device telemetry.
The Publish/Subscribe Model and Topic Wildcards
MQTT's message routing depends on a broker component. A publisher sends a message to the broker, and the broker looks up all matching subscribers by the topic the message carries and forwards it. Publishers and subscribers are fully decoupled in time, space, and traffic: they need not know each other's IP addresses, need not be online at the same time, and their traffic rhythms are independent of each other.
Topics use the slash / as a hierarchical separator, forming a layered path similar to a file system. A temperature sensor can publish data to sensor/temperature/room1. If subscribers could only filter messages by exact match, then once device counts passed ten thousand, the configuration overhead of enumerating every topic one by one would overwhelm the operations side.
MQTT defines two wildcards to reduce this management cost:
- The single-level wildcard
+: matches any value within one level. A subscription tosensor/+/room1receivessensor/temperature/room1andsensor/humidity/room1, but does not matchsensor/temperature/room1/sub. - The multi-level wildcard
#: matches all remaining trailing levels, and can only appear at the end of a topic. A subscription tosensor/#receivessensor/temperature/room1,sensor/humidity, and every other message whose topic starts withsensor/.
These two wildcards let the subscription granularity be as coarse or as fine as needed: when connecting to an entire workshop, the platform subscribes to factory/floor1/#; when connecting to a single PLC, it subscribes to factory/floor1/PLC01/temperature. The application layer no longer needs to poll repeatedly — the decision-making moves into the broker's topic-tree matching engine.
QoS Levels: An Engineering Choice in Three Reliability Steps
MQTT defines three Quality of Service (QoS) levels, which increase the cost of reliability progressively, from fire-and-forget to four-way handshake confirmation.
- QoS 0 (at most once): after sending, no acknowledgment is awaited, nothing is stored, nothing is retransmitted. Messages may be lost. Suitable scenarios: high-frequency sensor reporting — losing a sample or two does not affect trend judgment; telemetry streams on intranets with extremely large data volumes.
- QoS 1 (at least once): after sending, the publisher waits for a PUBACK acknowledgment and retransmits if it does not arrive before the timeout. The message is guaranteed to arrive at least once, but subscribers may receive duplicate copies. Suitable scenarios: most control commands — the safety risk of executing a command twice is absorbed by idempotency at the application layer; device state-change notifications.
- QoS 2 (exactly once): a four-step handshake (PUBLISH→PUBREC→PUBREL→PUBCOMP) ensures that a message is delivered only once within the protocol-delivery scope of one MQTT session. The cost is that both the client and the broker must maintain packet state. It can be used for messages that genuinely need protocol-level duplicate delivery eliminated, but it cannot replace business transactions, device-side idempotency, or safety control loops.
Selection must consider loss tolerance, duplicate tolerance, disconnect-and-reconnect semantics, and business idempotency together. Most projects combine QoS 1 with business keys, state machines, and deduplication tables, using QoS 2 only when its protocol-delivery guarantee is genuinely necessary. At every QoS level, "business exactly once" across brokers, databases, and physical devices must be guaranteed separately by the application protocol. Personal-safety functions such as emergency stops and interlocks belong in local safety systems and must not rely on MQTT QoS as their sole safeguard.
Retained Messages and Will Messages
MQTT anticipated a thorny problem in IoT scenarios: devices leave the network without saying goodbye.
Retained messages let a publisher set RETAIN=1 on a message. The broker caches the last retained message for that topic and pushes it immediately whenever a new subscriber connects. A newly powered device, or a platform that has just restarted, can thus obtain the current state without waiting for the next data report. A concrete usage: a gateway periodically reports device/gateway01/status with retain set, and the platform receives the "online" status the moment it comes online.
Will messages are registered at connection time through WILL_TOPIC and WILL_MESSAGE. When the broker detects that the connection has broken abnormally (heartbeat timeout, half-open TCP connection), it broadcasts to that will topic on the client's behalf. Other subscribers that receive the message know the device may have lost power or lost network connectivity, and can trigger alarm or service-migration logic accordingly.
These two mechanisms fill in the blind spot of the publish/subscribe model regarding device-state awareness. Under the traditional HTTP model, a server cannot proactively learn whether a client is alive; MQTT achieves passive detection through the broker's session and heartbeat mechanisms, at the cost of requiring the broker to maintain connection state and will information.
The following code demonstrates common operations based on the paho-mqtt 2.x library (the callback API was restructured in 2.0; its constructor and signatures are incompatible with 1.x — see the version notes in Chapter 6, Section 6.1).
import paho.mqtt.client as mqtt
import time
def on_connect(client, userdata, flags, reason_code, properties):
if reason_code == 0:
# Subscribe to topics after a successful connection
client.subscribe("sensor/temperature/#", qos=1)
def on_message(client, userdata, msg):
print(f"topic: {msg.topic}, payload: {msg.payload.decode()}, qos: {msg.qos}")
client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
client.on_connect = on_connect
client.on_message = on_message
# Register a will message: the broker publishes it on the client's behalf when the connection drops
client.will_set("device/status", "offline", qos=1, retain=False)
client.connect("localhost", 1883, keepalive=60)
client.loop_start()
# Publish a retained message
client.publish("sensor/temperature/room1", '{"t": 25}', qos=1, retain=True)
time.sleep(2)
client.publish("sensor/temperature/room2", '{"t": 23}', qos=0)
client.loop_stop()
client.disconnect()This code covers three basic operations: subscribing, will setup, and publishing. A production environment must additionally handle: the reconnect callback (on_disconnect), configuration of the session-cleanup flag (clean_session), and the release logic for QoS 2 packet identifiers. These session-management-level issues tend to be the first weak points to surface once the device count scales up, and each should be exercised and verified in load testing before rollout.
The core engineering takeaway of this subsection: the publish/subscribe model, topic wildcards, and the three QoS levels form a scenario-oriented, trade-off-capable messaging system. Retained messages and will messages are design additions aimed at the IoT field's "unreliable devices with hard-to-predict states." In practice, the broker's topic-tree matching performance and session-state management are the real bottlenecks of large-scale deployment.
9.2.2 MQTT Sessions and Keep-Alive
The publish/subscribe model solves message routing, but communication reliability ultimately rests on connection management. Are subscriptions preserved after a device loses the network? How does the broker distinguish "briefly offline" from "gone for good"? In engineering, the answers to these two questions determine system resource cost, message reliability, and reconnection-recovery capability. MQTT manages the connection lifecycle with two mechanisms, the session and keep alive; only when they work well together can tens of thousands of devices maintain business continuity over unreliable networks.
Session State: Clean Session and Session Expiry
An MQTT client and a broker maintain a session between them, recording the client's subscription list, unacknowledged QoS 1/2 messages, and the Will message. Whether the session is persisted is decided at connection time by the Clean Session flag (MQTT v3.1.1) or the Session Expiry Interval (MQTT v5.0); MQTT 5.0 replaced 3.1.1's Clean Session flag with Session Expiry Interval, where Session Expiry Interval = 0 corresponds to a one-off session and any value greater than 0 to a persistent session. These two parameters split the scenarios into two typical strategies:
Clean Session = true (Session Expiry Interval = 0 in v5.0): every connection is a brand-new session, and the broker keeps no previous subscriptions or offline messages. Once the connection breaks, all state is destroyed immediately. This is the choice for pure uplink scenarios — for example, a sensor that periodically uploads temperature: after a disconnect, reconnecting does not need to restore historical subscriptions; establishing a new session is enough. The cost is that the platform cannot deliver precisely in downlink scenarios, because messages sent while the device is offline are simply lost.
Clean Session = false (Session Expiry Interval > 0 in v5.0): the broker persists the session state. After the client disconnects, the broker keeps its subscriptions and undelivered messages and restores them automatically when the client reconnects with the same Client ID. This is essential in downlink control scenarios: if the device happens to be offline when the platform issues a command, the broker buffers the message and pushes it in one batch once the device comes back online. The cost is that the broker's memory footprint grows linearly with the number of devices.
The Session Expiry Interval added in MQTT v5.0 allows setting the session's survival time in seconds, offering finer granularity than v3.1.1's "keep forever or not at all." In engineering there is no need to agonize over an exact value; you only need to confirm three boundaries: the upper limit of device reconnection frequency, the memory the platform can bear, and the business's tolerance for historical messages. A common practice is to set a reasonable extension based on the device's typical offline duration, rather than directly using 0xFFFFFFFF for never-expiring — the latter gradually consumes broker memory across large device fleets, and a reconnection storm in extreme cases can overwhelm the broker.
Keep Alive: The Heartbeat That Decides Life or Death
A long-lived connection needs a mechanism for both sides to confirm that "the other party is still there." With the Keep Alive mechanism, the client declares a time interval (in seconds) when the connection is established, defining the maximum time between two consecutive messages (including PINGREQ). Under the Keep Alive rules of MQTT 3.1.1 and 5.0, if the broker receives no MQTT control packet within 1.5 times that interval, it must disconnect the client's network connection and trigger the will message as configured.
The Keep Alive value depends on the business scenario and power constraints. Battery-powered devices usually use a longer Keep Alive interval to reduce heartbeat frequency; scenarios that need fast offline detection use a shorter one. MQTT v5.0 allows the server to reject the client's declared Keep Alive value and return the server-required Keep Alive — particularly useful in industrial settings, where the operations team flattens the heartbeat frequency of tens of thousands of devices through a unified broker-side threshold, preventing a few long-heartbeat devices from slowing fault discovery. When selecting a value, you must also consider the carrier network's connection keep-alive policy: some mobile-network base stations may actively release connections after a certain period without data, so the client's heartbeat interval must be smaller than that value.
Disconnection and Automatic Reconnection Strategies
Network instability is the norm in the Internet of Things. The MQTT protocol itself does not define a reconnection strategy; that is the client implementation's responsibility. Common strategies include:
- Fixed-interval reconnection: simple to implement but inflexible. When the network cannot recover for a long time, the fixed interval keeps wasting power, and when large numbers of devices drop out simultaneously it can trigger a broker avalanche.
- Exponential-backoff reconnection: wait a short interval at first and double it after each failure, up to a maximum. It balances brief dropouts against long outages, though the initial delay may leave an individual device offline slightly longer.
- Exponential backoff with random jitter: adds a random offset, avoiding large numbers of devices reconnecting at once and avalanching the broker — the "good enough" choice for most IoT projects.
Most MQTT client libraries (such as Eclipse Paho) have built-in automatic reconnection options. Engineering experience shows that exponential backoff combined with random jitter strikes a reasonable balance among implementation complexity, power control, and coordination at scale. Only the rare scenarios that require millisecond-level recovery, such as real-time production-line control, consider a fixed interval or even a pre-established backup connection.
Beyond reconnection strategy, the transport layer has one more route worth the attention of weak-network scenarios: MQTT over QUIC. Brokers such as EMQX 5 and NanoMQ already offer commercial support — QUIC is based on UDP, so on reconnection the session can be restored with 0-RTT; connection migration lets a device switch from Wi-Fi to cellular without the connection breaking as the IP changes; and streaming transport eliminates TCP's head-of-line blocking. For connected-vehicle terminals, mobile inspection devices, and other scenarios that switch networks frequently, it is becoming the pragmatic option besides TLS over TCP.
The Cooperation Boundary Between Heartbeat and Session
One boundary often overlooked in engineering deserves emphasis here: a heartbeat timeout does not necessarily destroy the session. The timeout verdict only triggers the broker to cut the TCP connection and execute the will message (if any); whether session state is retained depends on Clean Session or the Session Expiry Interval. In other words, even if the broker rules the client offline, the device can still recover as long as the session has not expired.
This boundary is a source of confusion on some broker implementations. A frequent misconception is "heartbeat timeout = session deletion." In reality, a heartbeat timeout is responsible only for connection-level state cleanup, while session expiry is what handles application-level state cleanup. When configuring operations alarms, engineers need to distinguish two kinds of timeout: the offline alarm triggered by a heartbeat timeout, and the session-destruction alarm triggered by session expiry. The former is routine operations — devices drop out and reconnect quickly; the latter is the real anomaly — the device may be gone for good.
There is no universally winning "best value." Selection principle: high-density sensor reporting (uplink only) uses a short session expiry with a long heartbeat; controllable devices (needing downlink) use a long session expiry with a short heartbeat, combined with will messages for fast offline detection.
Key MQTT 5 Features: Subscriber Scaling and Fault Localization
While refining session management (Session Expiry Interval), MQTT 5.0 also brought a set of features directly related to scaling and troubleshooting, which are worth enabling first in engineering.
Shared subscriptions are the standard answer to horizontal scaling on the subscribing side. Add the $share/{group}/ prefix when subscribing (for example $share/monitor-g1/home/+/temperature), and subscribers in the same group no longer each receive the full message stream — the broker spreads messages within the group automatically, delivering each message to only one member of the group. When the platform's subscription service needs to scale from a single instance to many, there is no need to build partitioning logic yourself: adding or removing subscribers completes the scale-out, and load balancing is the broker's job.
Reason codes turn "cannot connect, cannot subscribe, was disconnected" from guesswork into reading the packet. The v3.1.1 CONNACK returned only an integer return code; MQTT 5 carries named reasons in CONNACK, SUBACK, DISCONNECT, and other packets — for example, 0x87 Not authorized points to a permission configuration error, and 0x9E Shared Subscriptions not supported points to a broker version too old. The time to localize large-scale reconnection failures is thereby greatly shortened.
Topic Alias targets constrained bandwidth: the topic string is carried only in the first PUBLISH and registered as an alias; subsequent packets transmit only a two-byte alias value. For links with deep topic hierarchies, small per-packet payloads, and NB-IoT traffic billing, this overhead saving is considerable.
Enhanced authentication supports challenge–response extended authentication through the AUTH packet, allowing integration with external authentication systems such as Kerberos and OAuth beyond TLS, so that device access authentication aligns with the platform-side identity system — see Chapter 8, Section 8.2 for how this connects with device identity.
Sessions and heartbeats form the foundation of MQTT connection reliability. But keeping the connection alive is only the starting point — the reliability parameter that actually carries business requirements is the QoS level, which the next section will expand on.
9.2.3 MQTT in Practice: Smart-Home Monitoring
The previous subsection took apart sessions and heartbeats; now we put the two mechanisms to the test in a worked example. Using a purpose-built smart-home monitoring scenario, we combine publish/subscribe, QoS levels, and will messages to see how they cooperate in actual engineering.
Case: multi-point temperature and humidity monitoring in a residence. Sensors are deployed in several rooms, reach the internet through a home gateway, and report data to a cloud platform at fixed intervals. The platform receives and stores the data and pushes an alarm to the user's phone when humidity exceeds a preset threshold. The system must also detect and update device state within one heartbeat cycle after an abnormal disconnect (for example, a sensor suddenly losing power).
This scenario covers the three typical MQTT message flows: periodic reporting, alarm push, and state awareness.
Step 1: Devices Publish Sensor Data
Each sensor is an MQTT client that connects to the broker and publishes data to topics at fixed intervals. The scenario uses QoS 1, guaranteeing the data reaches the broker at least once — it will not be lost to momentary packet drops the way QoS 0 allows, nor generate the extra acknowledgment round trips of QoS 2.
# Illustrative code, not production-grade, only for demonstrating the core MQTT flow (based on paho-mqtt 2.x)
import paho.mqtt.client as mqtt
import json
import time
import random
DEVICE_ID = "sensor_living_room_01"
BROKER = "mqtt.homecloud.com"
PORT = 1883
TOPIC_TEMP = f"home/{DEVICE_ID}/temperature"
TOPIC_HUMI = f"home/{DEVICE_ID}/humidity"
TOPIC_WILL = "home/devices/status"
def on_connect(client, userdata, flags, reason_code, properties):
print(f"Device {DEVICE_ID} connected successfully, reason_code: {reason_code}")
client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=DEVICE_ID, protocol=mqtt.MQTTv311)
client.will_set(
topic=TOPIC_WILL,
payload=json.dumps({"device": DEVICE_ID, "status": "offline"}),
qos=1,
retain=True
)
client.on_connect = on_connect
client.connect(BROKER, PORT, keepalive=60)
client.loop_start()
try:
while True:
temperature = round(random.uniform(20.0, 30.0), 1)
humidity = round(random.uniform(40.0, 80.0), 1)
client.publish(TOPIC_TEMP, json.dumps({
"value": temperature, "unit": "C", "timestamp": time.time()
}), qos=1)
client.publish(TOPIC_HUMI, json.dumps({
"value": humidity, "unit": "%", "timestamp": time.time()
}), qos=1)
print(f"[{DEVICE_ID}] Published Temp={temperature}C, Humi={humidity}%")
time.sleep(30)
except KeyboardInterrupt:
pass
finally:
client.loop_stop()
client.disconnect()The key engineering choices in this code: set a will message when connecting to the broker, covering the abnormal-disconnect scenario; publish temperature and humidity data at fixed intervals; include a timestamp with each publication so the subscribing side can judge data freshness without depending on the broker's clock. retain=True makes the broker keep the last will message, so a new subscriber obtains the device's latest state as soon as it connects.
Step 2: The Cloud Subscribes and Stores
The cloud platform runs a subscriber program that uses the + wildcard to subscribe to every sensor's data topics and the status topic.
# Illustrative code, not production-grade, only for demonstrating MQTT subscription and alarm triggering (based on paho-mqtt 2.x)
import paho.mqtt.client as mqtt
import json
BROKER = "mqtt.homecloud.com"
PORT = 1883
TOPIC_TEMP_ALL = "home/+/temperature"
TOPIC_HUMI_ALL = "home/+/humidity"
TOPIC_STATUS_ALL = "home/devices/status"
device_status = {}
def on_connect(client, userdata, flags, reason_code, properties):
print(f"Platform subscriber connected successfully, reason_code: {reason_code}")
client.subscribe([(TOPIC_TEMP_ALL, 1), (TOPIC_HUMI_ALL, 1), (TOPIC_STATUS_ALL, 1)])
def on_message(client, userdata, msg):
topic = msg.topic
payload = json.loads(msg.payload.decode())
if topic.endswith("/temperature"):
print(f"[Storage] Temperature data: {payload}")
elif topic.endswith("/humidity"):
# Alarm rule triggered
if payload.get("value", 0) > 75:
sensor_id = topic.split("/")[1]
client.publish(f"home/alarm/{sensor_id}", json.dumps({
"type": "humidity_high",
"device": sensor_id,
"value": payload["value"],
"threshold": 75,
"timestamp": payload["timestamp"],
# Idempotency key: QoS 1 may deliver duplicates; the subscriber deduplicates on this key
"dedup_key": f"{sensor_id}-humidity-high-{int(payload['timestamp'])}"
}), qos=1)
print(f"[Alarm] {sensor_id} humidity reading exceeds the preset threshold!")
elif topic == "home/devices/status":
device_status[payload["device"]] = payload["status"]
print(f"[Status] Device {payload['device']} status: {payload['status']}")
client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id="cloud_monitor")
client.on_connect = on_connect
client.on_message = on_message
client.connect(BROKER, PORT, keepalive=60)
client.loop_forever()The key points of the code: subscribing to all sensors' temperature and humidity topics with the + wildcard means the platform need not know the sensors' specific IDs; when humidity exceeds the preset threshold, a QoS 1 message is pushed to the alarm topic carrying a unique alarm key (dedup_key) in the payload, and the subscriber deduplicates on that key — alarms must not be lost, and duplicate deliveries must not turn into duplicate notifications; this is exactly the conclusion of Section 9.2.1: application-layer idempotency is usually more intuitive and easier to debug than protocol-layer exactly-once. Will messages are processed to update device state in real time.
Step 3: Will Messages and Disconnect Detection
Suppose sensor_living_room_01 suddenly loses power and its TCP connection breaks. Once the broker senses the heartbeat timeout (triggered by the keepalive=60 setting), it immediately publishes the preset will message {"device": "sensor_living_room_01", "status": "offline"}. On receiving this will, the platform marks the corresponding device offline in device_status. Note that the will is published only when the broker detects an abnormal disconnect; a normal client disconnect does not trigger it. will_set together with keepalive=60 forms a "heartbeat + will" death-detection combination — a direct engineering embodiment of the timers discussed in Section 9.2.2.
Engineering Risks and Trade-off Analysis
Risk one: high-frequency publishing and broker throughput bottlenecks. Suppose the number of sensors is large and each publishes at a fixed interval; the broker's throughput pressure depends on the total number of sensors and the publishing frequency. A single-node broker can usually cope at small scale, but once the device count grows to thousands or more, cluster deployment or message sharding must be considered. Scaling has two ends to consider: on the access side, partition on the first level of home/{device_id}, using consistent hashing to spread different devices across different broker nodes; on the subscription side, use MQTT 5 shared subscriptions (see Section 9.2.2) — multiple platform subscriber instances join the same $share group, the broker spreads messages within the group automatically, and scaling is simplified from rewriting client partitioning logic to adding or removing subscriber instances.
Risk two: will-message backlog. During a widespread network outage, the broker publishes wills for a large number of devices in a short time. If the subscriber cannot keep up, will messages pile up in the queue. Solutions: add backpressure on the subscribing side to limit concurrent processing, and use batch operations for database writes.
Risk three: client ID conflicts. When multiple devices connect to the broker with the same client_id, all but the first are kicked offline. In engineering practice, assign unique IDs at the factory, or use a hash of the device's hardware identifier as the client_id.
Table 9-2 Message configuration for the smart-home monitoring scenario
| Message type | Recommended QoS | retain | Engineering notes |
|---|---|---|---|
| Periodic sensor data | 1 | false | Occasional duplicates allowed, but no loss |
| Alarm push | 1 + idempotency-key deduplication | false | Must not be lost; duplicate deliveries are deduplicated by the alarm's unique key — QoS 2's state-maintenance and round-trip cost is worth paying only when alarms must not repeat and the link has no idempotency layer |
| Will status | 1 | true | New subscribers get device state immediately |
This case shows the complete MQTT workflow in a lightweight IoT scenario: devices publish data periodically over long-lived connections, the platform receives everything uniformly through wildcard subscriptions, alarms achieve no loss and no duplication through QoS 1 plus idempotency-key deduplication, and device dropouts are sensed promptly through will messages. There is no complex rebalancing, sharding, or transaction machinery — this is exactly MQTT's original intent: under constrained bandwidth and compute, do what must be done reliably.