5.2 The Data Path from Device to Cloud
5.2.1 Data Collection and Edge Protocol Conversion
An industrial site rarely grows according to one unified protocol — PLCs (Programmable Logic Controllers) speak Modbus RTU over serial links, high-end devices support OPC UA, temperature and humidity sensors reach the gateway over 4–20 mA signals, and photovoltaic inverters use proprietary SunSpec extension frames. When data from all of these must be gathered onto the same platform, the first obstacle is not bandwidth or compute but the protocol divide. The first duty of the data collection layer is not to "get the numbers up" but to "build a unified semantic outlet on top of protocol fragments".
Engineering Characteristics of Common Industrial Protocols
Modbus is one of the protocols that has long been in wide use on industrial sites. Its frame structure is minimal: address code + function code + data field + CRC (RTU mode), or MBAP header + function code + data field (TCP mode). The engineering benefit is that any MCU can implement a master or a slave in a small amount of code, and debugging tools are ready at hand. The price is the absence of security: Modbus has no authentication, encryption, or session management, and exposing it to the public internet is tantamount to handing over control of the device. In real projects, Modbus is usually used only inside closed wired networks, and it reaches the cloud only after security isolation by an edge gateway.
OPC UA (OPC Unified Architecture) is the opposite extreme. It defines a complete information model, security mechanisms (X.509 certificates + signing + encryption), and transport protocols (the binary UA Binary or HTTPS). Interoperability does not come from "everyone using the same frame structure" but from the address-space model — every data point's type, unit, metadata, and parent-child relationships are themselves described as metadata. The price is a heavier protocol stack: a typical implementation needs far more firmware space than a simple protocol, which is unfriendly to 8-bit MCUs. OPC UA therefore suits high-end devices (such as CNC machine tools and robot controllers) and heterogeneous-system integration that demands interoperability.
The most common engineering combination is: Modbus RTU/TCP at the field layer, with the Modbus → OPC UA or Modbus → MQTT conversion completed inside the edge gateway. The selection principle is plain: on the device side it is decided by hardware resources; on the platform side it is decided by the requirements for interoperability and security.
The Edge Gateway's Three Layers of Responsibility
An edge gateway is not a simple "data pass-through box"; it carries work at three levels:
Protocol conversion: convert fieldbuses and analog signals — Modbus, Profibus, CAN, 4–20 mA, digital I/O — into the IP protocols (MQTT, HTTP, OPC UA) needed for the cloud. Conversion is more than "re-wrapping"; it also involves data-type mapping, byte-order conversion, and unit scaling. For example, a raw 16-bit value in a Modbus register must be multiplied by a gain factor and converted into a floating-point number before being sent to the cloud platform.
Data preprocessing: the edge side does not simply pass raw values through. Typical operations include filtering (removing jump glitches), deadband compression (not sending when the change magnitude is below a threshold), aggregation (computing the mean/max within a fixed time window), and timestamp normalization (standardizing on UTC rather than device-local time). The value of preprocessing is less uplink bandwidth consumption and lower cloud storage and compute cost, while avoiding the "garbage in, garbage out" contamination of data.
Local caching and resumable uploads: unstable networks are the norm in the field. The edge gateway needs a small database or a ring buffer to hold data while the connection is interrupted and to re-upload it in time order once the connection recovers. Three common cache-strategy designs exist: full caching with FIFO eviction, compressed caching (storing only the residuals of an estimation model), and caching only critical alarms. The choice depends on the cache size and on how much data integrity the business demands.
Seen from a broader view, the industrial data field in 2025–2026 is seeing the rise of the Unified Namespace (UNS) — organizing device data in semantic namespaces (such as place/line/machine/sensor) and publishing it in real time in an event-driven manner, together with specifications like Sparkplug B, replacing the traditional chain of "collect, store, then query". UNS carries forward the same thread as the normalization approach of this section, pushing unified data from inside the platform out to a cross-system industrial data layer (the details of semantic interoperability are covered in Chapter 9).
A worked example: data collection from the combiner boxes of a photovoltaic plant produces large volumes of DC current, voltage, and temperature points every day. Without preprocessing, a single plant's annual data volume balloons quickly; after deadband compression and minute-level aggregation, the volume actually uploaded can be reduced substantially, while the information loss for generation-efficiency analysis stays controllable. The exact compression ratio depends on how frequently the equipment varies and on how much granularity the business tolerates; in engineering practice it is advisable to determine the deadband threshold by replaying one week of trial-run data.
Synchronization Strategies Between Edge Nodes and the Cloud
The synchronization strategy depends on latency tolerance and the required level of data consistency:
- Real-time synchronization: device-state data (binary switch values, fault flags) needs low-latency response and usually rides MQTT QoS 1/2 or the OPC UA publish/subscribe pattern. The edge gateway pushes the moment it detects a change, with no caching.
- Batch synchronization: periodically collected continuous data is packaged and uploaded over fixed time windows. The gateway maintains a local time-series database (such as SQLite or an edge edition of InfluxDB) and pushes uniformly at time-window boundaries. Batch synchronization reduces connection overhead but adds latency on the order of the window length.
- Event-driven synchronization: synchronization is initiated only when an alarm threshold is crossed, a device comes online or goes offline, or a firmware update completes; it is used to cut traffic during non-critical intervals.
In practice the three strategies are usually combined — real-time for state, batch for continuous values, event-driven for events. A heartbeat is also needed between the edge node and the cloud: the gateway periodically sends heartbeat packets carrying its own status (CPU, memory, cache water level), and the cloud uses them to judge whether the gateway is online and whether the reporting strategy needs adjusting.
Tool Example: A Modbus-to-MQTT Conversion Flow in Node-RED
Node-RED is one of the most common visual-programming platforms for edge gateways. The following is a textual description of a typical conversion flow:
- Modbus Read node: configure a Modbus TCP connection (IP:port placeholder
<gateway-ip>:502), function code 3 (read holding registers), starting address 0, and 2 registers to read (a 32-bit floating-point value). - Function node: receive
msg.payload(a Uint16Array), assemble it into an IEEE 754 floating-point number according to the byte order (big-endian or little-endian), multiply by the scaling factor (e.g. 0.1), and attach the device ID and a timestamp. - MQTT Publish node: configure the server address (e.g.
mqtt://<cloud-broker>:1883), the topicfactory/sensor1/temperature, QoS 1, and a JSON payload:{"deviceId":"PLC-01","ts":<unix-timestamp>,"value":25.6,"unit":"°C"}.
Engineering notes: watch for Modbus address offset (many documents number starting addresses from 1 while the actual protocol starts from 0); confirm the floating-point byte order with the device manufacturer; design MQTT topics with a hierarchical structure so the platform can route them. At the debugging stage, these details often cost more time than the protocol itself.
Practical Boundaries
Protocol conversion is not a cure-all. When the number of devices passes a certain scale and protocol fragmentation is extreme (Modbus, BACnet, Profibus, and CIP coexisting), a single gateway's CPU and memory become the bottleneck. Layered conversion is then required: lower-layer gateways bridge only the physical layer to IP protocols, while upper-layer aggregation gateways complete the semantic mapping. The other boundary is real-time behavior: if the field demands strictly deterministic latency (such as synchronized servo-motor control), you must bypass the gateway and use the fieldbus's isochronous communication (EtherCAT, Profinet IRT) directly. Data collection at the platform layer suits only non-real-time or soft-real-time management scenarios.
5.2.2 Message Queues: Data Buffering and Decoupling
At a parking-lot entrance in the early morning, cars line up in a long queue. The geomagnetic sensor deployed in each parking space fires a status message the instant a car pulls in or out. The backend data-processing module has barely finished computing the previous position update when the flood peak arrives — a burst of messages lands almost simultaneously, the database connection pool saturates in an instant, and the application server's memory climbs rapidly. Without an intermediate layer for buffering, the load would punch straight through the connection pool or burst the application server's memory.
This scene is not unique to parking lots. When the vibration sensors, temperature-humidity probes, and power meters of dozens of production lines report at the same time, even with a long interval per sensor, the aggregated throughput is enough to crash a single-machine program. The core problem the message queue solves is not "how fast messages are sent" but decoupling the rate of data production from the rate of consumption. Producers simply send at their own pace; consumers pull according to their own processing capacity; the broker in the middle acts as a reservoir, storing temporarily at flood peaks and releasing smoothly at troughs. Without a message queue, the data path is tightly coupled — a slow or failed link anywhere back-pressures upstream and causes cascading blockage; with a message queue, the producers' and consumers' lifecycles, processing speeds, and health states are all independent, and a jitter in one link does not spread to the whole system.
Buffering and Decoupling: Two Layers of Engineering Value
The buffering layer addresses the "bursts far above the average" character of IoT traffic. A device running steadily reports a few dozen readings per hour, but during a device restart, a firmware upgrade, or a production-takt changeover, a few minutes of data can equal a normal full day. Budgeting resources for peak capacity is unacceptably expensive. A message queue lets the backend plan resources around the average load: burst traffic waits in the queue while consumers keep pulling at their maximum processing capacity. Monitoring the queue's water level can serve as the trigger for elastic scaling — consumer instances scale out automatically as the level rises and scale back in as it falls, consuming on demand.
The decoupling layer solves the topological dependency of multi-consumer scenarios. Sensor data usually must be handed at the same time to a real-time alarm engine, a time-series database writer, and a visualization downsampling service. Without a message queue, the sensor must push data synchronously to all three modules — the producer must know every downstream address, protocol, and availability state. Whenever a consumer is added or taken offline, the producer code must change with it. With the Publish/Subscribe pattern, the sensor writes data to a single topic, and the alarm engine, database writer, and downsampling service each subscribe to that topic. Consumers can come and go at any time without sensing one another's existence.
Another easily overlooked value is uplink/downlink isolation. The uplink is devices reporting continuously and concurrently; the downlink is one-shot command delivery that expects a reply. When both share a single queue, the backlog from an uplink flood blocks the dispatch of downlink commands and makes control latency uncontrollable. Separate the uplink and downlink topics, configure different consumer groups and independent resource allocations for them, and even a fully saturated uplink queue will not affect the immediate dispatch of control commands.
Choosing a Communication Model: Point-to-Point vs. Publish/Subscribe
Message queues offer two infrastructure-level communication models, and the basis for choosing is the number of consumers a message has.
Point-to-Point serves "send once, consume once" scenarios. When the platform issues a "start the fan" command, only one device terminal needs to receive it. The logic is simple and the resource overhead low — a good fit for the downlink.
Publish/Subscribe serves multi-consumer scenarios. A temperature value reported by a sensor may at the same time be written to the time-series database, trigger an alarm rule, appear on a large display screen, and be archived to cold storage — each consumer processes it independently, with no dependency between them.
In practice the two are rarely used alone. A typical layered scheme: the uplink uses publish/subscribe, with different data types assigned to different topics (such as sensor-temp, sensor-vibration, device-status); the downlink uses point-to-point, with each command carrying a unique message ID and the device returning an execution confirmation after consuming it; asynchronous communication between the platform's internal components also goes point-to-point, ensuring that a critical event needs to be processed only once.
The Three Pillars of Reliability
Persistence: messages are written to disk at the same time they are written to memory. Kafka appends sequentially to log files and, together with the operating system's page cache, turns random disk writes into sequential writes, so single-node write throughput can reach a high level. In practice, configure the strategy per topic according to data importance: control commands persisted to all synchronous replicas (acks=all), telemetry persisted to the leader replica (acks=1), and debug logs optionally not persisted at all (acks=0). These settings are example values; production environments must tune them against data-integrity requirements and performance budgets.
Acknowledgment (ACK): MQTT's QoS model provides the reference basis — QoS 0 permits message loss, QoS 1 guarantees at-least-once delivery but may duplicate, and QoS 2 is strictly once. QoS 1 is enough for most device reporting, and duplicate messages are absorbed by the consumer's idempotent handling. The consumer returns an ACK after finishing a message; if it does not return in time, the queue redelivers.
Dead Letter Queue (DLQ): when a message still cannot be processed correctly after retries exceed the maximum, it is moved to a dedicated dead-letter topic. Operators read the dead-letter messages through an independent consumer, analyze the cause of failure, and decide whether to replay, repair, or discard. A common trap is a dead-letter queue without independent monitoring and alerting: dead-letter messages pile up silently and gradually drag down the main queue's delivery efficiency.
Kafka Partitions and Consumer Groups: Horizontal Scaling
As the device fleet grows to the tens of thousands, a single-node message queue is no longer dependable for throughput or availability. The architecture based on partitions and consumer groups is the scaling approach validated by industrial-grade practice today.
Kafka splits a topic into multiple partitions, and the partition is the basic unit of parallel processing and fault tolerance. Within a partition, messages keep their write order; across partitions, they are mutually independent. Producers assign partitions by device ID or timestamp, dispersing load naturally. Each partition can have multiple replicas; when the leader fails, a follower takes over automatically.
Consumer groups deliver horizontal consumption. Multiple consumers within a group consume one topic jointly, and each message is processed by only one consumer. When the number of consumers in the group matches the number of partitions, Kafka scales linearly; consumers beyond the partition count sit idle; with fewer consumers than partitions, one consumer handles several partitions at once. The partition count is usually planned with an upper bound early on — partitions can be added but not removed.
Kafka supports two subscription-isolation modes, broadcast and cluster: multiple consumer groups on the same topic each consume independently (the publish/subscribe pattern), while multiple consumers within the same group consume jointly (the point-to-point pattern). A common configuration for the IoT platform uplink is multiple consumer groups: one for real-time alarms (low latency), one for batch writes to the time-series database (high throughput), and one for offline analysis (latency tolerated), each group advancing its offsets independently.
Engineering Checklist
- Is room for partition growth reserved according to device scale? Too few partitions limit parallelism; too many add management overhead.
- Is a reasonable message-retention period (retention.ms) set for every topic? Expired data is deleted automatically, keeping the disk from filling up.
- Are dead-letter queues configured for critical topics, with backlog volume monitored independently?
- Do the consumers implement idempotent processing and manual offset commits?
- Are resource-limit parameters (such as max.in.flight.requests.per.connection and fetch.max.bytes) configured for producers and consumers?
- Are uplink and downlink topics separated, with an independent priority set for the downlink topics?
Buffering and Peak Shaving
Kafka Producer and Consumer Example (Python)
# producer.py — sample code; parameters are reference values, tune per production scenario
from kafka import KafkaProducer
import json
import random
import time
producer = KafkaProducer(
bootstrap_servers=['kafka-1:9092', 'kafka-2:9092'],
value_serializer=lambda v: json.dumps(v).encode('utf-8'),
acks=1, # example: acks=1 for telemetry, consider acks=all for control commands
retries=3, # example: number of retries
max_in_flight_requests_per_connection=5
)
device_id = "sensor_01"
while True:
data = {
"device_id": device_id,
"temperature": round(random.uniform(22.0, 30.0), 2),
"humidity": round(random.uniform(40.0, 70.0), 2),
"timestamp": time.time()
}
future = producer.send('sensor-data', key=device_id.encode(), value=data)
result = future.get(timeout=5)
print(f"Sent offset: {result.offset}")
time.sleep(10)# consumer.py — sample code using manual commit
from kafka import KafkaConsumer
import json
consumer = KafkaConsumer(
'sensor-data',
bootstrap_servers=['kafka-1:9092'],
group_id='data-cleaning-service',
enable_auto_commit=False, # manually commit offsets
value_deserializer=lambda m: json.loads(m.decode('utf-8')),
max_poll_records=100
)
for message in consumer:
data = message.value
print(f"Device: {data['device_id']}, Temp: {data['temperature']}, "
f"Humidity: {data['humidity']}, Time: {data['timestamp']}")
if data['temperature'] > 45.0:
print("ALERT: High temperature detected!")
consumer.commit() # commit after successful processingThe producer's acks=1 balances reliability against latency and suits most IoT uplinks; enable_auto_commit=False combined with an explicit consumer.commit() ensures that offsets are committed only after a message has been processed successfully, avoiding the data loss that follows when a failed consumption can no longer be retried. For high-integrity scenarios such as control commands, set acks=all.
With a message queue for buffering and decoupling, protocol-converted data can finally flow among the backend's many components without blocking one another. The time-series database takes up this link — responsible for structured data storage in the vertical domain, and for handling the writes and queries of massive numbers of timestamps and points in the IoT setting.
5.2.3 Common Data-Transmission Problems and Fault-Tolerance Mechanisms
A message queue can buffer and shave peaks, but it does not guarantee that data transmission is absolutely reliable. In real projects, the interaction path between devices and the cloud often has to cross unreliable wireless networks: a smart-parking geomagnetic sensor may lose packets to link congestion while uploading; the Wi-Fi that a factory's PLC collector depends on suffers signal attenuation from metal machinery; and at the instant a shared power-bank cabinet opens its door, the Bluetooth gateway may briefly drop the connection from electrical interference.
When network quality cannot guarantee "perfect delivery every time," the transmission path cannot avoid three engineering questions: What if a message is lost? What if a message is duplicated? How does a broken connection resume? MQTT provides a message-delivery framework through three QoS levels: QoS 0 is at most once; QoS 1 is at least once and may duplicate; QoS 2 uses PUBLISH → PUBREC → PUBREL → PUBCOMP to provide "exactly once" message delivery between the two endpoints of one MQTT session. It does not guarantee that a database write, business action, or physical-device operation executes exactly once end to end; those still require an idempotency key, state readback, and compensation.
The risk of duplicate delivery is best shown through a running scenario. A shared power-bank cabinet's door-opening command rides QoS 1: the server sends "open locker 3"; the gateway has already executed the unlock and is about to return the ACK when the network flickers and the ACK is lost; the server times out and retransmits, and the gateway receives the same command again. If the application layer does not defend itself, the cabinet's lock mechanism executes the unlock action twice — even though the second attempt cannot physically execute because of the mechanical limit, it still leaves an invalid log entry and wears the relay contacts.
The table below summarizes the main characteristics of the three levels, for weighing during selection:
| QoS level | Semantic guarantee | Typical communication steps | Example scenario | Engineering cost |
|---|---|---|---|---|
| QoS 0 | At most once | 1 step (publish and done) | High-frequency non-critical status reporting | No retransmission, no deduplication; reliability depends entirely on the link |
| QoS 1 | At least once | 2 steps (publish + acknowledgment, with timeout retransmission) | Command dispatch, alarm forwarding | The application layer must deduplicate idempotently; the broker must buffer unacknowledged messages |
| QoS 2 | MQTT message exactly once | 4 steps (publish + three-way handshake) | Messages that explicitly need protocol duplicates eliminated and whose endpoints have sufficient resources | Cannot replace business idempotency or safety control; broker and client must maintain a full state machine |
Selection conclusion: the stronger the reliability, the greater the resource overhead. Do not reflexively reach for QoS 2; use QoS 0 for stateless quantities; QoS 1 with application-layer idempotency covers the vast majority of scenarios.
With QoS as the transport contract, packet loss and duplication are supported at the infrastructure level. Another common problem, however, is reconnection after a disconnect. MQTT provides the persistent session mechanism for this (the CleanSession=false field in the connect packet). When a client connects with a persistent session, the broker keeps every message the client has not acknowledged (QoS 1 and QoS 2) plus the messages produced on the subscribed topics while the client was offline. When the device comes back online, the broker releases the stored messages all at once. This mechanism solves the problem of unacknowledged messages vanishing into thin air when a device drops off momentarily from a PLC restart or a communication-module glitch — the broker keeps them for you until you come back. Note that the persistent-session semantics differ between MQTT 3.1.1 and MQTT 5.0: 5.0 introduces the Session Expiry Interval, letting a client declare explicitly at connect time how long the session is retained, whereas in 3.1.1 the session lifetime depends on the broker implementation — when selecting, confirm the protocol version in use and the broker's behavior.
Idempotency design: a lesson no engineering project escapes. Even with the client and the broker cooperating at QoS 1, the application layer cannot dodge duplicate handling. Example: the cloud's barrier-gate management service issues a "raise the barrier" command carrying the globally unique ID cmd-1234. The controller finishes the action, but the ACK is lost on the way back, and the broker triggers a retransmission. The controller receives a second command with the same ID. If the business logic is "raise the barrier on command received", the second command — though physically unable to raise the barrier again — makes the system record a spurious log entry that confuses the operators' alarm judgment on "barrier-raising failure".
The standard remedy is idempotency design: before processing a business command, the receiver takes the globally unique ID from the message and checks it against a local cache (for example, the Redis SETNX command) or a database unique index to confirm whether the ID has already been processed. If it has, the message is discarded; if not, it is executed and the ID is recorded. QoS 1 then owns the network-layer semantic guarantee and the idempotency mechanism owns application-layer deduplication — each attends to its own duty.
For data reordering, QoS itself gives no guarantee — it promises only "definitely delivered" or "delivered only once", never the order of arrival. In practice, embed a monotonically increasing sequence number or timestamp in each message, and have the consuming end sort by sequence number, discard stale data, or merge. This topic is tightly connected to the ordering design of time-series data writes and is expanded in Section 5.4.
Engineering judgment for this section: do not count on the protocol alone to solve everything. When choosing a QoS level, ask first: would losing this message cost a life? If yes, choose QoS 2; if not, choose QoS 1 and do idempotency well in the application layer. But one boundary must be stated plainly: even QoS 2 is only "no loss, no duplication" at the message-semantics level; the final line of defense for personal safety is the deterministic interlock and shutdown logic on the edge side — a local signal trips the relay directly, without passing through any network or message queue, and cloud-side message semantics must not be counted on as the backstop. A network disconnection is not to be feared — just enable the persistent session. Reordering is handled by sorting on the in-message sequence number at the consumer; the concrete implementation is left to the database chapters.