Skip to content

6.1 IoT Development Languages and Communication Protocols

6.1.1 Python for Rapid IoT Prototyping

Example: you take over the technology selection for a smart greenhouse project — the sensor drivers are written in C, and the device-side protocol stack needs rapid validation. The key question is not which language is "better," but the core tension of the prototyping stage: the team must get the full chain — from sensor acquisition to cloud visualization — running within limited time, while at this stage the maintenance cost of operating across languages, debugging multiple development environments, and keeping different compiler toolchains alive often exceeds the benefit they bring.

Python has secured its footing in scenarios like this not because of syntactic sugar or community popularity, but because it naturally covers the three ends of an IoT project — device, gateway, and backend. With one language stack, a single developer supports the repeated iterations of the prototyping stage at low context-switching cost.

On the device side, the main control chip usually runs bare metal or an RTOS, and C dominates register operations and IO drivers. But runtime implementations such as MicroPython and CircuitPython let Python run on resource-constrained microcontrollers — practicable on common platforms such as the STM32 (ARM Cortex-M family) and the ESP32 (Xtensa or RISC-V architecture), though actual compatibility must be verified by testing. During prototyping, you can drive peripheral protocols such as GPIO, I2C, and SPI directly from Python to validate a sensor's timing logic quickly, and only after the data link is confirmed weigh whether to migrate the driver back to C or Rust. Even when the lower layer does not use MicroPython, Python often wraps hardware drivers into callable modules through C extensions, acting as glue at the system boundary.

On the gateway side, Python's asynchronous networking frameworks (asyncio, aiohttp) and its rich protocol client libraries let a developer build, with relatively little code, a gateway node that supports concurrent access from many devices. The gateway's job is to maintain the list of LAN sub-devices, handle multiple asynchronous connections, and reformat heterogeneous protocol data into a unified form before uploading it to the cloud — nearly every one of these responsibilities has an off-the-shelf library in the Python ecosystem, so there is no need to implement network buffering, protocol encoding/decoding, or other low-level logic from scratch.

On the backend side, web frameworks such as Flask, FastAPI, and Django can quickly build RESTful interfaces for device registration, data query, and alarm rules. During prototyping, one developer covers both the gateway and the backend with the same Python syntax, avoiding the introduction of another language's compiler chain and deployment process — the simplification this brings to the chain of decisions is often underestimated.

Implementing an MQTT Client

MQTT (Message Queuing Telemetry Transport) is a publish/subscribe protocol over TCP/IP, designed specifically for constrained devices and low-bandwidth networks. Through topics, it decouples publishers from subscribers in time: a publisher only sends messages to the broker and need not care which subscribers are listening. paho-mqtt is a widely used MQTT client library, maintained by the Eclipse Paho project, that provides a consistent API across many languages.

Below is Python code for a temperature-and-humidity sensor simulating data transmission (based on paho-mqtt 2.x, released in 2024; install with pip install "paho-mqtt>=2.0"):

python
import paho.mqtt.client as mqtt
import json
import time
import random

BROKER = "localhost"
PORT = 1883
TOPIC = "greenhouse/sensor/temperature"
CLIENT_ID = "sensor-01"

def on_connect(client, userdata, flags, reason_code, properties):
    if reason_code == 0:
        print("Connected successfully")
    else:
        print(f"Connection failed, reason code: {reason_code}")

client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=CLIENT_ID)
client.on_connect = on_connect
client.connect(BROKER, PORT, keepalive=60)
client.loop_start()

try:
    while True:
        payload = json.dumps({
            "device_id": CLIENT_ID,
            "timestamp": time.time(),
            "temperature": round(random.uniform(20.0, 30.0), 2),
            "humidity": round(random.uniform(60.0, 80.0), 2)
        })
        client.publish(TOPIC, payload, qos=1)
        time.sleep(5)
except KeyboardInterrupt:
    client.loop_stop()
    client.disconnect()

This code demonstrates the core operating pattern of an MQTT client: connect to the broker, construct a JSON payload in a loop, and publish messages at the specified QoS level. The example uses qos=1, which suits collected data with basic integrity requirements that can tolerate a few duplicates; devices with extremely constrained memory and bandwidth can drop to qos=0, saving the extra overhead of acknowledgment packets. One engineering detail worth noting is the keepalive=60 setting — it defines the heartbeat interval between client and broker. If the gateway is deployed on an unstable Wi-Fi network, this value can be shortened appropriately (to, say, 15 seconds) so that the broker notices a broken connection faster, preventing subscribers from continuing to receive stale state from that device. For the complete protocol mechanisms of QoS grading, session persistence, and the Will Message, see Section 9.2 of Chapter 9.

The trap beginners are most likely to step into here is the version trap: in version 2.0 (released in 2024), paho-mqtt reworked its callback API. The 1.x-era mqtt.Client(client_id=...) construction and the def on_connect(client, userdata, flags, rc) signature raise exceptions outright under 2.x — the constructor must explicitly declare CallbackAPIVersion.VERSION2, the callback signature becomes (client, userdata, flags, reason_code, properties), and the former integer return code is replaced by a reason_code object that carries its own name and semantics. A large share of online tutorials are still stuck at 1.x, and copying their code verbatim fails on the very first connection; whenever you pick up any MQTT example, first check the library's major version, then check the callback signature. The protocol itself has not changed — only the client library's interface contract has. Watching how the versions of your dependency libraries evolve when making technology choices is a mindset that runs through this whole chapter.

Serialization Choices: JSON versus Protocol Buffers

The example code uses JSON to carry its data. JSON is a human-readable text format with extremely low debugging cost — every message is directly readable, with no extra decoding tools required. But the redundancy of a text format becomes a bottleneck under constrained bandwidth or high message frequency. In the example, a greenhouse has a hundred-odd sensor nodes, each reporting every 5 seconds a JSON message containing device ID, timestamp, temperature, humidity, light, and CO₂ concentration, with a message body of roughly 150 bytes; a single node's uplink traffic is then about 108 KB per hour — roughly 78 MB per month per node (150 bytes × 720 messages/hour × 24 × 30) — and a system of a hundred-odd nodes generates about 8–25 GB of uplink data per month; storage replicas, retransmission after disconnects, and protocol-framing overhead will multiply the actual footprint several times over.

Protocol Buffers (Protobuf) is the alternative. You first define the message structure in a .proto file; compiling it generates classes that can read and write that structure. A Protobuf-serialized binary payload is markedly smaller than the JSON form of the same data, and serialization/deserialization is faster — but the exact reduction depends on the value ranges of the numbers and the lengths of the strings in the data schema, so no universal percentage can be given. The cost is that messages are no longer self-describing text — debugging requires decoding tools (such as protoc --decode), and the introduced compilation step adds complexity to the build pipeline.

A common engineering trade-off: JSON suits the prototyping stage and interfaces facing web frontends; Protobuf suits internal communication on the operational link between devices and the cloud. Some teams perform protocol conversion inside the edge gateway: when pushing to devices on the internal network, the gateway uses Protobuf to keep LAN traffic down; when reporting to the cloud, it converts to JSON to reduce parsing complexity on the cloud side. The concrete approach: define a unified device message structure in the .proto file; the gateway deserializes the binary data it receives, populates a unified internal model, and then decides the serialization format according to the reporting target.

The Risk Boundary of the Prototyping Stage

Python's efficiency advantage in the prototyping stage does not mean it suits every later stage. When the prototype evolves into a production system, three typical issues demand attention:

  1. Concurrency model: CPython's GIL limits parallel execution of CPU-intensive Python threads within one interpreter, but I/O-intensive asynchronous connections are not necessarily blocked by the GIL. A bottleneck may lie in protocol parsing, blocking callbacks, serialization, the network, or CPU. Profile first, then choose an event loop, multiple processes, native extensions, or another runtime.
  2. Type safety: the absence of runtime type checking raises maintenance cost in large multi-person projects. A common problem: a field reported by a device is a string during prototyping, gets converted to a float by the gateway in production, and the downstream consumer code still assumes a string — in Python, such a problem surfaces only at runtime.
  3. Dependency management: the loose structure of Python virtual environments and requirements.txt easily introduces hidden compatibility problems in continuous deployment. Deep dependency graphs and version conflicts among indirect dependencies can cause service startup failures in production, and the diagnostic path is longer than with a statically typed language.

A mature evolution strategy, therefore, is: use Python in the prototyping stage to get the full chain running, and reserve an interface abstraction layer at the system boundary (for example, abstract the device data reporting path into a Reporter interface — JsonReporter while testing in Python, a ProtobufReporter implemented later when migrating to Java). When data volume and concurrency requirements reach the threshold that justifies a rewrite, gradually migrate the core gateway service or data aggregation service to a statically typed language such as Java or Go. The key to this path is not "which language to pick as the final platform" but when to decide to switch to a static type system to manage complexity.

Table 6-1 Python versus Java/Go across the prototyping and production stages

DimensionPython (prototyping stage)Java / Go (production stage)
Per-message throughputEnough to support prototype validationHigher, suited to high-concurrency links
Development iteration cycle (same feature)Less code, changes take effect immediatelyCompile, package, restart — longer cycle
Runtime resource usageRelatively high (interpreted + garbage collection)Lower after optimization, can reach high resource efficiency
Cross-language integration costLow (glue nature, easy to call C libraries)Requires a bridging layer or RPC interface
Production-grade ecosystemRicher web/data-processing ecosystemMore complete enterprise frameworks, containerization, and observability support

The comparison in the table indicates typical magnitudes; actual differences depend on the specific implementation, degree of optimization, and business model.

Looking back at the smart greenhouse example, Python can, at least through the first few iteration cycles, get the full "sensor acquisition → gateway upload → cloud display" chain running, validating in a very short time whether the data format and alarm logic are sound. Once the flow runs end to end, you can then evaluate whether the gateway service needs a performance rewrite — leaving decision space for introducing a microservice architecture later.

In the next section, we look at how Java takes over the development of production-grade IoT applications.

6.1.2 Java in Enterprise IoT Development

Python fits prototypes, data processing, and many I/O-bound services, while Java has clear advantages in static typing, long-running services, and Spring ecosystem integration. As scale grows, device count alone cannot prove that Python must fail or Java must be faster. Load-test the target protocol, message size, concurrent connections, latency percentiles, and failure-recovery scenarios before choosing a language and process model.

An enterprise IoT backend must meet three core challenges: highly concurrent device access, stable service governance, and strict data consistency. Java has accumulated more than two decades of engineering experience in these areas — from JDBC to JPA, from Servlet to Spring Boot, from EJB to microservices, each layer of abstraction has lowered the barrier to building complex systems. The Spring Boot plus Spring Cloud stack has become the skeleton of many enterprise projects, and a typical IoT backend platform likewise builds its core services on this system.

Spring Boot: Standing Up an IoT Backend Service Quickly

The core idea of Spring Boot is "convention over configuration." You do not need to hand-configure complex XML; a single @SpringBootApplication annotation brings up a standalone service with embedded Tomcat. For an IoT backend, this means you can stand up an endpoint that receives device data within minutes.

Example: a smart-meter data collection service that must handle reporting requests from a large number of devices at once. Implementing it with Spring Boot takes roughly three steps. First, add the spring-boot-starter-web and spring-boot-starter-actuator dependencies in pom.xml. Second, create a @RestController exposing the POST endpoint /api/v1/device/data to receive meter readings in JSON format. Third, combine @EnableScheduling with @Scheduled to implement scheduled data aggregation, converting raw readings into minute-level statistics stored in the database.

This code is about 50 lines and involves no database configuration, no message queue, no distributed transactions — you can run it first to validate message format and throughput, then progressively introduce production-grade components such as MQTT, caching, and rate limiting. This is precisely Spring Boot's value: from prototype to production, it takes the incremental-enhancement route, not a teardown and rebuild.

Integrating the Eclipse Paho MQTT Client

Devices typically run on resource-constrained hardware and prefer the lightweight MQTT protocol for asynchronous communication rather than synchronous HTTP requests. The most commonly used MQTT client in the Java world is Eclipse Paho, which offers both blocking and non-blocking API modes. Below is a typical piece of Spring Boot configuration code.

java
// MqttConfig.java - Spring Boot MQTT configuration and callbacks (illustrative code)
import org.eclipse.paho.client.mqttv3.*;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class MqttConfig {

    @Bean
    public MqttClient mqttClient() throws MqttException {
        String brokerUrl = "tcp://your-mqtt-broker:1883"; // illustrative address, replace before deployment
        String clientId = "iot-backend-service-01";
        MqttClient client = new MqttClient(brokerUrl, clientId);

        MqttConnectOptions options = new MqttConnectOptions();
        options.setCleanSession(false);
        options.setAutomaticReconnect(true);
        options.setConnectionTimeout(10);
        options.setKeepAliveInterval(30);

        client.setCallback(new MqttCallback() {
            @Override
            public void connectionLost(Throwable cause) {
                // illustrative: log the event and raise an alarm; can integrate with Spring Actuator health checks
            }

            @Override
            public void messageArrived(String topic, MqttMessage message) {
                // illustrative: write reported point values to a message queue or store them directly to the database
                // Spring Cloud Stream can handle the asynchronous processing here
            }

            @Override
            public void deliveryComplete(IMqttDeliveryToken token) {
                // illustrative: confirm the command was delivered successfully
            }
        });

        client.connect(options);
        client.subscribe("/iot/device/+/data"); // wildcard + matches any device ID
        return client;
    }
}

This code configures an MQTT client with a non-clean session. cleanSession(false) means the broker retains offline messages for this client — no data is lost after a device disconnects and reconnects. automaticReconnect has the client automatically attempt reconnection when the connection drops, which in large-scale industrial deployments is practically standard.

When the Paho client receives point values such as temperature and humidity reported by devices, what happens in the messageArrived callback is far more complex than the example — it must unpack the raw payload into semantically meaningful point structures and handle timestamps, thread pools, backpressure, and connection health. IoT DC3 illustrates such a collection link: the Driver SDK publishes standardized point values to the internal messaging port, which Data then consumes. RabbitMQ is the default adapter, while brokers such as Kafka may also be selected as internal adapters. dc3-driver-kafka, by contrast, is a southbound data-source Driver; the two have different responsibilities.

RESTful API Design Guidelines

After device data enters the backend, a unified and extensible northbound interface is needed to serve frontends, mobile apps, and third-party systems. RESTful APIs are the most universal choice today. API design in IoT scenarios has a few special constraints:

  • Clear resource paths: center on the device, with path levels expressing ownership. For example, /api/v1/devices/{deviceId}/points/{pointId}/history denotes querying the history of a specific point under a specific device.
  • Pagination and time ranges: device data is inherently time-series in nature, so query interfaces must support startTime, endTime, page, and size parameters to avoid pulling oversized payloads in one go.
  • Versioning: embed the version number in the API path (/api/v1/) or implement it through the Accept-Version request header, to guarantee backward compatibility.
Figure 6-1 IoT REST API Endpoint Design (Illustrative)Under one version prefix, device write and history read paths split by resource semantics.Figure 6-1 IoT REST API Endpoint Design (Illustrative)Under one version prefix, device write and history read paths split by resource semantics.CallersUnified /api/v1 Resource EndpointsInternal ServicesDeviceReport / CommandFrontend UserQuery Devices & HistoryThird-Party SystemRules & AlarmsPOST /devices/{id}/dataWrite path: validate · dedupe · enqueuePOST /devices/{id}/command202 Accepted · Async DispatchGET /devices/{id}/points/{pid}/historystartTime · endTime · page · sizePOST /alarms/rules · GET /alarms/activeRule Creation & Active AlarmsAccess LayerAuth · Validate · Dedupe · QueueControl LayerCommand Queue & ReceiptsQuery LayerTime Window & PagingHistory Parameterized by Time WindowReal-Time Active AlarmsReport DataSend CommandHistory QueryRules / AlarmsWriteDispatchQueryRouteWrite path / commands (POST)Read path / queries (GET)Rules & AlarmsFigure 6-1 Report, command, and query endpoints share one versioned contract but enter the access, control, and query services separately.
Figure 6-1 IoT REST API Endpoint Design (Illustrative)

Figure 6-1 shows a common IoT backend endpoint layout — CRUD plus point-to-point commands. The key point: device data reporting uses POST, and control commands also use POST — the former is data processing, the latter is command delivery; the semantics differ, and so do the resource paths. The command endpoint /api/v1/devices/{id}/command usually responds asynchronously, returning 202 Accepted to indicate the command has been queued; it is subsequently pushed to the target device over the MQTT channel.

In the Java ecosystem, Spring Boot paired with Spring HATEOAS makes it convenient to build APIs that satisfy Level 3 of the REST maturity model — responses carry link information (for example, _links.self, _links.next) that helps clients discover subsequent operations automatically. In actual IoT projects, however, most teams stop at Level 2 (resources + HTTP verbs), because developers on the device side and in third-party systems are unfamiliar with hypermedia navigation, and keeping things simple proves more reliable.

Where Java Sits in the IoT Backend

Returning to the judgment at the start of this section: Python answers "does it work," Java answers "is it stable." From running the MQTT communication link in Python at the prototype stage, to building a horizontally scalable service cluster in Java + Spring Boot in production, this is a technical path many IoT teams have traveled. A typical reference project chooses Java as its primary language while retaining some flexibility in the protocol driver layer to support extension in other languages — precisely a confirmation of this two-language collaboration philosophy. In engineering practice, it is advisable to settle language boundaries at the very start of architecture design: the data acquisition chain can tolerate short-term fluctuation, so use Python to fail fast; the core business chain requires consistency and auditability, so use Java to hold the baseline.

6.1.3 IoT Communication Programming: Choosing Among MQTT, REST, and gRPC

The previous two sections showed the tool ecosystems Python and Java bring to protocol implementation, but what truly determines a system's communication efficiency is how well the protocol's characteristics match the scenario. An IoT platform often handles three very different kinds of communication at once: data reporting from the device side, northbound API exposure, and internal calls among backend microservices. These three scenarios differ enormously in their demands on latency, throughput, resource consumption, and development complexity — no single protocol covers them all. MQTT, REST, and gRPC are the three solution families with the widest coverage today; this section starts from protocol characteristics and, grounded in real architectures, gives a selection approach rather than a list of features.

MQTT: Built for the Device Side

MQTT has a clear design target — constrained devices and unreliable networks. It adopts the publish/subscribe model; its fixed-header overhead is minimal, only a few bytes, and it builds in mechanisms for coping with device disconnection, such as quality-of-service grading (QoS 0/1/2), persistent sessions, and the Will Message (the protocol mechanisms are detailed in Section 9.2 of Chapter 9). The publish/subscribe pattern inherently decouples producers and consumers: a sensor only pushes data to a topic, without caring who is subscribing.

This pattern matches large-scale device data distribution scenarios. Many cloud platforms make MQTT the first choice for device access, and the core reason is not "lightweight" but that it builds high-frequency needs — offline buffering, quality grading, topology decoupling — into the protocol layer. Between device and gateway, MQTT runs over a long-lived connection carrying heartbeats; the broker buffers offline data; QoS 1 ensures at-least-once delivery. This machinery solves the key problems of device-side communication reliability.

Engineering value: MQTT is advantageous at the edge when a system needs long-lived connections, publish/subscribe, persistent sessions, and broker routing. Whether it suits a battery-powered device still depends on network attachment, Keep Alive, wake cycles, and the carrier link. QoS 0 can serve high-frequency telemetry that tolerates loss; QoS 1 provides at-least-once delivery and requires business deduplication; QoS 2 eliminates duplicate delivery only within the protocol scope of one MQTT session. No QoS level replaces business idempotency across brokers, databases, and physical devices or local safety controls.

Boundary: MQTT is not a general-purpose data transfer protocol. Its broker is a potential single point when deployed as a single instance, so large-scale deployments need a clustering scheme (such as EMQX or NATS) to safeguard availability. MQTT does not fit synchronous control scenarios with extreme real-time requirements — the asynchronous publish/subscribe model cannot guarantee millisecond-level response.

REST: The Universal Choice for Northbound Interfaces

REST (Representational State Transfer) is built on HTTP, manipulating resource URIs with standard methods. Its engineering value lies not in performance but in universality and ecosystem — every language has a mature HTTP client, it is naturally firewall-friendly, and the OpenAPI specification has made automated interface documentation standard.

Engineering value: REST fits northbound API scenarios best. Device management, data query, and command delivery interfaces are exposed externally for web frontends, mobile apps, or third-party systems to call. One common misjudgment here is using REST for service-to-service calls: REST's HTTP header overhead and serialization/deserialization cost create unnecessary latency when microservices interact frequently. Another misjudgment is using REST for device-side data reporting — for constrained devices, the computational overhead and bandwidth consumed by JSON serialization/deserialization will drastically shorten battery life.

Boundary: REST fits request/response patterns and does not fit streaming push or event-driven scenarios. Long polling and SSE (Server-Sent Events) can serve as compensating options, at the cost of increased connection management and resource consumption.

gRPC: The Performance Choice for Service-to-Service Calls

gRPC is Google's open-source high-performance RPC framework, built on HTTP/2 and Protocol Buffers (Protobuf). Protobuf's binary encoding is markedly smaller than JSON and also parses faster. In a microservice architecture, gRPC suits synchronous service-to-service calls — when two backend services need to exchange structured data frequently and are latency-sensitive, gRPC's strongly typed interface definitions and streaming capability effectively reduce the production incidents caused by misaligned fields.

Unlike the other two, gRPC's value delivery has a precondition: the .proto contract comes first. Once the number of microservices passes a certain scale, the constraining force of strongly typed interfaces matters far more than the performance gain — the code-generation mechanism forces the server's and client's interface contracts to agree, which is more reliable than documentation-based maintenance; HTTP/2 multiplexing incidentally reduces the connection count, which is also friendlier to the gateway layer's load. Its costs are equally concentrated: TLS/mTLS is strongly recommended in production, though the protocol itself does not mandate it; clients depend on generated code, and firewalls may block HTTP/2 traffic; on constrained microcontrollers, the memory overhead of Protobuf libraries often exceeds the budget. These costs are absorbable inside a microservice team, but once they cross an organizational boundary — for example, exposing gRPC interfaces directly to the device side or to third parties — they become hard to bear. gRPC's niche is therefore firmly confined to the space between backend services: forward, it cannot reach the devices; outward, it cannot reach partners.

Performance Trade-offs and Where Each Protocol Belongs

The core differences among the three protocols in their applicable scenarios are shown in Table 6-2. The performance descriptions in the table are based on a comparison of protocol design specifications and common engineering practice; they point to no specific benchmark and serve only to aid selection judgment.

Table 6-2 Scenario characteristics of MQTT, REST, and gRPC compared

DimensionMQTTREST (HTTP/1.1)gRPC (HTTP/2)
Communication modelPublish/subscribe (asynchronous)Request/response (synchronous)Request/response, streaming (synchronous/asynchronous)
Protocol overheadVery low, small fixed headerFairly high, HTTP headers carry metadataLow, header compression + Protobuf serialization
QoS support3 built-in levelsNone, relies on application-layer retryNone, relies on application-layer retry
Device-side resource requirementsVery low, fits constrained MCUsLow, needs a basic HTTP stackFairly high, needs HTTP/2 + Protobuf libraries
Bandwidth adaptabilityExcellent, fits high-latency lossy networksModerate, header overhead is visible in low-bandwidth scenariosModerate, better than REST after header compression
Development complexityMedium, must manage topics and sessionsLow, standard HTTP, mature toolchainMedium-high, requires defining proto files
Typical scenariosSensor data reporting, command downlinkNorthbound APIs, third-party integrationInter-microservice RPC, streaming push

One simple judgment can be distilled from the table: MQTT holds a mature niche at the edge, REST holds the ecosystem advantage at open northbound interfaces, and gRPC achieves the highest efficiency in internal calls within the cloud backend.

A Layered Protocol Architecture

Figure 6-2 shows where the three protocols are deployed in a standard IoT platform. Each layer chooses the "best" protocol for its scenario, forming a multi-layer complementary structure.

Figure 6-2 Protocol Layering in an IoT PlatformMQTT serves southbound devices, gRPC internal calls, REST northbound APIs.Figure 6-2 Protocol Layering in an IoT PlatformMQTT serves southbound devices, gRPC internal calls, REST northbound APIs.Device LayerSensorsRuns MQTT ClientsPLCRuns MQTT ClientsActuatorsRuns MQTT ClientsGateway / Edge LayerMQTT BrokerOffline Cache · Pub/SubProtocol AdaptationModbus / OPC UA etc.Platform Service LayerDevice ManagementgRPC ServicesData StoragegRPC ServicesRule EnginegRPC ServicesInter-service: sync gRPC + async message queueNorthbound App LayerWeb FrontendRESTMobile AppRESTThird-Party SystemsRESTMQTT Pub/SubMQTT Continuous StreamREST Status RegistrationREST Northbound APIgRPC-Web AuxiliaryMQTT (device/edge)REST (northbound)Auxiliary / Optional PathPlatform services (internal gRPC)Figure 6-2 Protocols complement each other by layer; no single protocol is forced across devices, services, and external systems.
Figure 6-2 Protocol Layering in an IoT Platform

Key Points for Protocol Selection

  • Device data reporting: MQTT first. For battery-powered devices, unstable networks, and devices that can send only small amounts of data, MQTT is the soundest default choice. QoS 1 guarantees at-least-once delivery, and the broker can cache offline messages. Do not force REST or gRPC onto the device side — their resource consumption will drastically shorten battery life.
  • Northbound APIs: REST first. When interfaces need to be accessed by web frontends, mobile apps, or partner systems, REST's universality keeps integration cost lowest. Ecosystem tools such as OAuth 2.0, rate limiting, and OpenAPI documentation are far more mature than those for MQTT or gRPC.
  • Service-to-service calls: gRPC first. When two backend services need to transfer structured data frequently and are latency-sensitive, gRPC's Protobuf serialization plus HTTP/2 multiplexing can markedly raise throughput. When there are many microservices, strongly typed interfaces prevent incidents.
  • Event-driven: bring in a message queue. When data must be broadcast to multiple consumers, use MQTT's pub/sub mechanism or introduce RabbitMQ/Kafka. One scenario: a temperature sensor reports over MQTT to the broker; the data processing center consumes the MQTT message and calls the device registry service over gRPC to query metadata; the processed result is provided to a web dashboard through a REST API.
  • Real-time control and streaming data: for control commands requiring sub-second response, use gRPC bidirectional streaming between services; for video streams and the like, use WebRTC or a dedicated streaming protocol.

Engineering Risks and Trade-offs

Multi-protocol coexistence is not without cost. The gateway layer must run protocol adaptation modules that convert MQTT traffic into internal gRPC calls, adding a layer of processing latency and operational cost. The same data stream may be buffered twice — in MQTT and in the message queue — driving system complexity up.

One common engineering trap is forcing REST onto the device side for the sake of uniformity. Another is abusing REST inside the microservices, so that service-to-service call latency runs out of control and a rewrite to gRPC is eventually forced. In practice, you can adopt the approach of "a layered main line, with adapters converging": between device and gateway run only MQTT (or, for legacy devices, Modbus/OPC UA); from gateway to platform service layer, converge onto one internal bus (gRPC + message queue); and the platform exposes one unified REST API northbound. This main line covers most communication scenarios. What remains — real-time video streaming, file upload, firmware upgrade, and the like — each goes over its own dedicated protocol, with no forced unification.

This section built a decision framework for communication programming starting from protocol characteristics. The core conclusion: do not pursue a single one-size-fits-all protocol — pick the best option under the current constraints for each layer. At the same time, protocol choice feeds back into how service boundaries are drawn — whichever layer an access point lands on, the corresponding service responsibilities and deployment boundary should be drawn on that same layer; Section 6.2 makes this constraint concrete when it discusses service decomposition.

From Industrial Software to AI Agents · Building a multi-protocol, cloud-native, open-source industrial IoT platform ready to evolve toward AI agents