6.3 IoT DC3 Engineering Practice

Author: Zhang Hongyuan · © 2016–2026 · All Rights Reserved · Copyright & License

6.3.1 IoT DC3 Project Architecture Overview: Module Division and Core Components

Chapter 2 gave the layered blueprint of the IoT platform; this section grounds it in compilable, deployable modules using IoT DC3. When reading this architecture, the most important thing is to distinguish three boundaries: how northbound requests enter the center services, how Drivers synchronize metadata with the Manager, and how point commands and data flow asynchronously through RabbitMQ.

Module Division: Northbound Unified Entry, Four-Center Collaboration, Southbound Protocol Adaptation

The northbound access layer gets its unified entry from dc3-gateway. Built on Spring Cloud Gateway, the Gateway routes /api/v3/auth/**, /api/v3/manager/**, /api/v3/data/**, and /api/v3/agentic/** to the corresponding centers and applies the Authentic filter on protected routes. Route targets use fixed service names and environment variables, with no dependency on a standalone registry.

The platform service layer contains the four centers that actually exist today:

• dc3-center-auth: authentication, authorization, tenant, and OAuth/MCP (Model Context Protocol) management.
• dc3-center-manager: metadata management for Drivers, devices, templates, points, and attributes, plus the gRPC business-registration and query interfaces offered to Drivers.
• dc3-center-data: point-value ingestion, latest-value and history queries, point-command and custom-command submission, execution-receipt processing, and alarm data capabilities.
• dc3-center-agentic: model configuration, session management, and Spring AI @Tool tool calling.

The current architecture has no standalone "Command Service". The point read/write entry belongs to Data; Data publishes commands to RabbitMQ, and Drivers consume them asynchronously and return the results.

The southbound protocol layer consists of multiple independent Driver services, such as MQTT, Modbus TCP/RTU, OPC UA, S7, and IEC 104. The Driver SDK isolates protocol differences behind capability interfaces such as DriverProtocol, DriverReadService, DriverWriteService, and DriverCustomService. At startup, a Driver calls the Manager's gRPC driverRegister through DriverRegisterService to complete business registration; at runtime, it receives point commands and custom commands over RabbitMQ and reports point values, status, events, and execution receipts.

Infrastructure and Communication Boundaries

IoT DC3 places relational data, time-series data, and asynchronous messaging behind replaceable boundaries. The default development stack uses PostgreSQL/TimescaleDB and RabbitMQ, while Caffeine provides an in-process hot cache. DC3_DB_TYPE, DC3_TSDB_TYPE, and DC3_MQ_TYPE select the relational dialect, time-series adapter, and messaging adapter respectively. RabbitMQ Exchanges, queues, TTL, dead letters, and ack/nack are default-adapter details and should not be presented as mechanisms shared by every broker. The platform still has no separate registry such as Nacos; dc3-driver-kafka is a southbound data-source Driver and is not the internal Kafka adapter.

The division of labor between synchronous and asynchronous is as follows:

1. External clients reach Auth, Manager, Data, and Agentic synchronously through the Gateway.
2. Drivers call the Manager synchronously over gRPC to complete business registration and metadata queries.
3. Data delivers point read/write and custom commands asynchronously to the target Driver through RabbitMQ.
4. Drivers report point values, status, events, and command receipts asynchronously to Data through RabbitMQ.

Figure 6-6 IoT DC3 Modules and Data Flows

Technology Stack Selection

In the 987c96d50 snapshot of 2026-08-29, mainline uses Java 21, Spring Boot 4.0.6, Spring Cloud 2025.1.1, and Spring AI 2.0.0. The northbound side uses REST/HTTP; the management contract between centers and Drivers uses gRPC + Protobuf; device-side communication uses whichever client each protocol Driver chooses. The data and messaging layers isolate concrete products behind port adapters. Version numbers and adapter inventories are volatile facts and must be rechecked against build files and the official capability matrix on upgrade.

6.3.2 Device Data Collection and Protocol Adaptation Layer Implementation

The collection layer converts heterogeneous field messages into the platform's unified point values. It must handle protocol connections, encoding and decoding, device and point metadata, read/write semantics, and exception recovery — but it should not push platform business such as alarm rules or history queries into the Driver. IoT DC3 fixes this boundary through independent Driver services and the Driver SDK.

The Driver SDK's Real Capability Interfaces

IoT DC3 currently has no DeviceDriver abstraction that all drivers implement, and no global ConnectionManager provided uniformly by the SDK. Protocol capabilities are composed from fine-grained interfaces:

public interface DriverCustomService extends DriverLifecycle,
        DriverMetadataListener, DriverHealth, DeviceHealth,
        DriverProtocol, DriverCommand, DriverValidator {
}

public interface DriverProtocol {
    ReadPointValue read(Map<String, AttributeBO> driverConfig,
            Map<String, AttributeBO> pointConfig,
            DeviceBO device, PointBO point);

    Boolean write(Map<String, AttributeBO> driverConfig,
            Map<String, AttributeBO> pointConfig,
            DeviceBO device, PointBO point,
            WritePointValue writePointValue);
}

On the SDK side, DriverReadService and DriverWriteService first resolve the device, point, and attribute metadata, then delegate to DriverProtocol to communicate with the real device. A protocol implementation is responsible only for its own protocol's connections, encoding/decoding, and reads and writes: the MQTT Driver manages subscriptions and publications, the Modbus Driver handles registers and byte order, and the OPC UA Driver handles nodes and sessions. Connection pools, heartbeats, and backoff strategies are implemented by each Driver according to its protocol's characteristics — one cannot assume a single global set of fixed reconnection parameters.

Metadata, Point Values, and the Cache Boundary

The Driver SDK uses Caffeine to cache metadata such as Drivers, devices, points, and attributes, avoiding a cross-service query on every collection cycle. At startup, DriverRegisterService performs business registration and metadata synchronization with the Manager over gRPC; this is not service-registry behavior.

After a successful protocol read, DriverSenderService.pointValueSender publishes the standardized point value to the messaging port. Once standardized, the point value enters the selected broker; caching and persistence are handled uniformly on the Data side. The default data path is:

1. The Driver parses the protocol data and produces a PointValue.
2. DriverSenderService publishes it to the RabbitMQ point-value exchange.
3. PointValueReceiver in Data consumes the message and explicitly acks, rejects, or nacks/requeues it.
4. Below the batching threshold, values are saved directly; above it, they first enter PointValueJob's in-process batch buffer and are then written in asynchronous batches.
5. Data writes the latest value into the local Caffeine hot cache while persisting it through TsdbStore; on a cache miss it queries the selected time-series store.

Two kinds of Caffeine are easily confused here: the Driver side caches metadata, while the Data side caches the latest point values. The project has replaced the old Redis Repository layer with local caching, and the current Compose has no Redis service either.

Active Polling and Passive Reporting

Drivers such as MQTT and TCP can receive device-initiated reports in callbacks; protocols such as Modbus RTU and serial links are usually polled actively by the Driver's scheduled tasks. Whether the data comes from a subscription callback or a timed read, it should end up in the same DriverSenderService → RabbitMQ → PointValueReceiver path. Each serial driver designs its own scheduling structure around its protocol's characteristics; the concrete implementation should be taken from the corresponding Driver's source code.

Collection-layer tuning should likewise follow the real bottlenecks: on the Driver side, watch connection count, polling period, and protocol timeouts; on the selected broker, watch routing, backlog, and acknowledgments; on the Data side, watch consumption rate, batching interval, cache hits, and time-series writes. Mis-writing these parameters as a "Driver two-level caching scheme" would put both the troubleshooting target and the responsibility boundary entirely out of place.

6.3.3 Inter-Microservice Communication: From REST to Asynchronous Messaging

IoT DC3 uses REST, gRPC, and RabbitMQ at the same time, but the three are not mixed arbitrarily. REST serves the northbound interfaces; gRPC serves the center–Driver management contracts that need immediate responses; RabbitMQ serves point commands, execution receipts, and uplink data. To judge whether a link is described accurately, the key is not whether it is called the "control plane" or the "data plane," but going back to the actual producers, consumers, and acknowledgment semantics.

Synchronous Links: Gateway Routing and the Driver Management Contract

External requests are first routed by the Gateway to Auth, Manager, Data, or Agentic. The Gateway locates the center services with fixed service names and environment variables such as CENTER_*_HOST and GATEWAY_ROUTE_*_URI.

After a Driver starts, DriverRegisterService calls the Manager's driverRegister over gRPC to complete business registration; metadata that needs immediate responses — devices, points, and attributes — is likewise queried through the gRPC Facade. These calls belong to the synchronous management path, but that does not mean point commands are executed synchronously all the way to the physical device over REST or gRPC.

Asynchronous Links: Point Commands, Receipts, and Point Values

The point read/write entry sits in Data. Data publishes the command to RabbitMQ according to the target Driver's service name; the Driver's PointCommandReceiver consumes it and calls DriverReadService or DriverWriteService, after which DriverSenderService publishes the execution result. Custom commands follow the same kind of path through CommandReceiver.

The uplink direction also uses RabbitMQ: the Driver publishes point values, device status, Driver status, events, and alarms to the corresponding exchanges, and consumers in Data or Manager handle them according to their responsibilities. The real semantics of a device command are therefore "submit — execute asynchronously — result receipt," not "an HTTP request blocks until the device finishes executing."

@RabbitHandler
@RabbitListener(queues = "#{pointCommandQueue.name}")
public void pointCommandReceive(
        Channel channel, Message message, PointCommandDTO command) {
    // validate expireAt and commandId, execute read/write serially per device,
    // send the result receipt before ack; on failure reject or nack/requeue as appropriate.
}

PointCommandReceiver checks expireAt before execution, deduplicates by commandId, and uses a device-level lock to keep protocol operations for the same device from interleaving. The driver-specific command queues are also configured with TTL and a dead-letter exchange. Idempotency here rests on command-DTO validation and a local deduplication cache — lightweight, in-process deduplication on the Driver side; strict cross-instance idempotency, where required, should be designed as a separate mechanism at a higher layer.

RabbitMQ Is Currently the Only Message Middleware

The current messaging port provides RabbitMQ, Kafka, RocketMQ, Pulsar, ActiveMQ, and MQTT 5 adapters, selected through DC3_MQ_TYPE, with only one active in a deployment. RabbitMQ remains the default. Choosing another broker is not a migration completed by renaming a component: official capability matrices and contract tests must verify delayed delivery, dead letters, ordering, acknowledgments, retries, and observability. The repository's dc3-driver-kafka is a southbound data-source Driver, not the same layer as the internal Kafka messaging adapter.

Figure 6-7 IoT DC3 Inter-Service Communication

IoT DC3's communication trade-offs can be summed up in one sentence: the synchronous links solve "get the management result right away," and the asynchronous links solve "cross the device network and the differences in service rates reliably." This boundary is consistent with the current source code and deployment manifests.

6.3.4 Engineering Checklist: Coding Standards, Logging, and Monitoring

Once the code of a microservice architecture is split apart and running, problems that were easy to notice inside the original monolith become hard to trace. A null-pointer exception surfaces on only one node; a device-online log scatters across different containers — these scattered fragments make it difficult to piece together the full state of the system. This section presents a four-layer engineering checklist covering coding standards, the logging system, health checks, and metrics monitoring — the items that mark the watershed between a microservice system that "runs" and one that "can be operated."

Checklist Overview

Table 6-4 lists the practice items that must be covered, across four dimensions. Each item has a corresponding actionable verification method; none relies on intuition.

Table 6-4 IoT Microservice Engineering Checklist

Dimension	Checklist item	Verification method	Notes
Coding standards	Static-analysis tool integration	Enforced to pass at build time	E.g. SonarQube / Checkstyle / SpotBugs, with configuration files kept in version control
Coding standards	Unified exception handling	Full coverage by handler classes	Use @ControllerAdvice or custom interceptors; keep try-catch from polluting business logic
Logging system	Standardized log levels	Output by ERROR/WARN/INFO/DEBUG	No direct System.out; the log format uniformly includes timestamp, thread, and traceId
Logging system	Trace-ID injection	Every request carries a traceId	Inject with Micrometer Tracing or manually via MDC; device-event logs also carry the traceId
Health checks	Custom Actuator endpoints	/actuator/health returns business status	At minimum check database connections, message-queue status, and driver heartbeats
Health checks	Startup/liveness/readiness probes	Kubernetes readiness probe configurable	/actuator/health/liveness and /actuator/health/readiness kept separate
Metrics monitoring	Prometheus endpoint exposure	The scraper can pull /actuator/prometheus	Register Micrometer metrics — business metrics such as device collection counts, message-processing latency, and point read/write counters
Metrics monitoring	Grafana alert rules	Test the trigger after configuring thresholds	E.g. "device heartbeat timeout > 30 seconds" raises an alarm, notified via DingTalk/email

For each item's actual configuration, refer to the official Spring Boot Actuator documentation. Actuator provides dozens of built-in endpoints, of which /health, /info, /metrics, and /prometheus are the most critical for microservice operations. In IoT scenarios, determining a device heartbeat timeout is often not a simple node-liveness check — a custom health endpoint is needed to aggregate device-level status.

A Custom Health Endpoint Example

Suppose a protocol driver component needs to report whether the devices it connects to are online. The default /actuator/health checks only the Spring container and the database — it cannot show "whether the driver's TCP connection to the PLC is working." The following code shows how to extend business health checks with Spring Boot Actuator's HealthIndicator interface:

@Component
public class DeviceDriverHealthIndicator implements HealthIndicator {

    private final List<DeviceConnection> connections;

    public DeviceDriverHealthIndicator(List<DeviceConnection> connections) {
        this.connections = connections;
    }

    @Override
    public Health health() {
        long offlineCount = connections.stream().filter(c -> !c.isAlive()).count();

        if (offlineCount == 0) {
            return Health.up()
                .withDetail("totalConnections", connections.size())
                .withDetail("status", "all devices online")
                .build();
        }

        return Health.down()
            .withDetail("totalConnections", connections.size())
            .withDetail("offlineCount", offlineCount)
            .withDetail("status", offlineCount + " device(s) offline")
            .build();
    }
}

This code exposes the device driver's connection status as a health-check metric. When offlineCount>0, the overall status is marked DOWN, and a Kubernetes readiness probe can immediately cut traffic away on that basis.

Metrics Visualization and the Alerting Flow

Metrics data needs an aggregation layer before it becomes useful. The recommended practice is:

1. Metrics exposure: each microservice enables management.endpoints.web.exposure.include=health,info,metrics,prometheus in application.yml.
2. Data collection: Prometheus pulls each node's /actuator/prometheus endpoint periodically in pull mode.
3. Visualization: Grafana connects to the Prometheus data source and configures dashboards for connected-device counts, message-queue backlog, API response percentiles, and more.
4. Alerting: set thresholds that trigger alert notifications (by integrating Prometheus Alertmanager, for example).

The heart of this pipeline is the selection of business metrics. Common IoT metrics include the device registration success rate, message publish QPS, point-query P99 latency, and driver disconnection frequency. Only after baseline values are set for these metrics does one truly have "observability" into system anomalies.

Key Judgments in the Engineering Checklist

A few items in the checklist are easily neglected early in a project:

• Log traceIds must run end to end: device data travels from the driver to the message queue to the data service; if every hop cuts the traceId, debugging means combing through three or four log files to piece timestamps together. Uniformly injecting a traceId costs little and pays back enormously.
• Custom health checks should not be just "UP/DOWN": return detailed status key-value pairs, so operators can see at a glance "which device is offline" or "which database connection pool is full".
• Alert rules need severity tiers: a device heartbeat timeout can raise a WARNING alarm; a continuous gap in core point data must raise a CRITICAL alarm and notify the on-call engineer.

---

The following summarizes the monitoring-system design as a layered architecture diagram; each metric type corresponds to a different collection and storage path.

Figure 6-8 Microservice Observability Architecture

Three-Pillar Observability: From Device Command to Final State

The observability of an AIoT system cannot stop at answering "is the process alive" — it must be able to follow one business action from the device through to its final state. The recommendation is to build a unified model around the three pillars of logs, metrics, and traces:

• Traces: generate the same traceId for every "API → Gateway → Data → Driver → device receipt" chain; use OpenTelemetry semantic conventions to describe span names, attributes, and status.
• Metrics: device connectivity rate, message received/duplicate/out-of-order rates, command success rate, acknowledgment latency, and alarm count; every metric must state its denominator, window, and aggregation method explicitly, preventing same-named metrics from drifting in meaning.
• Logs: structured output with fields that at minimum include timestamp, traceId, spanId, tenant, user, device, Tool, approval ID, and error code. Security events such as approvals, command receipts, and model decisions get their own labels for compliance audit.

The binding among the three matters more than the tools themselves: traces and logs share IDs, metrics and alarms share labels, and human approvals and device receipts can be linked back to the original request. Without unified IDs, replaying an incident after the fact can only be done by stitching logs together by hand. This chapter fixes only this general skeleton; when model and tool calls appear in the chain, how LLM/Tool child spans, token-cost metrics, and model/Prompt version labels are incorporated into the three pillars is expanded in Section 7.4.3.

Canary Release and Rollback

Deployment practice before production rollout should make "canary release + independent rollback" a default capability rather than an improvised remedy after an incident:

• Every release is tied to a manifest: image digests, Compose/K3s configuration, dependency versions, and configuration items;
• Changes first pass through shadow traffic or shadow-writes (reading real requests without producing external side effects);
• Production entry is canaried along tenant/device dimensions, observing the data path, command receipts, and business metrics;
• On regression, roll back component by component: image rollback, configuration rollback, and dependency rollback are mutually independent;
• Traces are retained after rollback, to make it easier to review the cause of failure and the boundary of drift;
• High-risk OTA, driver upgrades, and edge-node changes must go through separate approval and separate batches; upgrading all gateways in one full sweep is not allowed.

Neither canary release nor rollback is a "process ritual"; their value is turning "looks better" into evidence-backed change management: who approved it, what was changed, what was observed, and how to undo it next. If the release unit also contains non-code assets such as models or Prompts, the requirements for version registration and per-component rollback tighten by one more level; they are discussed specifically in Section 7.4.3.

Engineering Collaboration and Multi-Repository Version Alignment

The first engineering problem after microservices land is often not technical but collaborative. IoT DC3 keeps the center services and the protocol Drivers in one repository, where the module is the boundary; when drivers are maintained by different teams or even different organizations, the Drivers, the Driver SDK, and the deployment manifests are often split into multiple Git repositories that release independently. Multiple repositories buy the freedom of decoupling, at the cost of making "which exact code is running in production" hard to answer, and three disciplines are needed to keep things aligned: split repositories by cadence of change, so that only interface contracts flow across repositories; make every image tag traceable back to a source commit — tag with semantic versions or short commit hashes, and never accept latest alone; keep Driver SDK interface evolution backward-compatible, with major versions aligned to the platform contract, have each Driver declare the SDK version range it supports in its dependency manifest, and before a platform upgrade check the compatibility matrix first, then schedule the driver upgrade batches.

← 6.2 Microservice Architecture Methodology6.4 Engineering Summary and Further Reading →