14.2 An End-to-End IoT DC3 Project

Author: Zhang Hongyuan · © 2016–2026 · All Rights Reserved · Copyright & License

14.2.1 Project Background and Requirements Definition

Most failed IoT projects do not fail at coding — they fail before the first line of code is written, in the requirements definition stage. Teams spend long hours discussing "we want to build a powerful IoT platform," yet nobody defines the concrete engineering boundaries of "powerful." The feature list runs to dozens of items, every priority is P0, and at delivery the core path does not work while the peripheral features are exquisitely polished. This "requirements gilding" is especially common in IoT projects, because access to the physical world involves many dimensions and long chains of constraints, and both the requirements side and the development side easily overlook the existence of engineering boundaries.

IoT DC3 is an open-source industrial IoT platform with a clear-cut position. Its design goal is to connect field devices and cover the core capabilities of device management, data collection, a rule engine, and data services — not to become an all-embracing "Internet of Everything operating system." This pragmatic positioning makes it an ideal reference object for understanding the engineering boundaries of an IoT platform. In a typical open-source IoT platform architecture, the core consists of a few modules with clean responsibilities — device management, data persistence, rule engine, and protocol adaptation — while protocol drivers are deployed independently and communicate asynchronously with the main services through a message queue. This decoupled design dictates what the requirements definition stage must answer: in your scenario, how many protocols must the protocol drivers support? What is the peak throughput of device uplink data? To what level do the rule engine's real-time requirements reach?

This means the engineering boundaries of an open-source project are not necessarily the boundaries your project actually has to face. In the requirements definition stage, the most critical deliverable is not "how much can be done" but "what will not be done this round." That requires you, building on an understanding of the platform's capabilities, to run a drill-down review of the real business scenario.

The following paragraphs put the methodology of Section 14.1.1 to work on an example that runs through this chapter: building a smart-factory management platform on IoT DC3.

Consider a mid-sized electronics manufacturing plant with about 2,000 devices, including SMT placement machines, reflow ovens, AOI (Automated Optical Inspection) units, and temperature/humidity sensors. Its current engineering pain points: device status is tracked by manual inspection, and data formats are inconsistent — some devices support Modbus TCP, some output only serial data, and a few aging devices speak a custom binary protocol. Production anomalies are reported only after an operator notices them, and the average time from fault occurrence to manual confirmation is on the order of forty minutes.

After several rounds of discussion with the plant's operations team, the business requirements converged into four core goals: unified device access with real-time status collection; historical data storage and trend analysis; alarm rule configuration with multi-channel push (shop-floor dashboards, WeChat, email); and a first attempt at predictive maintenance based on device data. These four requirements map one-to-one onto the plant's operational pain points: device access solves the data silos, storage and analysis solve "having data but not seeing it," alarms solve the lagging response, and predictive maintenance solves reactive repair.

For this example, the functional modules can be divided as follows.

Device access module: responsible for protocol adaptation. The smart factory involves Modbus TCP, serial links (custom protocol), and some newer devices that support MQTT. Different protocols map to different drivers; the drivers run close to the field devices, and the collected data is reported to the cloud through a message queue rather than connecting directly to the core services. This layer does no data storage — only format conversion and data forwarding.

Device management module: responsible for device registration, grouping, status tracking, and lifecycle management. Metadata such as start/stop state, firmware version, online status, and the production line a device belongs to is maintained here.

Data center: responsible for receiving, persisting, and querying the collected data. A time-series database stores device point values, while a relational or document database stores device configurations and event records. The alarm engine works with the data center and raises an alarm when a value crosses the configured threshold.

Intelligent analysis module: responsible for model training, inference, and rule linkage. This round takes the lightweight path — start with statistics-based anomaly detection (such as outlier identification and trend drift) instead of rushing deep-learning models into production. The concrete engineering implementation of this module is covered in later sections; it is also the entry point for integrating AI capabilities later on.

Application and service layer: this layer serves people and business systems. Field operations staff understand device status through device lists, data dashboards, and alarm pages; production management systems read device events, work orders, and statistical results through interfaces; and systems such as the MES (Manufacturing Execution System) and ERP (Enterprise Resource Planning) complete cross-system coordination through APIs (Application Programming Interfaces).

Once the functional modules are divided, one more easily neglected task remains: setting boundaries. In this example, the following capabilities are explicitly assigned to phase two or phase three: device OTA (Over-the-Air) upgrades, the device shadow, multi-tenant isolation (there is currently a single plant), and a fully automatic production-scheduling scheme based on reinforcement learning. The point of boundary definition is that it lets both the development team and the business side know this is a starting point, not an endpoint. The team can iterate with focus on the four requirements instead of scattering effort on the illusory goal of a "do-everything platform." At every requirements review, one question — "does this feature directly serve the four core requirements?" — makes most gilded requirements disappear on their own.

The deliverable of the requirements definition stage is a requirements document that can be reviewed, contested, and revised, accompanied by an explicit list of functional modules and a boundary statement (including an explicit "will not do" list). The document does not pursue perfection, but it must carry priorities and trade-offs. Once the requirement boundaries are clear, the downstream architecture design, testing, and acceptance have a stable basis for judgment; vague boundaries drag all of these stages into repeated rework.

14.2.2 System Architecture Design

IoT DC3 can be understood as four layers: the southbound device layer, the protocol Driver layer, the platform service layer, and the application presentation layer. The value of this layering is not the diagram — it is making explicit which calls can be synchronous, which data must be asynchronous, and who is responsible for service addressing and configuration.

Responsibilities of the Four Layers

• Southbound device layer: sensors, PLCs, controllers, and third-party systems, using protocols such as MQTT, Modbus, OPC UA, and IEC 104.
• Protocol Driver layer: each protocol is deployed independently, responsible for connection, encoding/decoding, point read/write, and status reporting. Drivers can be pushed down to edge nodes as the site requires.
• Platform service layer: Auth handles authentication and authorization; Manager handles metadata for drivers, devices, templates, points, and attributes; Data handles point values, commands, receipts, alarm data, and queries; Agentic handles models, conversations, and Spring AI Tools.
• Application presentation layer: web clients, third-party applications, and API clients access the platform uniformly through the Gateway.

Current Service Governance and Messaging Infrastructure

IoT DC3 currently has no Nacos or other separate service registry. Gateway routes and gRPC channels use fixed service names, the Compose network resolves them through DNS, and addresses can be overridden with environment variables such as CENTER_*_HOST and GATEWAY_ROUTE_*_URI. Default configuration lives in the project YAML, and deployment parameters are injected through environment variables.

Internal messages pass through a unified messaging port, with RabbitMQ as the default adapter. The code also provides Kafka, RocketMQ, Pulsar, ActiveMQ, and MQTT 5 adapters, selected by DC3_MQ_TYPE. Data hands point commands and custom commands to the messaging port; the Driver consumes them, performs the protocol operations, and returns result receipts, point values, status, and events. dc3-driver-kafka is a southbound data-source Driver and is distinct from the internal Kafka adapter.

Figure 14-5 IoT DC3 Layered Architecture

The engineering trade-off is that management and metadata queries need immediate results and therefore use REST/gRPC, while device commands and uplink data need asynchronous decoupling and rate isolation and therefore use the unified messaging port. RabbitMQ is the default adapter. Clear boundaries matter more than component count.

14.2.3 Core Module Implementation

To understand how IoT DC3 is implemented, read the source along three real call chains instead of fitting it onto the generic template of "service registry + Kafka + standalone command service."

Driver Business Registration and Metadata Synchronization

After a Driver starts, DriverRegisterService calls the Manager's driverRegister over gRPC. What gets registered is the Driver's business identity, configuration, and metadata — not an IP entry in a registry such as Nacos. Runtime metadata such as devices, points, templates, and attributes is likewise queried through the Manager facade and cached in the in-process Caffeine cache inside the Driver.

Point Value Reporting and Data Processing

Protocol implementations perform real device reads and writes through DriverProtocol. Data obtained by reading or subscribing is converted into the unified PointValue, then handed to the messaging port by DriverSenderService; the default RabbitMQ adapter performs the concrete publish. Data's PointValueReceiver receives from the same port: below the batching threshold it saves directly, and above it messages enter the in-process PointValueIngestBuffer for batch writes. Data also keeps a local Caffeine cache of the latest values and writes history through TsdbStore, whose default implementation is TimescaleDB. Alarm-rule processing starts after persistence completes.

Point Commands and Result Receipts

The entry point for point reads and writes sits in Data. Data hands commands keyed by Driver service name to the messaging port. The Driver's PointCommandReceiver checks expireAt and commandId, serializes protocol operations for one device with a device-level lock, and calls DriverReadService or DriverWriteService. Success or failure results return to Data through the same port. Ack, reject, nack/requeue, TTL, and dead-letter exchanges are concrete semantics of the default RabbitMQ adapter; another adapter must demonstrate equivalent acknowledgment, retry, expiry, and failure-isolation behavior.

Engineering Boundaries

• There is no standalone Command Service; the command entry point and receipt handling belong to Data.
• The default data plane uses RabbitMQ. After replacing the broker, commands, receipts, point values, status, and events still pass through the same messaging port, but acknowledgment, ordering, dead-letter, and delay capabilities must be reverified for the adapter.
• There is no two-level Redis device shadow; the Driver caches metadata, and Data caches the latest point values in a local Caffeine cache.
• There is no unified DeviceDriver or global ConnectionManager; protocol drivers are implemented against capability interfaces, each with its own connection model.

Reading the code along these three chains lets you separate "synchronous management calls" from the "asynchronous device data flow" precisely, and to locate the responsibility boundaries for performance and reliability directly.

14.2.4 Device Access and Data Flow

The core challenge of device access is not network connectivity but converging protocol semantics. MQTT, Modbus, and OPC UA differ in connection model, timing, and data representation — MQTT relies on devices publishing proactively, Modbus is polled by the Driver, and OPC UA can subscribe to node changes. The Driver layer must converge these heterogeneous protocols into the unified PointValue and command model. The protocol entry points differ; the data path after entering the platform is what stays uniform.

From Device Payload to Point Value

Take the MQTT scenario: device payloads can use JSON, but the topics and field structures are defined by the specific Driver — there is no single fixed payload mandated platform-wide. The Driver handles connection, subscription, deserialization, and device/point mapping, then calls the unified sender service.

Below is a simplified example of a device attribute-report JSON structure. It illustrates the field-design thinking only and is not a mandatory format for all IoT DC3 MQTT drivers:

{
  "deviceCode": "device-001",
  "timestamp": 1700000000123,
  "values": {
    "temperature": 25.6,
    "humidity": 68.2,
    "pressure": 1013.2
  },
  "qos": 1,
  "msgId": "a1b2c3d4"
}

• deviceCode corresponds to a device identity already registered on the platform; the Driver obtains this mapping from Manager metadata synchronization at startup.
• The keys inside values are point identifiers; the values can be numeric, string, or boolean, and the Driver determines the type from the template definition.
• msgId is used for uplink deduplication; on the consuming side, Data makes the idempotency judgment based on the msgId (or the combination of deviceCode + timestamp).

In real projects, once the number of points runs into the hundreds, the CPU cost of JSON parsing and serialization becomes significant. At that point consider switching to Protobuf or MessagePack — the payload structure stays unchanged, only serialization/deserialization is swapped in the Driver layer, and the Data side keeps a unified consuming interface.

Components and Functions at Each Stage of the Data Flow

Table 14-2 shows the responsibilities and risks along the path from a device through the messaging port, cache, and time-series storage port.

Table 14-2 Responsibilities and risks at each stage of the uplink point-value data flow

Stage	Component	Primary responsibility	Concurrency/consistency constraint	Key risk
Protocol access	Device-side protocol (MQTT/Modbus/OPC UA)	Send or respond to data per the protocol specification	Connection keep-alive, heartbeat	Transient network drops losing data; duplicate topic/node subscriptions after reconnect
Protocol parsing	Driver (DriverProtocol implementation)	Deserialize raw payloads and convert them into PointValue objects per Manager metadata	Connection and concurrency models depend on the protocol implementation; Driver caches metadata in local Caffeine	Payload drift, blocking calls, or mishandled connection state causing parse and resource failures
Message delivery	DriverSenderService → messaging port	Publish a unified PointValue; the default RabbitMQ adapter maps it to the relevant exchange	Routing, acknowledgment, ordering, persistence, and batching depend on the selected adapter	Production outruns consumption; broker capacity or retention mismatch
Async consumption	Data's PointValueReceiver	Receive from the messaging port and either save directly or enter PointValueIngestBuffer by threshold	Acknowledgment and redelivery must match the adapter contract; buffer thresholds require measurement	Backlog, duplicates from redelivery, or widened impact from batch failure
Cache update	Data → local Caffeine cache	Keep the latest point values visible to this instance for fast queries	JVM-local state; do not assume strong consistency across instances	Stale values, inter-instance differences, JVM memory pressure
Persistence	Data → TsdbStore	Write point-value history; the default adapter is TimescaleDB	Batching, retention, aggregation, and query behavior depend on the TSDB adapter	Write or query bottlenecks; retention, indexing, or partition mismatch
Alarm triggering	Data → alarm-rule processing	Evaluate rules after persistence and create alarms	Define idempotency for duplicates, retries, and alert creation	False or missed alarms; replay-induced alarm storms

Asynchronous Receipts for Downlink Commands

Downlink commands take the reverse asynchronous path. The client calls Data's point-command API through the Gateway, and Data hands the command body to the messaging port. The target Driver consumes it, performs the device operation, and returns the result receipt through the same port. The default RabbitMQ adapter maps the traffic to exchanges such as dc3.e.point_command. Clients should subscribe through WebSocket or poll Data's command-status API rather than assume that HTTP blocks until the device responds.

Before execution, the Driver uses commandId for deduplication and expiry checks and a device-level lock to serialize protocol operations for one device. The command-ID issuer, retention window for deduplication state, and whether that state is shared across instances must follow the current API and implementation and be verified with replay tests. A local lock and in-process deduplication do not automatically provide global exclusion across Driver instances.

Capacity Observation and Bottleneck Diagnosis

The principle of capacity design is: observe first, optimize later. In the default stack, watch message rate, backlog, and unacknowledged messages in the RabbitMQ console; watch consumption and write latency at Data's monitoring endpoint; and observe hypertables, queries, and disk IO on the TimescaleDB/PostgreSQL side. When another adapter is used, switch to its corresponding metrics. Consider partitioning, hot/cold tiering, or replacing an adapter only after load tests prove that one link is the bottleneck. Repository "support" for a broker or time-series database does not prove that the target load has been validated.

Engineering checklist:

• [ ] Device connection stability: use MQTT last-will messages and an automatic-reconnect policy; configure timeout and retry on the Modbus Driver.
• [ ] Uplink message idempotency: on the Data side, deduplicate by msgId or deviceCode + timestamp to avoid duplicate writes.
• [ ] Downlink command de-duplication: the client generates a global UUID as the commandId; set a timeout on the Driver-side device lock (for example, 30 seconds).
• [ ] Backlog alarm threshold (example): alarm when RabbitMQ queue depth exceeds 10,000 and holds for 60 seconds; the actual threshold should be calibrated against the baseline and the SLA.
• [ ] Slow database writes (example parameters): monitor track_io_timing for the dc3_point_value table and set PostgreSQL log_min_duration_statement = 200ms; actual parameters should be calibrated against the on-site load.

Figure 14-6 Device Access & Data Flow

14.2.5 Building AI Operations Capabilities (Not Out of the Box)

In the IoT DC3 source snapshot 987c96d50, Agentic Center implements model configuration, conversation management, Spring AI @Tool invocation, and Web/HTTP chat. The Gateway's /mcp endpoint follows revision 2025-06-18 and handles initialize, notifications/initialized, ping, tools/list, and tools/call; it declares only the Tools capability and implements neither Resources, Prompts, nor Tasks. The project Compose contains no TensorFlow Serving, no training jobs, and no model volumes, and there is no default path by which Agentic subscribes to Data's real-time point-value stream. This section therefore discusses predictive maintenance only as an optional engineering extension — it must not be written up as a current out-of-the-box capability.

Rules First, Then Statistics, Then Models

Anomaly detection comes in three tiers: fixed thresholds handle explicit red lines; statistical methods such as sliding windows, IQR, and Z-score handle slow drift; supervised or unsupervised models handle multivariate coupling, temporal dependency, and patterns that resist hand-written rules. The three tiers are not substitutes for one another. A model earns its introduction only when the baseline rules cannot meet the need and data quality, labels, and returns are sufficient to support it.

An Optional Predictive Maintenance Extension

If a project genuinely needs model inference, design it within the following boundaries:

1. Obtain point data through Data's history-query API, under tenant authorization.
2. Perform time alignment, missing-value handling, windowing, and training outside the platform.
3. Deploy the model as a standalone inference service protected by authentication.
4. Have an authorized job read data from Data and call the inference service.
5. Write the inference result back to a clearly named derived point, for example bearing_anomaly_score.
6. Reuse the existing rule and notification chains to judge thresholds and durations.

Model type, window length, and thresholds must be validated by data. LSTM, a window of 32, and a threshold of 0.85 are hypothetical examples only — they must not be written as IoT DC3 defaults. Spring AI Tools suit the orchestration of queries, explanations, and controlled execution; they do not amount to high-frequency streaming inference. MCP, likewise, only exposes the authorized Tools to external agents; it does not train or deploy models.

Figure 14-7 Predictive Maintenance Extension Example

The security boundary includes at minimum input range validation, authentication and rate limiting on the inference endpoint, model-version auditing, tenant isolation, and permissions on derived points. AI capabilities must not bypass the platform's existing governance logic.

14.2.6 Deployment and Testing

The deployment stage must verify that the components IoT DC3 actually provides can start completely inside the container network and that Driver business registration, point-value reporting, and point-command receipts all run through. For the 987c96d50 snapshot dated August 29, 2026, the default development stack is based on PostgreSQL/TimescaleDB and RabbitMQ. Platform services include Gateway, Auth, Manager, Data, and Agentic, with protocol Drivers enabled by the selected stack. Optional stacks provide other brokers, TSDBs, and observability components. The template contains no Nacos and no model-inference container or model volume.

The Current Compose Topology

x-app-runtime-env: &app-runtime-env
  DC3_MQ_TYPE: rabbitmq
  DC3_TSDB_TYPE: timescale
  POSTGRES_HOST: dc3-postgres
  RABBITMQ_HOST: dc3-rabbitmq
  CENTER_AUTH_HOST: dc3-center-auth
  CENTER_MANAGER_HOST: dc3-center-manager
  CENTER_DATA_HOST: dc3-center-data
  CENTER_AGENTIC_HOST: dc3-center-agentic

services:
  postgres:
    container_name: dc3-postgres
  rabbitmq:
    container_name: dc3-rabbitmq
  gateway:
    environment: { <<: *app-runtime-env }
  auth:
    environment: { <<: *app-runtime-env }
  manager:
    environment: { <<: *app-runtime-env }
  data:
    environment: { <<: *app-runtime-env }
  agentic:
    environment: { <<: *app-runtime-env }
  mqtt:
    environment: { <<: *app-runtime-env }

Start with podman compose. depends_on expresses only the dependency relationship — you still need healthcheck plus application-level retries to wait until PostgreSQL and RabbitMQ are truly ready. Containers address each other by service names such as dc3-postgres, dc3-rabbitmq, and dc3-center-*; localhost must not be treated as another container. Sensitive variables should be injected from .env or a secret manager; never commit real credentials.

From Zero to the First Point: A Versioned Acceptance Sequence

Services being up does not count as a successful deployment — only the full chain running through does. The sequence below corresponds to snapshot 987c96d50 and selects the built-in Virtual Driver to avoid additional dependencies on an MQTT broker, Topic, and vendor payload. Every generated ID and Token must be replaced with the real value returned by the previous step. If the repository commit differs, read that version's README and official "First Device: End to End" first; do not mix commands across versions. This is a verifiable acceptance order, not a promise that every line will remain copyable in future releases.

Step 1: Get the code.

git clone https://github.com/pnoker/iot-dc3.git && cd iot-dc3

Expected: a complete repository containing dc3/, dc3-center/, dc3-driver/, the Makefile, and .env.example.

Step 2: Start the infrastructure.

make up-db        # Make target defaults to podman compose; with mainland-China registry mirrors use make up-db-cn

Expected: the PostgreSQL and RabbitMQ containers are running; on first start the database is initialized in the order extensions, common, auth, data, manager, history, agentic.

Step 3: Verify service health.

podman ps
podman exec dc3-postgres psql -U dc3 -d dc3 -c '\dt dc3_auth.*'

Expected: dc3-postgres and dc3-rabbitmq show status Up; the tables of the auth schema are listed. Host-mapped ports defer to .env (the current Quick Start uses PostgreSQL 35432 and RabbitMQ AMQP 35672; inside the containers they remain 5432/5672).

Step 4: Start the platform services and exchange for a token.

source dc3/env/dev.env.sh
make up-dev       # equivalent to make up STACK=dev; start order: Auth first, Gateway last
curl -s -X POST http://localhost:8000/api/v3/auth/token/salt \
  -H 'Content-Type: application/json' -d '{"tenant":"default","name":"dc3"}'

Expected: a salt valid for 5 minutes is returned; then call /api/v3/auth/token/generate (carrying the salt and the password hashed per the rules — the hashing rules defer to the official Quick Start) to exchange it for a token valid for 12 hours. From then on, every request carries the three headers X-Auth-Tenant, X-Auth-Login, and X-Auth-Token. The Gateway is the only external HTTP entry point (port 8000); the direct ports of Auth/Manager/Data are for debugging only.

Step 5: Confirm Driver registration and prepare device metadata.

curl -s -X POST http://localhost:8000/api/v3/manager/driver/list \
  -H "$H_TENANT" -H "$H_LOGIN" -H "$H_TOKEN" -H 'Content-Type: application/json' -d '{}'

Expected: the list of Drivers started with the stack. A Driver appearing here means the gRPC business registration described in Section 14.2.3 succeeded. Next, follow the official Quick Start for the same release to create a profile, a point such as Temperature/FLOAT/READ_WRITE, and a device bound to the Virtual Driver, then record the deviceId and pointId. To use MQTT or another protocol instead, first confirm that its Driver, southbound service, and attribute model are enabled, then replace the Driver-specific steps in this sequence.

Step 6: Configure the Virtual Driver's point attribute and wait for automatic reporting.

Obtain the actual attributeId from the Point Attribute list registered by the Virtual Driver, then call /api/v3/manager/point_attribute_config/add to write configValue for the deviceId and pointId from the previous step. After this configuration, the Virtual Driver produces point values without inventing a nonexistent generic MQTT Topic or payload. Use the request body from the official First Device page for the same version. attributeId is registered at runtime and must not be hard-coded in this book.

Step 7: Query the point value over REST.

curl -s -X POST http://localhost:8000/api/v3/data/point_value/latest \
  -H "$H_TENANT" -H "$H_LOGIN" -H "$H_TOKEN" -H 'Content-Type: application/json' \
  -d '{"deviceId":"<DEVICE_ID>","pointId":"<POINT_ID>","page":{"current":1,"size":10}}'

Expected: the latest records for that point are returned (fields such as rawValue, calValue, numValue, and createTime) — proof that the uplink path "Driver → messaging port → Data → time-series storage port" is through. The default adapters correspond to RabbitMQ and TimescaleDB.

Step 8: Issue a write command.

curl -s -X POST http://localhost:8000/api/v3/data/point_command/write \
  -H "$H_TENANT" -H "$H_LOGIN" -H "$H_TOKEN" -H 'Content-Type: application/json' \
  -d '{"deviceId":"<DEVICE_ID>","pointId":"<POINT_ID>","value":"26.5"}'

Expected: the API returns a commandId immediately and the command executes asynchronously; only READ_WRITE/WRITE_ONLY points are writable, and a command expires by default after about 10 seconds (expireAt) — once expired without being executed, it fails.

Step 9: Check the command receipt.

curl -s "http://localhost:8000/api/v3/data/point_command_history/get_by_command_id?commandId=<COMMAND_ID>" \
  -H "$H_TENANT" -H "$H_LOGIN" -H "$H_TOKEN"

Expected: the command status and receipt are visible; if the status is expired or failed, use the commandId together with the receipt details to locate the cause (common causes in Section 14.3.5).

Step 10: Close out with the logs.

podman logs dc3-center-data --tail 50
podman logs dc3-driver-virtual --tail 50  # use the actual service name in the current Compose file

Expected: the Data log shows point-value consumption and save records, and the Driver log shows registration and read/write execution records. In the default stack, use the RabbitMQ console to inspect backlog and dead letters. With another DC3_MQ_TYPE, inspect the adapter's equivalent metrics. Both uplink and downlink now have inspectable evidence.

Smoke and Performance Testing

Table 14-3 Smoke-test scenarios and expected results

Scenario	Verification action	Expected result
Service startup	podman compose ps and readiness	Infrastructure and required services healthy
Driver registration	Start one protocol Driver	Manager receives the gRPC business registration
Data reporting	Configure the Virtual Driver's point attribute as in Step 6 and wait for a report; for another protocol, use that Driver's official access procedure for the same release	The point value enters Data through the messaging port and is written through TsdbStore; defaults are RabbitMQ and TimescaleDB
Command dispatch	Call Data's point-command API	The messaging port delivers to the target Driver and the result receipt returns to Data; default RabbitMQ semantics are observable
Failure recovery	Pause the selected broker or consumer, then resume	The adapter's declared redelivery, failure isolation, backlog, and alarm behavior matches configuration

Performance testing should separately observe Driver collection and lock waits, backlog and acknowledgment state in the selected messaging adapter, Data consumption and batch saves, and write and query latency in the selected TsdbStore. The default stack uses RabbitMQ and TimescaleDB/PostgreSQL. Another adapter requires its own metrics; an unexecuted tuning report cannot substitute for measurement.

Figure 14-8 IoT DC3 Container Deployment Architecture

14.2.7 Reproducible Experiments, Acceptance Metrics, and the Evidence Package

A screenshot of a successful deployment proves only that the services were up at one moment; it cannot prove that the system works repeatably under fixed load, fault, and security constraints. A publication-grade case study must let third parties know what version ran, on what data, how the load was applied, how the metrics were computed, and where the raw results live. Projects without measured results may describe their design and method, but must not pass numbers off as results.

Freeze the Environment Manifest First

Save an immutable manifest for every experiment round, recording at least:

• IoT DC3 Git commit/tag, uncommitted patches, and repository state;
• Container image digests, Compose file, and environment-variable template versions;
• OS, CPU, memory, disk, network, Podman, JDK, Python;
• DC3_TSDB_TYPE, DC3_MQ_TYPE, their service versions, the Driver, and device/simulator firmware versions;
• Model provider, model ID, service version, prompt hash, and Tool schema version;
• RAG corpus, chunking, embedding, reranker, and index versions;
• Test-data name, license, split, and SHA-256;
• Seed, time zone, NTP/clock conditions, and run duration.

Secrets and personal data must never enter the manifest; use environment-variable names, credential IDs, or redacted digests. When an external provider cannot guarantee determinism, record the region, request parameters, and repetition count — do not claim the seed fully reproduces the outputs.

The Workload Must Be Replayable

"Simulate a large number of devices" cannot be reproduced. Pin down the device count, points per device, reporting frequency, payload size, read/write ratio, command ratio, duration, and warm-up time. Fault experiments must additionally fix the network latency/loss, disconnection windows, consumer pauses, broker/database restart moments, number of concurrent agent sessions, and the timeout/error-injection ratios for models and Tools.

Baselines must be explicit too, for example: rules only, no AI; agent without RAG; read-only Copilot; constrained agent. Change only the primary variable in a single comparison; if hardware, data, and model all change at once, the differences cannot all be attributed to one component.

A Metrics Dictionary: Define the Denominator Before Reporting Numbers

Table 14-4 Metrics dictionary and suggested aggregation

Layer	Metric	Denominator/window	Suggested aggregation
Device access	Registration success rate, stable online rate, reconnection time	Target devices/test window	Ratio, P50/P95
Data path	Reception rate, duplicate rate, out-of-order rate, end-to-end latency	Expected messages/received messages	Ratio, P50/P95/P99
Command path	Success rate, acknowledgment latency, expiry rate, duplicate-execution rate	Submitted commands	Ratio, P50/P95
Storage	Write throughput, write/query latency, growth	Fixed workload and window	Rate, P95, bytes
Reliability	Backlog recovery, dead letters, RTO, RPO, data gaps	Each fault scenario	Duration, count
RAG	Recall@k, faithfulness, refusal accuracy	Versioned evaluation set	Ratio and confidence interval
Agent	Task success, correct parameters, privilege escalation, takeover, duplicate side effects	Golden tasks/attack sets	Ratio, zero-tolerance items
Cost	Cost per 10,000 telemetry messages, per task, per successful task	Explicit billing and resource boundary	Currency, tokens, CPU-hours

The latency endpoints must be fixed. For example, end-to-end telemetry latency can be defined from the simulator's generation time to Data's persistence acknowledgment; command acknowledgment latency can be defined from the API accepting the action to the Driver's receipt. Different chapters and figures must use the same definition.

Repeated Runs and Uncertainty

Each scenario should be run independently several times, reporting the sample count, the median or mean, the standard deviation or confidence interval, and P95/P99 for the long tail. Keep warm-up data separate from the formal samples. LLM experiments need per-task results and traces saved, so that one successful answer never stands in for overall capability.

If the sample size is insufficient, state the limitation explicitly; if a metric has not been run, fill in NA (not executed) rather than 0. 0 means it did not occur after measurement; NA means there is no evidence — the two mean completely different things.

Fault and Security Test Cases

The minimal experiment package covers at least:

1. Duplicate telemetry and out-of-order timestamps;
2. Reconnection after a brief Driver or network disconnection;
3. Consumer pause and backlog recovery for the selected messaging adapter;
4. Database unavailability and recovery;
5. Insufficient user permissions and cross-tenant requests;
6. Model timeouts, Tool timeouts, and dirty returns;
7. Action executed but the receipt lost;
8. Replay with the same idempotency_key;
9. Manual takeover and kill switch.

For each case, record the expected state, the actual state, side effects, logs, and the recovery outcome. Device-control experiments should prefer simulators, shadow mode, or non-safety-critical devices; never bypass PLC/SIS interlocks for the sake of a demo.

The Publication Evidence Package

For every experiment cited in the book, save:

experiments/EXP-14-E2E-01/
├── README.md              # reproduction steps and known limitations
├── manifest.json          # versions, environment, and data hash
├── workload.yaml          # workload and fault parameters
├── commands.txt           # actual commands executed
├── raw/                   # raw metrics, logs, and per-task traces
├── summary.json           # metric definitions and summary
├── failures/              # failure samples and postmortems
└── figures/               # method for generating figures from raw

Measured numbers in the text must link back to the experiment ID and the location of the raw results. Data that cannot be made public should be represented by a redacted sample or a substitute generator, with an explanation of how it differs from the real data. Experiment scripts, data, and third-party components must also state their licenses.

Experiment card EXP-14-E2E-01

• Hypothesis: under fixed device load and fault windows, the system meets the pre-defined data, command, security, and recovery thresholds;
• Fixed items: commit, image digest, hardware, dependencies, data hash, seed, model/Prompt/Tool/RAG versions;
• Baselines: no AI, read-only Copilot, constrained agent;
• Metrics: the items from this section's metrics dictionary that were actually executed;
• Thresholds: set by scenario SLOs and risk analysis; high-risk execution without approval, cross-tenant privilege escalation, and duplicate device side effects are zero;
• Results: when the manuscript carries no real experiment package, all entries are marked NA — no promotional numbers are pre-filled.

Reproducibility does not mean different environments produce identical microsecond-level results; it means a third party can reconstruct the main conditions, recompute the metrics, explain the differences, and judge whether the conclusions hold within the declared boundaries.

Figure 14-9 Five Steps of a Reproducible Experiment & the Evidence Pack

← 14.1 Overview of the Full Project Lifecycle14.3 Common Pitfalls and Best Practices →