10.3 Industrial Time-Series Data and Rule Engines
10.3.1 Time-Series Database Selection and the Data Model
Once data has converged from the edge gateways into the platform layer, the first question to settle is: what do we store it in?
Data streams in industrial settings have a temperament of their own. A CNC machine tool's vibration sensors report thousands of sample points per second, and a hundred-odd temperature probes on a production line each report one point every two seconds; taken together and counted by the year, write pressure easily runs past ten million or even a hundred million points per day. More important, these values natively carry timestamps — the defining characteristic of time-series data.
Relational databases and dedicated time-series databases each have their own boundaries. With partitioning, batch writes, appropriate indexes, and extensions, PostgreSQL can also carry large time-series workloads. A dedicated TSDB may offer more direct capabilities for compression, retention, and time-based aggregation. Whether either choice is "not cost-effective" can be determined only by benchmarks under the target write, query, retention, transaction, and operations conditions; a product category is not itself a performance conclusion.
Core Characteristics of Time-Series Databases
The purpose-built design of time-series databases for industrial data streams can be summarized in four points: LSM-Tree (Log-Structured Merge-Tree) style structures convert random writes into sequential appends, buying high write throughput; partitions are split automatically by time window, so queries scan only the relevant partitions; downsampling and aggregate computation are pushed down into the storage layer for execution; and partitions are expired and cleaned up automatically according to a retention policy. The engine-level principles behind these mechanisms — the write path, compression encoding, continuous aggregation, and hot/cold tiering — were dissected one by one in Section 5.4 of Chapter 5; this section will not repeat them and answers only the question industrial projects agonize over more often: which specific product to pick.
Selecting Among Mainstream Time-Series Databases
The choice facing an industrial IoT platform is not "whether to use a time-series database" but "which one". The mainstream products differ in where their capabilities end in industrial scenarios.
Table 10-4: Feature comparison of mainstream industrial time-series databases
| Feature dimension | InfluxDB (1.x / 3.x) | TimescaleDB | TDengine |
|---|---|---|---|
| Architecture type | Standalone TSDB engine (self-developed storage) | PostgreSQL extension | Standalone TSDB engine (self-developed storage) |
| Data model | Measurement + tags + fields | Hypertable + columns | Supertable + tags + columns |
| Write performance | Depends on version, schema, batching, hardware, and durability settings; benchmark it | Depends on PostgreSQL configuration, partitioning, indexes, and batching; benchmark it | Depends on version, table model, hardware, and replica settings; benchmark it |
| SQL compatibility | Custom InfluxQL/Flux | Full PostgreSQL SQL | SQL-like (limited Join/window-function support) |
| Clustering and high availability | 1.x open-source edition has no clustering; 3.x supports clusters | Based on PG streaming replication; must be built yourself | Supported in the enterprise edition; no native clustering in the open-source edition |
| Applicable scenarios | Small-to-medium monitoring, operations monitoring, IoT platforms | Production lines needing complex SQL analysis and integration with the PG ecosystem | Large-scale industrial point collections demanding high throughput and high compression |
There is no absolutely right answer. One note first: InfluxDB 2.x (the release that introduced Flux and the TSM storage rework) is treated as a transitional version in the official roadmap — the current main lines are 1.x and 3.x, which is why the table compares only those two series. If a team already leans heavily on PostGIS and complex business queries, TimescaleDB reuses the existing SQL skill stack; if the scenario is simply "sensors write → monitoring reads → alarms", InfluxDB is the lighter option; if annual data volume runs into billions of points and high compression is required, TDengine's columnar storage option is worth evaluating.
IoT DC3 was not designed around any single time-series database; instead, its data center layer abstracts the storage interface, allowing the underlying time-series storage engine (TimescaleDB, TDengine, and so on) to be switched as needed in production.
Point and Tag Design: The Key to the Data Model
The power of a time-series database depends not only on the storage engine but even more on a sensibly designed data model. In IoT DC3 practice, one time-series record is modeled as a PointValue — each value carries five fixed attributes:
- device_id (device ID): links to the physical device instance.
- point_id (point ID): uniquely identifies a sensor or register address.
- value (numeric value/state): the actual engineering value after normalization.
- event_time (acquisition timestamp): the time stamped at the device or the gateway.
- unit (unit): the unit context (such as °C, kPa, rpm), used for semantic interpretation.
Beyond these, tags are optional dimension fields that support multi-dimensional queries — for example, retrieving all temperature points related to a given process step with "line = Line A AND step = welding".
-- Illustrative: IoT DC3 time-series table structure based on TimescaleDB
CREATE TABLE point_value (
device_id VARCHAR(64) NOT NULL,
point_id VARCHAR(64) NOT NULL,
event_time TIMESTAMPTZ NOT NULL,
value DOUBLE PRECISION NOT NULL,
unit VARCHAR(16),
quality SMALLINT DEFAULT 1, -- 0=bad, 1=normal
-- Optional: tags column (predefined via the thing model)
tags JSONB DEFAULT '{}'::jsonb,
PRIMARY KEY (device_id, point_id, event_time)
);
-- Partition by device and time (Hypertable)
SELECT create_hypertable('point_value', 'event_time', chunk_time_interval => INTERVAL '1 day');
-- Add a space dimension for device-id-based partitioning
SELECT add_dimension('point_value', 'device_id', number_partitions => 16);Two pitfalls are easiest to fall into at the data-model design stage.
First, tag-cardinality explosion. Attaching a large set of tags — "line, process step, device model, manufacturer, batch number" — to every single record buys query flexibility, but it can inflate the time-series database's inverted index beyond control. On one industrial line, several hundred points each carrying six or seven tags can leave the index several times the size of the data itself. Keep the primary-dimension tags to three to five, and resolve the remaining dimensions through foreign keys into metadata tables; do not stuff everything into the time-series table.
Second, time partitioning that does not distinguish primary from secondary data. Vibration and temperature samples from the same device can differ in sampling frequency by two orders of magnitude. Forcing both into one uniform time partition wastes serious storage on the low-frequency data. The better approach is to split tables or partition keys by point type: high-frequency vibration goes to short windows (partitioned hourly, say), low-frequency temperature to long windows (grouped daily).
The choice of data model also directly determines the consumption cost of the downstream rule engine and AI models. A good model has already settled the division of labor on the device-access side — "tags for filtering, value for computing, time for alignment" — while a bad model pushes all the trouble onto the data-processing layer, sharply increasing query complexity and adding further system latency.
When designing a time-series data model, work through a checklist item by item:
- Is every point_id defined with explicit semantics in the thing model (physical meaning + data type + unit)?
- Have the cardinality and possible values of the tags been assessed in advance?
- Is the partitioning strategy split according to sampling-frequency differences?
- How is the retention policy set — how long is raw data kept, and how is downsampling executed?
- When does write concurrency peak, and has the peak write rate been verified by load testing?
This section has stayed at the data-model level. With clean, queryable time-series data in place, the next step is to set the data in motion — consumed by the rule engine, triggering alarms or automated decisions. That is exactly what Section 10.3.2 unfolds.
10.3.2 Rule Engine Principles and Industrial Alarm Design
The time-series database persists the data, solving the problem of "storing it at all". But the real value in industrial scenarios lies in "reacting fast": a device temperature crossing a threshold must raise an alarm immediately, a run of abnormal vibration values must trigger the shutdown sequence, and joint multi-parameter judgment must weigh temperature, pressure, and current together in one rule. If this layer of logic is hard-coded in application code, changing a single threshold requires a redeployment — unacceptable. That is precisely the value of the rule engine: it pulls "evaluate — act" out of business code and turns it into a configurable, hot-updatable rule set.
Event-Driven Processing and Condition Evaluation
The input to industrial alarming is typically a stream of time-series point data. The rule engine runs in an event-driven fashion: every newly reported point value is pushed into the engine's inferencing working memory as an event. The engine uses a refined Rete algorithm for efficient pattern matching — it compiles rule conditions into a network structure and matches incrementally, avoiding a full recomputation over all facts on every trigger. Rete's advantage shows most clearly once the rule count passes a hundred; with only a few dozen rules, a simple linear scan is acceptable, and there is no need to over-engineer the selection.
Taking the IoT DC3 platform as an example, the rule engine module receives PointValues from the data center (normalized point values carrying semantic tags, units, and timestamps). Engineers write rules in the rule center, such as "Motor 1 bearing temperature > 85 °C sustained for more than 10 seconds". Each time the rule engine receives a temperature point value, it begins condition evaluation and triggers the action when the window closes.
The following rule-definition fragment shows the configuration of condition evaluation and action execution:
{
"ruleId": "bearing-temp-high-001",
"name": "Motor1 bearing temperature too high",
"description": "Detects motor1 bearing temperature staying above 85°C for 10 seconds",
"priority": 10,
"condition": {
"type": "continuous",
"measurement": "temperature",
"deviceId": "motor-01",
"pointId": "bearing-temp",
"operator": ">",
"threshold": 85,
"durationSeconds": 10
},
"action": {
"type": "alarm",
"severity": "critical",
"notify": ["sms", "email"],
"hookUrl": "http://alert-service/api/v1/alarms"
},
"enabled": true
}The semantics of this configuration: when the bearing-temp point of device motor-01 stays above 85 for 10 seconds, an alarm with severity level critical is triggered, notification goes out by SMS and email, and the REST interface of the external alarm service is called. The rule weight priority:10 determines its execution priority within the conflict set — the higher the value, the earlier it executes. Note that this is an engineering example; rule definitions in an actual production environment will vary with the platform and protocol, but the core structure is similar.
Rule Priority and Conflict Resolution
When multiple rules match at the same time (a temperature-over-limit alarm and a vibration-anomaly alarm triggering together, for example), the engine must decide which one to execute first. Mainstream rule engines such as Drools place the candidate items whose conditions are satisfied on an Agenda and order their execution by a conflict resolution strategy; the default ordering turns mainly on two criteria:
- Salience: engineers explicitly assign each rule an integer value; the higher the value, the higher the execution priority. This is the most commonly used mechanism. Emergency alarm rules are usually assigned high values to ensure they execute before non-emergency rules. When unspecified, the default is 0.
- Activation recency: when salience ties, the rule activated most recently executes first (like the last-in-first-out of a stack). For industrial alarming this is a reasonable default — when the same rule is triggered repeatedly, the activation carrying the newest facts gets handled first.
- Agenda groups: rules are sorted into groups, and the engine executes them in group order. This suits scenarios divided by process stage — running the "data quality check" group first, then the "condition judgment" group, for example. Within a group, salience still does the ordering.
One widespread misreading deserves correction: "by default the engine activates only the rule with the more specific condition" is an optional strategy (specificity) in engines such as CLIPS, not Drools's default behavior — Drools's default is salience plus activation recency. So two rules with overlapping conditions (for example, temperature > 90 and temperature > 85 both satisfied) will by default both be activated and executed in sequence, and eliminating duplicate notifications is up to the engineer: the usual moves are to let the specific rule override the general one with a higher salience, or to rely on an alarm-suppression window to merge alarms from the same source (see later in this section).
A common engineering trap is over-reliance on salience without grouping, which leaves the ordering in disarray as the rule count grows. Once the rule count passes 50, introduce agenda groups split along business stages (data quality → condition judgment → alarm generation → work-order creation), and keep each group to no more than 10 rules.
Alarm Severity Levels and Notification Channels
On the factory floor, an alarm is not a single event — it is an operational flow that escalates level by level. Three severity levels are generally used (an engineering convention, not a standards mandate):
- Info: the threshold is being approached but not yet exceeded. Notification: log records and monitoring-dashboard labels; no active push.
- Warning: the threshold is exceeded but still within the safety boundary, and the device can keep running. Notification: the work-order system, email, a flashing dashboard.
- Critical: the threshold is exceeded and device safety is affected, or a cascading line stop may follow. Notification: SMS, voice-call alarms, or an automatic shutdown command from the MES.
The choice of notification channel depends on the response-time requirement. A reasonable tiered structure is as follows:
| Alarm level | Response-time requirement | Recommended channels | Work order required |
|---|---|---|---|
| Critical | Within minutes | SMS + phone + MES interface | Yes |
| Warning | Within hours | Email + dashboard | Yes |
| Info | Routine inspection | Dashboard + logs | No |
Splitting channels is not for "feature richness"; it is to reduce operational noise. The result of pushing every threshold violation once by SMS is that operations staff go numb to SMS and miss the genuine emergencies. The pragmatic engineering judgment is to let Info-level rules dominate in number while Critical rules are kept under strict control, to avoid alarm fatigue. At the same time, alarm suppression should be set: the same alarm type on the same device fires only once within a configured time window (30 minutes, for example), unless the situation escalates.
Rule Engine State Transitions
A running rule engine does not have only the two states "activated — executed". A properly designed rule engine should support the following state transitions: a rule is created in DRAFT (draft), moves into ENABLED (active) by manual enabling, enters MATCHED (matched) upon receiving a matching event, becomes EXECUTED (executed) once the engine selects it, and, after execution and a fact update, resets back to ENABLED. A rule can also be moved manually from ENABLED or DRAFT into DISABLED (disabled), and finally into DELETED (deleted). Note that the MATCHED/EXECUTED pair of runtime states is a state model custom to the IoT DC3 rule center, used to describe the rule life cycle in this book's examples — not the standard semantics of general-purpose rule engines such as Drools, where the corresponding concepts are the Activation on the agenda and its Fire. The core value of this state-machine design is hot updates: a rule can move from DRAFT to ENABLED, and recover from DISABLED, without restarting the service. Modifying alarm thresholds while the production line keeps running is exactly the hard requirement that industrial scenarios place on a rule engine. One caution for real deployments: the transition from ENABLED to MATCHED depends on the facts in working memory — if historical data has not been cleared, a newly added rule may instantly match stale facts and raise a false alarm. When enabling a rule, therefore, clear the device's old facts, or attach a time constraint such as timestamp > now - 5s to the rule condition.
The Rule-to-Model Transition Boundary
Rule engines excel at explicit, enumerable condition checks. But when the judgment shifts from "temperature > 85" to something that depends on vibration-spectrum features and pattern recognition against historical fault modes, rule configuration is no longer adequate — thresholds turn fuzzy, and the judgment depends on historical data and feature extraction. At that point the rule engine should be treated as a trigger layer, with analytical reasoning handed to a trained AI model: on detecting a basic feature (an RMS value above the baseline, for instance), the rule engine calls a REST interface to pass the feature data to an inference service; the service returns a fault probability, and the rule engine generates an alarm of the corresponding level from a probability threshold. Section 10.4 will unfold this "rules + model" hybrid chain.
Before deploying a rule engine, walk through the alarm scenarios of every device type on the line and sort them with the following checklist: "which suit hard-coded thresholds, which need time windows, and which must lean on historical data". Once sorted, most scenarios fall within the rule engine's reach, and the remainder is left for model integration. This division rests on engineering experience — it guides task splitting, not precise statistics.
Rule engine engineering checklist (must verify before production-line deployment):
- [ ] Does every rule have an explicit priority (Salience) and group (Agenda Group) set?
- [ ] Do the notification channels of each alarm level match the response-time requirements, and is there over-pushing?
- [ ] Is alarm suppression configured: the same alarm type on the same device fires only once within the set window?
- [ ] Has rule hot-updating been tested (after switching from DRAFT to ENABLED, are old facts cleared)?
- [ ] Rule execution performance: have the rule-count ceiling and the Rete network depth been stress-tested in a development environment?
- [ ] Is a REST interface reserved for the model layer, so fixed thresholds can later be upgraded to probabilistic judgment?
10.3.3 Data Quality and Outlier Handling
In the "sense-judge" chain formed by time-series data and the rule engine, input quality determines output quality. Data acquisition on the factory floor is not an ideal environment: sensor aging, communication interference, PLC buffer overflow, and gateway disconnection all produce missing values, glitches, and duplicates in the data. Fed unprocessed into a rule engine or AI model, such problems mostly end in false or missed alarms — and are hard to trace afterwards.
But the first step of industrial data-quality governance is not "cleaning"; it is marking. In platforms such as IoT DC3, every point value carries a timestamp and a status field (such as the quality flag), which distinguishes "normal", "suspect", and "bad" values. Cleaning strategies should act on marked data, not blindly modify the raw records.
Missing-Data Handling
Missing industrial time-series data may result from sensor failure, network interruption, shutdown, or changes to the acquisition task. Determine the cause first, then decide whether interpolation is appropriate. A count of consecutive points is not a universal threshold: the same three missing points mean entirely different things for millisecond-scale vibration and hourly tank-temperature data. Forward fill and linear interpolation may generate derived series for analysis only. The original gaps, quality codes, method, and maximum interpolation duration must be preserved; control, safety interlocks, and incident forensics must never present interpolated values as measurements.
Glitch Filtering
A glitch shows up as a single point, or a few consecutive points, deviating sharply from the normal range — commonly called a "spike". The common engineering filter is median-based over a sliding window: set the window length (5 points, say), compute the median inside the window, and judge the current value a glitch if its absolute deviation from the median exceeds a preset threshold (three times the standard deviation of normal operation, for instance). The replacement value can be the median or the window mean. Threshold setting must take the device's operating condition into account: sharp swings during a normal start or stop must not be treated as glitches.
Duplicate Removal
Duplicates are usually caused by redundant reporting from the gateway or the protocol. The simplest approach uses device ID plus timestamp as a unique key and makes the receiving end idempotent. Time-series databases themselves usually support timestamp-based deduplication, but a conflict-resolution strategy must be designed: if two records share a timestamp but differ in value, the two common options are to keep the record with the newest timestamp, or to mark it as "conflicted" and leave it to human judgment.
Reading the Code Example
The following Python cleaning code shows the basic operations of missing-value fill, glitch filtering, and deduplication. The abs_dev in the code is the absolute deviation of the current value from the sliding median; note that it is not the standard MAD of statistics (median absolute deviation, defined as median(|x−median(x)|), which takes a second median over the whole window) — the standard MAD is more robust but must be computed per window, and this illustrative implementation takes the lighter compromise. In production this logic generally sits in the edge gateway or the platform's preprocessing stage, and its thresholds must be fine-tuned against the device's process parameters.
import pandas as pd
import numpy as np
# Assume df is a temperature series with column 'value' and a timestamp index
# Step 1: Generate an analysis copy only; validate limit against process dynamics and the sampling interval
df['value_filled'] = df['value'].ffill(limit=validated_gap_limit)
df['is_imputed'] = df['value'].isna() & df['value_filled'].notna()
# Step 2: Median-based sliding-window glitch filtering (window=5)
window = 5
df['median'] = df['value_filled'].rolling(window, center=True).median()
df['abs_dev'] = np.abs(df['value_filled'] - df['median'])
# Three times the mean absolute deviation from the sliding median is used as an illustrative threshold; calibrate it against actual operating conditions
threshold = 3 * df['abs_dev'].rolling(window, center=True).mean()
mask = df['abs_dev'] > threshold
df['value_clean'] = np.where(mask, df['median'], df['value_filled'])
# Step 3: De-duplicate by timestamp (keep the first value; suits most frame-based reporting scenarios)
df = df[~df.index.duplicated(keep='first')]These cleaning strategies cannot solve every problem. When data quality stays persistently low, investigate the device or communication link before relying on algorithmic patching. A project should define quantifiable quality metrics together with their calculation rules and owners. Cleaned and quality-labeled data can be queried by a diagnostic Agent through a controlled Tool. MCP only exposes the Tool; it neither issues device commands on behalf of the platform nor guarantees that the model's judgment is correct.