3.6 On-Device AI and Adaptive Sampling
3.6.1 On-Device AI: A TinyML Overview and Deployment Tools
From "Transmit Only, Never Judge" to "Sense and Judge at the Edge"
A workshop has one hundred vibration sensors deployed, and every weekly routine inspection finds three machines whose bearings are worn enough to need replacement. The problem is that a week before those bearings fail, a specific "precursor" pattern appears in their vibration spectra — the early fault signature hides in the noise, and a fixed-threshold trigger simply cannot catch it. The conventional approach is to upload all the vibration data to the cloud for analysis, but each sensor collects thousands of acceleration data points per second; for one hundred sensors the bandwidth bill alone is substantial, and even if the cloud did analyze it, the latency could never keep up with an emergency stop.
The better approach is to have the sensor node itself learn to recognize this frequency pattern, and upload only the segments that "look like a bearing fault." Data volume drops sharply, and response latency falls from seconds to the order of a sampling period. That is the problem TinyML addresses in the sensing layer — fitting a machine-learning inference engine into a microcontroller (MCU) with only tens of KB of RAM, so that it can "understand" its own sensor data.
What Is TinyML
TinyML is short for Tiny Machine Learning. It is not a new family of algorithms; it is a body of engineering techniques for deploying and running machine-learning models on severely resource-constrained MCUs. Typical target hardware is the ARM Cortex-M series (M0/M3/M4/M7), RISC-V cores, and even 8-bit microcontrollers. These chips typically carry only tens to a few hundred KB of SRAM, no more than a few MB of Flash, and run at clocks between tens and a few hundred MHz.
Seen from the IoT sensing layer, TinyML lets a sensor node not only "measure" but also "compute" and "judge." It embeds on-device intelligence directly into the last centimeter closest to the physical world. An intelligent sensor is precisely "a smart data terminal device that integrates a sensor and a microprocessor into one unit, with environmental sensing, data processing, intelligent control, and data communication functions." TinyML is exactly what injects stronger data-processing capability into that "microprocessor" — instead of merely running fixed logic or threshold comparisons, it can perform classification, regression, or anomaly detection from historical data patterns. Mapped against the AIoT architecture discussed in Chapter 1, this corresponds to making the "acquisition" step intelligent: rather than shipping all the data to the cloud first, the node reaches a preliminary verdict while the data is being collected.
Why On-Device AI Is Needed
The reasons can be understood along three dimensions.
Bandwidth and cost. The sensing layer is often the data bottleneck of an IoT system. A mid-sized plant may have thousands of sensor nodes; if every node uploads a complete raw data packet every few seconds, the wireless gateways at the aggregation tier and the cloud storage behind them are quickly overwhelmed. TinyML lets nodes complete feature extraction and preliminary judgment locally, uploading only the business-relevant "events" or "summaries." In a typical wireless sensor network, this means longer battery life and lower transmission costs; the actual compression ratio depends on signal sparsity and model capability.
Latency and reliability. Many protective actions require millisecond-level response — an inspection camera that spots a product defect must trigger the rejection mechanism immediately. Waiting for data to travel to the cloud, complete inference, and return as a command imposes a round-trip latency that usually exceeds 100 ms, by which time the line has already run a dozen more units. On-device inference brings response latency down to the order of a sampling period and does not depend on connection quality. Even when the network is down, the local node keeps running independently. This division of labor — "train in the cloud, infer at the edge, respond on the device" — is already widely used in practice.
Power and privacy. Traditional AI models run on GPUs or cloud servers, drawing tens to hundreds of watts. TinyML inference typically draws at the milliwatt level and can run on a battery for months or even years. At the same time, raw data need not be uploaded at all, which is valuable wherever user privacy is at stake — for example, when detecting occupant activity in a smart building, the node performs the pose judgment locally and uploads only an "occupied/unoccupied" boolean, never streaming video frames off-site, and thus stays clear of data-compliance red lines.
Core Techniques: Quantization and Pruning
Fitting a trained neural-network model onto an MCU with only tens of KB of memory is not a simple copy-and-paste job. Models exported by mainstream deep-learning frameworks (TensorFlow, PyTorch) typically use 32-bit floating-point (float32) weights and activations. A model with 100,000 float32 parameters already occupies about 400 KB of Flash for its weights alone (100K parameters × 4 bytes) — considerable for an MCU with only tens of KB of memory. Two things are needed: quantization and pruning.
Quantization is the most essential compression technique. It maps 32-bit floating-point numbers to 8-bit integers or even 1-bit binary values. After 8-bit quantization (int8), the model shrinks markedly and inference runs visibly faster, while for most classification and regression tasks the accuracy loss stays within engineering-acceptable bounds. More aggressive strategies include mixed precision (some layers kept in float16, others reduced to int8) and quantization-aware training (QAT), the latter bringing the quantized model's accuracy closer to the floating-point baseline.
Pruning removes unimportant connections or neurons from the model outright. After training, neurons whose weight magnitudes are near zero contribute little to the final output and can be cut away safely. Structured pruning can delete whole layers or channels, while unstructured pruning removes only individual connections. After pruning the model is smaller and its compute load lower, and it usually takes a few epochs of fine-tuning to recover accuracy.
The figure below shows the complete TinyML pipeline from training to deployment — the standard lifecycle an engineering team must face.
Closing the Validation Loop: PTQ, QAT, and Hardware Acceptance
Quantization cannot be judged by model file size alone. Post-training quantization (PTQ) estimates numerical ranges from representative calibration data after training is complete; it is cheap and well suited to establishing an INT8 baseline first. If the accuracy loss — or degradation on anomalous samples — is unacceptable, move on to quantization-aware training (QAT), which simulates quantization error during training. Whether FP16, INT8, or even INT4 is faster depends on the target NPU/MCU, operator support, memory bandwidth, and the runtime; a narrower bit width does not automatically mean an end-to-end speedup.
The representative calibration set must cover the real devices, operating conditions, environments, and anomalies — not just ideal samples drawn from the training set. Preprocessing, quantization parameters, and the model should ship as one release unit. After conversion, check in turn:
- whether the model and firmware load, and whether any operators fall back to a slow path;
- accuracy, recall, and false-alarm rate on the full validation set and on key subgroups;
- P50/P95 inference latency, peak RAM/Flash, cold start, and thermal stability under sustained operation;
- energy per inference and per unit of time;
- whether the system can roll back on power loss, model corruption, or a failed OTA update.
Mainstream Deployment Toolchains
Two TinyML toolchains dominate current engineering practice.
| Toolchain | Open-source/Commercial | Typical targets | Core strengths | Main cost |
|---|---|---|---|---|
| TensorFlow Lite for Microcontrollers (TFLM) | Open-source | Full ARM Cortex-M family, ESP32, RISC-V, etc. | Broadest platform coverage, flexible configuration, active community | Heavy manual tuning effort; drivers must be integrated yourself |
| STM32Cube.AI | Commercial | STM32-series MCUs (M4/M7/M55) | Highly automated, deeply integrated with STM32CubeMX, hardware acceleration | Platform lock-in; difficult to migrate across vendors |
TensorFlow Lite for Microcontrollers (TFLM). This is an open-source inference engine maintained by Google's TensorFlow team, with memory optimizations made specifically for MCU scenarios. The official documentation lists hardware such as the ARM Cortex-M0/M3/M4/M7 and ESP32 as validated platforms. TFLM's core achievement is compressing the model interpreter's code footprint down to the tens-of-KB level; it has no operating-system dependency, is implemented in pure C++, and runs directly on bare metal or FreeRTOS. The workflow: train the model in TensorFlow/Keras → quantize and convert with the TFLite Converter → export as a C byte array → embed it into the MCU project. TFLM offers the greatest flexibility and suits projects with strict kernel-compatibility requirements, but its configuration work is comparatively laborious.
STM32Cube.AI. This is STMicroelectronics' commercial tool, deeply bound to the STM32 MCU family. It reads Keras, ONNX, or TensorFlow Lite models and automatically generates C inference code optimized for Cortex-M cores, and it can invoke the hardware accelerators inside STM32 chips (the DSP extensions of the M4 and M7, the Helium vector extension of the M55). Inside the STM32CubeMX integrated development environment, an AI model can be configured directly as a peripheral, listed alongside hardware drivers such as UART and I²C in the same project file. For teams without much model-optimization experience, Cube.AI's automation is far more convenient — at the price of lock-in to the STM32 ecosystem.
Which path to choose depends on project constraints: with a non-STM32 chip, or when experiments need maximum freedom, TFLM is the more general choice; when the team has already settled on STM32 hardware and wants to deliver a prototype quickly, STM32Cube.AI saves a great deal of manual tuning.
Engineering Trade-Offs and Deployment Pitfalls
TinyML is not a cure-all. Its boundaries of applicability are clear: if the task requires understanding complex context (multi-turn dialogue or semantic image segmentation, say), the compute and memory of an MCU fall far short. For such language-understanding tasks, the current compromise is to sink a small language model (SLM) down to the edge gateway: models below the 3-billion-parameter scale, once quantized, can already run on gateway-class hardware, supporting operations scenarios such as equipment-manual Q&A, alarm summarization, and first-pass work-order screening; however, this takes several GB of memory and watt-level power — a gateway-side capability rather than one for sensor nodes, and on a different order of magnitude from TinyML. But for binary classification, a small set of keywords (a wake word and a few control commands), simple anomaly detection, or vibration pattern matching, TinyML is fully up to the task — and far cheaper than running large models in the cloud.
Several common engineering pitfalls deserve attention at deployment time:
- The accuracy of a quantized model must be revalidated on real hardware. The floating-point behavior of a simulator can differ from that of a real chip — especially the way precision loss accumulates on marginal activation values. A quantized model that passes validation on a PC may see its false-alarm rate spike once flashed onto the MCU.
- The preprocessing configuration must exactly match training. Details such as the input normalization parameters, the sliding-window size, and the downsampling ratio are nearly impossible to change after the firmware is flashed. Preprocessing logic should be packaged with the model at the code-design stage, not written into a configuration file on the firmware's outer layer.
- The model-update mechanism needs to be planned up front. When thousands of devices are already deployed in the field, updating the firmware over OTA is the practical approach. It requires the chip to support secure Flash erase/write and rollback protection, and the model file must not exceed the available Flash space.
Example: Deploying a keyword-spotting model on a Cortex-M4
Deploy a keyword-spotting model (recognizing three to five commands such as "power on," "power off," and "stop") on an MCU built around an ARM Cortex-M4 core with typical SRAM and Flash sizes. The trained full-precision model uses a common lightweight network structure. After int8 quantization and moderate pruning, the model is compressed to fit within the microcontroller's Flash, and the SRAM required at inference time (model weights plus intermediate activations) stays well below the typically available RAM. The power drawn by the whole inference process (sensor acquisition plus MCU computation) is low enough to sustain long-term battery-powered operation. This scenario shows how TinyML lets a resource-constrained sensor node "understand" spoken commands — with no need to upload the audio stream to the cloud at all.
TinyML is turning the "nerve endings" of the sensing layer from plain sensors into miniature brains with a basic capacity for judgment. The next section discusses another engineering strategy for cutting uplink data volume — adaptive sampling. The two are complementary: TinyML governs "whether to act" and "why to act," while adaptive sampling governs "how often to act." Combined, an edge node can sense and report at the required precision only when a meaningful, relevant event occurs.
3.6.2 Adaptive Sampling: Dynamically Adjusting the Data Acquisition Frequency
Fixed-frequency sampling carries a fundamental engineering contradiction: during quiet periods most of the sampling and bandwidth spent is wasted, yet when an anomaly occurs the cadence is too slow, and the critical information falls precisely into the gaps between samples. Adaptive sampling lets the sensor adjust its acquisition and reporting frequency dynamically according to how "interesting" the data is — saving power and bandwidth when calm, accelerating automatically when anomalous. It does not require every node to run a TinyML model, but it comes from the same lineage of thought as on-device AI: make decisions in the sensing layer, and cut ineffective transmission.
Three Basic Strategies
Event-driven sampling: the sensor normally sits in a low-power sleep, keeping only an ultra-low-power wake circuit alive to detect predefined events. What distinguishes it from an ordinary interrupt wake-up is the elementary logic added before the verdict — for example, an accelerometer declares a "suspected mechanical fault" only after detecting threshold-exceeding vibration several times in a row, and only then starts high-speed sampling. Sleep-period power can drop to an extremely low level (microamp-level values), but it barely reacts to slowly developing faults and easily misses them.
Deadband sampling: the sensor continuously monitors the rate of change of the physical quantity; when the change rate stays within a preset deadband it cuts the sampling frequency sharply, and when it exceeds the deadband it returns to full speed or even speeds up. In a concrete implementation, the sensor maintains a sliding window and computes the deviation between the current value and the window mean: if the deviation is smaller than the deadband, the next sample is skipped; if it exceeds the deadband, the sensor immediately takes a make-up sample and extends the observation window. Setting the deadband width relies on offline data analysis — too wide and slow changes are lost, too narrow and hardly any RF energy is saved.
Predictive-model sampling: deploy a lightweight autoregressive model (such as AR(1)) or a shallow decision tree that predicts the next value from the most recent samples. A small prediction residual means the environment is in steady state, and the sampling frequency can be lowered; a residual that suddenly grows means something new has happened that the model does not cover, and the sensor immediately enters high-rate mode. This approach uses prediction error to measure how "novel" the data is, and can catch early precursors that neither fixed thresholds nor change-rate rules recognize — at the cost of investing in model training and deployment processes.
A Hybrid-Strategy State Machine
In real engineering a single strategy is rarely used alone; the more common pattern is to package event-driven wake-up, the deadband criterion, and the predictive model into a finite state machine, with state transitions driven by consecutive growth in the model's prediction error. The following is the three-state switching logic of a vibration sensor (Figure 3-12). In the steady-state low-power state, the sensor samples at long intervals and performs only simple frequency-band energy computation and model prediction; once the model error grows to several times the baseline threshold in a row, it switches immediately into an accelerated-listening mode, sampling at a higher frequency but not uploading; if the residual stays above the threshold for several rounds, a fault is confirmed and the accumulated raw waveform is uploaded. On upload completion, the sensor resets to the steady state. All three transition conditions are illustrative values; real projects must recalibrate them against the frequency range and noise floor of the vibration signal. In the steady state the radio is completely off, and only the MCU runs model prediction at a low clock; the baseline threshold must be calibrated from offline data, typically set to a multiple of the maximum residual under normal operating conditions.
Example: Adaptive Sampling on a Vibration Sensor
Consider this scenario: wireless vibration sensors mounted on industrial rotating machinery, whose battery capacity must keep the maintenance interval no shorter than a target value. Under normal operating conditions the vibration amplitude is stable; when a bearing starts to wear early, high-frequency noise appears but the amplitude increment is tiny — a fixed-threshold trigger cannot perceive it at all, whereas the adaptive-sampling prediction model notices the change as the error grows in succession. For the great majority of the year the sensor stays in the steady-state low-power state, and battery life extends markedly compared with a fixed high-frequency sampling scheme, meeting the maintenance-interval requirement. More important, the consecutive growth of the model error reliably captures the transition window from stable to faulty — the same design lineage as the TinyML vibration-precursor recognition in Section 3.6.1, except that a much simpler statistical model replaces the neural network.
Engineering Implementation: Hybrid-Strategy Pseudocode
The following is an implementation skeleton of a hybrid strategy based on change rate and an AR(1) model. The sampling intervals, deadband, and error threshold are all illustrative values; an actual deployment must recalibrate them against signal characteristics and battery capacity. In a real product, predict_next_value can be replaced by the TinyML model mentioned in Section 3.6.1.
#define WINDOW_SIZE 10 // illustrative window size
#define DEADBAND 0.5f // rate-of-change deadband (illustrative value)
#define MODEL_ERROR_THRESH 2.0f // prediction residual threshold (illustrative value)
#define HIGH_FREQ_INTERVAL_MS 1000
#define LOW_FREQ_INTERVAL_MS 10000
static float sample_window[WINDOW_SIZE];
static int window_index = 0;
static int consecutive_model_error = 0;
static int current_interval = LOW_FREQ_INTERVAL_MS;
float compute_rate_of_change() {
float sum = 0;
for (int i = 0; i < WINDOW_SIZE; i++) sum += sample_window[i];
float mean = sum / WINDOW_SIZE;
return fabs(sample_window[(window_index - 1 + WINDOW_SIZE) % WINDOW_SIZE] - mean);
}
float predict_next_value() {
// AR(1) model: use the most recent sample value directly (illustrative)
return sample_window[(window_index - 1 + WINDOW_SIZE) % WINDOW_SIZE];
}
void sample_and_decide() {
float current = read_adc();
float rate = compute_rate_of_change();
float residual = fabs(current - predict_next_value());
sample_window[window_index] = current;
window_index = (window_index + 1) % WINDOW_SIZE;
if (rate > DEADBAND || residual > MODEL_ERROR_THRESH) {
consecutive_model_error++;
if (consecutive_model_error >= 2 && current_interval != HIGH_FREQ_INTERVAL_MS) {
current_interval = HIGH_FREQ_INTERVAL_MS;
trigger_high_frequency_mode();
}
} else {
consecutive_model_error = 0;
if (current_interval != LOW_FREQ_INTERVAL_MS) {
current_interval = LOW_FREQ_INTERVAL_MS;
trigger_low_frequency_mode();
}
}
if (consecutive_model_error >= 5) {
upload_buffer_to_edge();
consecutive_model_error = 0;
}
}Engineering Trade-Offs: Latency, Energy, and Missed-Detection Rate
Choosing a sampling strategy means trading among several conflicting indicators. Table 3-4 is a qualitative comparison; the actual magnitudes vary widely with hardware and operating conditions.
Table 3-4 A qualitative comparison of adaptive sampling strategies
| Indicator | Event-driven | Deadband | Predictive model | Hybrid strategy |
|---|---|---|---|---|
| Response latency | Very low (interrupt-level) | Medium (deadband-dependent) | Higher (error must accumulate) | Adjustable |
| Energy saving | Very high | Medium-high | High (RF sleep gains offset compute overhead) | Fairly high |
| Missed-detection rate | High (slow changes) | Medium | Low | Low |
| Implementation complexity | Low | Low | High (model training required) | Medium-high |
From a coverage standpoint, the hybrid strategy balances the needs of different scenarios: critical paths use "event-driven + deadband" to guarantee low latency, while secondary paths use the "predictive model" to catch slowly changing signals, maximizing battery life. One easily overlooked engineering detail: the first sample after waking from deep sleep may carry ADC settling error and should be discarded; the change-rate window size must be set from the signal's characteristic frequency — for mains-frequency vibration, a window sized to the sample count of a complete period covers exactly one cycle; and a newly deployed predictive model should run in a "full-rate sampling + model learning" mode, entering the adaptive phase only after enough samples have accumulated. The platform side should maintain a "sampling-frequency trajectory" field for each device, so the data integrity of downsampled periods can be analyzed after the fact, and so offline model recalibration can be combined with the historical-data archiving strategy of Chapter 5.