7.4 Multi-Model Support and Private Deployment
7.4.1 Supporting Multiple Large Models: GPT, Claude, DeepSeek, and Qwen
The value of Spring AI is not that it requires every model to expose the same protocol; it is that the ChatModel abstraction shields provider differences, while ChatClient provides a uniform way to invoke them. OpenAI, Anthropic, Ollama, and other implementations can each use their corresponding ChatModel; the business layer still handles conversations through prompt(), call(), stream(), and Tool Calling.
The current IoT DC3 implementation matches this abstraction. dc3_model_provider stores the provider type, base_url, api_key, default flag, enable status, and tenant information; the provider types currently include OPENAI_COMPATIBLE and ANTHROPIC. Specific models and their capability configuration are linked to a provider through dc3_model_config, and ChatClientFactory builds and caches the corresponding client in the way described in Section 7.2.1 — not repeated here.
So the accurate description of switching models is: configure a provider and model first, then let each request choose a model or fall back to the default. As long as the upper layers keep using ChatClient, Tool implementations usually need no rewriting per provider; but each provider's authentication, request options, Tool Calling capabilities, and return behavior still need separate verification — the adapter cannot be described as "change the configuration only, with no differences at all." The current project also has no policy engine that routes models automatically by task complexity or sensitivity; such routing would have to be implemented explicitly later.
| Model or access method | Current access path | Suitable scenarios | To verify |
|---|---|---|---|
| OpenAI-compatible services such as GPT, DeepSeek, and Qwen | OPENAI_COMPATIBLE → OpenAiChatModel | General conversation, Chinese-language operations, tool calling | Endpoint compatibility, model capability, cost and data compliance |
| Claude | ANTHROPIC → AnthropicChatModel | Long context, log and report analysis | Tool Calling, parameter differences, regional compliance |
| Local inference endpoints such as Ollama and vLLM | Configure the matching provider per the actual compatible protocol | Data stays on-site, private-deployment validation | Model format, throughput, GPU memory, context length, and function-calling stability |
Model selection should not rely on marketing parameters. A more reliable approach is to use the same batch of device queries, historical-value analyses, and Tool Calling use cases to measure each candidate model's latency, success rate, parameter accuracy, cost, and resource consumption, and only then decide the default model. Multi-model configuration provides replaceability; it does not mean automatic routing already exists.
7.4.2 Private Deployment Options: Security and Privacy Considerations
An engineer switched models by changing nothing in the configuration file except the endpoint address — and that operation rests on an important premise: a model service must be running locally. Private deployment is not simply "downloading a model file"; it spans four dimensions: model acquisition, inference engine selection, hardware adaptation, and operational management. In IoT scenarios, the drive toward private deployment usually comes from two clear requirements: data sovereignty and controllable latency.
Who Is Asking for Private Deployment
A factory's operations lead put it bluntly: "The device point data is my process recipe — once it leaves the plant, I can't sleep." In industry, energy, and healthcare, device configuration parameters, operating curves, and failure modes are core enterprise assets. Public-cloud LLM services promise transport-layer encryption, but inference happens in the cloud — the text of every request is sent to the model provider's data center. For production environments whose internal networks are not directly connected to the internet, this path simply does not work.
The other driver is inference latency. A cloud model call includes network transit time. When an operator says "close the feed valve of reactor No. 3," if the request must first travel over the internet to the cloud for inference and then return as a command, the extra few hundred milliseconds can stretch to seconds under network jitter. Local deployment keeps inference latency stably below 100 ms, unaffected by carrier network conditions.
Mainstream Options: Ollama, vLLM, and LocalAI
The toolchain for deploying large language models (LLMs) locally is now fairly mature. Three options are the most common in IoT scenarios, each with its own emphasis.
Ollama has the highest level of packaging: a single ollama pull qwen2.5:7b command brings up the service. Its model library is rich, with ready-made images for mainstream model sizes. It suits rapid validation, single-instance, low-concurrency scenarios — for example, a factory that only needs to serve a few operations engineers at a time.
vLLM requires users to pull models from HuggingFace manually and specify the path, so its level of packaging is moderate. Its strengths are production-grade throughput and multi-instance high availability. When you need to serve dozens of operators at once, or expose inference to external agents, vLLM's continuous batching and PagedAttention mechanisms squeeze GPU utilization to the limit.
LocalAI provides an interface fully compatible with the OpenAI API and is more flexible for containerized deployment. It is more tolerant of model formats — a single deployment can load models from different vendors at the same time. It suits scenarios that need to run multiple heterogeneous models on one machine.
All three options provide OpenAI-compatible endpoints, which is exactly the protocol standard Spring AI relies on. For the Agentic Center, switching inference engines only means changing base-url — architecturally no different from switching cloud models. A configuration example:
# application.properties (illustrative)
spring.ai.ollama.base-url=http://localhost:11434
spring.ai.ollama.chat.model=deepseek-r1:7b
# To switch to vLLM or LocalAI, just change this line:
# spring.ai.openai.base-url=http://localhost:8000/v1Hardware Is the Real Constraint
GPU resources are the threshold most teams face. Models of different parameter sizes differ markedly in GPU-memory requirements. Take a typical 7B-parameter model: it runs fine on consumer-grade GPUs, but how fast it actually runs and how long a context sequence it supports depend on quantization precision and sequence length. Larger models — those reaching the tens-of-billions parameter class — demand significantly more GPU memory and system memory. When a model's parameters exceed a single card's capacity, you need multi-card parallelism or CPU offloading — placing some layers in CPU memory and trading inference speed for availability. Both Ollama and vLLM support this technique. In IoT data-query scenarios, a 3–5 second latency per inference is usually acceptable — far better than not being able to deploy at all.
A Hybrid Pattern: Layered Decisions, Not Either-Or
Not every request needs to remain private. A more robust hybrid router decides first from data classification, tool permissions, cost, and measured task quality: sensitive data or low-risk queries may go to an accepted local model, while tasks permitted to leave the site and requiring stronger capabilities enter an approved cloud model. Specific model names and capabilities change, so this book does not bind a brand permanently to "simple" or "complex" tasks. The Agentic Center's dc3_model_provider table supports multiple providers and model selection per session. Automatic routing still requires separate policy, fallback, audit, and evaluation loops rather than one more if:
// Select the model backend by request characteristics
public ChatClient selectModel(ChatRequest request) {
if (request.containsSensitiveTags()) {
return ollamaChatClient; // sensitive data stays local
}
if (request.isSimpleQuery()) {
return ollamaChatClient; // low latency first
}
return openAiChatClient; // complex tasks go to the cloud
}This approach turns what looks like an either-or choice into a decision that can be tuned layer by layer.
Engineering Checklist: Before Starting a Private Deployment
- Confirm the model's parameter size and the estimated GPU-memory requirement, and check them against the server's GPU configuration (refer to the recommended requirements on the model's release page).
- Choose an inference engine: Ollama for rapid validation, vLLM for production throughput, LocalAI for coexisting heterogeneous models.
- Pull the model image and verify that the OpenAI-compatible endpoint works.
- Point the Agentic Center configuration's
base-urlat the local inference service. - Verify the tool-calling chain end to end: send a test message such as "query all offline devices."
- (Optional) Deploy hybrid routing logic to split traffic by query type and sensitivity.
Private deployment is not an all-or-nothing choice. Done right, it lets you find your own balance among data sovereignty, response speed, and model capability.
7.4.3 MLOps and LLMOps: From Version Registration to Production Regression
Deploying a model as an HTTP service solves only the "it can be called" problem. A production system must also answer: which model, which prompt version, which knowledge index, which tools, and which permission policy served the current request; whether quality regressed after the upgrade; and whether a single component can be rolled back when something goes wrong. Traditional MLOps governs data, features, training code, models, and deployments, while LLMOps additionally brings prompts, context, RAG indices, tool schemas, evaluation sets, and security policies into the release unit.
An AI Application Is Not One Model, but a Set of Interdependent Assets
Each release should generate an immutable manifest recording at least:
- model provider, model ID, and service version;
- system prompt, business templates, and their hashes;
- tool names, descriptions, input schemas, risk levels, and backend API versions;
- RAG corpus snapshot, chunker, embedding model, reranker, index, and filtering policy;
- security policies, tenant scope, approval rules, and output-filter version;
- offline evaluation sets, attack sets, and pass thresholds;
- releaser, approver, time, reason for change, and rollback target.
A model version without a tool-schema version can leave a new model calling a new interface with old parameters; an index version without a corpus snapshot cannot explain a knowledge regression; storing prompt text without recording policies makes it impossible to reproduce why the same request produced different tool catalogs under two tenants.
The Boundary Between MLOps and LLMOps
| Dimension | MLOps Focus | LLMOps Additions |
|---|---|---|
| Data | Training/validation data, features, labels | Prompts, conversations, RAG corpora, tool returns, human feedback |
| Assets | Models, training code, feature pipelines | Models, prompts, indices, tool schemas, policies, evaluation sets |
| Evaluation | Accuracy, recall, drift, service metrics | Faithfulness, refusals, trajectories, privilege escalation, cost, non-deterministic variance |
| Release | Model registry, canary rollout, rollback | Independent component versioning, read-only first, tiered autonomy, policy rollback |
| Monitoring | Data/concept drift, prediction quality | Ungrounded answers, tool failures, prompt injection, human rejections, context pollution |
The two are not substitutes. A predictive-maintenance model still needs data splitting, model registration, and drift monitoring; the agent that calls it must additionally govern prompts, tools, and approval policies.
Release Gates: Prove Nothing Breaks First, Then Grant Autonomy Gradually
A sound release process can be divided into five gates:
- Offline regression: run against a versioned golden set, an unanswerable set, and a security attack set;
- Shadow traffic: the new version reads real requests but produces no external side effects, and is compared with the old version;
- Canary tenants: open only to a limited set of tenants, devices, and users;
- Read-only first: open query tools first, then write operations that require confirmation;
- Expand scope: add devices and scenarios only after metrics are stable and the incident drill has passed.
At no stage should the model itself decide whether a release gate passes. Evaluation execution, policy judgment, and approval must sit outside the model.
Online Traces: From Outcomes Back to Versions and Side Effects
Every request should produce a correlatable trace recording the model and prompt versions, retrieved documents and their versions, the tool catalog, a summary of tool parameters, permission decisions, action confirmations, backend receipts, the final answer, tokens, latency, and cost. Sensitive parameters may be redacted or stored as hashes, but the trace must not lose correlatability entirely.
Monitoring should include at least: request success rate and P95 latency, tokens and per-task cost, the RAG rate of ungrounded answers, tool success/timeout/retry rates, human rejection rate, action expiration rate, cross-tenant interceptions, and security-test hits. When business outcomes appear with a delay, device alarms, work orders, and final states should also be linked back to the original trace.
Drift Does Not Happen Only in the Model
- Data drift: changes in device distribution, season, or operating conditions;
- Concept drift: the relationship between a feature and a fault changes;
- Knowledge drift: updates to manuals, firmware, and SOPs;
- Interface drift: changes to tool schemas or backend APIs;
- Policy drift: changes to permissions, approvals, and risk thresholds;
- Behavioral drift: a provider updates its service implementation while the model ID stays the same.
Continuous evaluation therefore must not trigger only on model upgrades. Whenever corpora, tools, policies, or key dependencies change, the corresponding regression sets should run.
Rollback Must Be Designed per Component
Full rollback is often too slow. In engineering terms, prepare separate rollbacks for models, prompts, retrieval configuration, tool schemas, and policies, and support degrading the system from a constrained agent to a Copilot, read-only Q&A, or deterministic rules. After a rollback, traces must remain readable, and an old model must never be paired with new tools.
Asset registration
→ Offline evaluation
→ Shadow traffic
→ Canary tenants / read-only tools
→ Online traces and continuous evaluation
→ Expand scope or roll back per componentThe value of release records is not more process; it is turning "the new version feels better" into an auditable judgment: which component changed, which metrics improved, which risks grew, who approved it, and how to restore the last known-safe combination.