A few years ago, running a multi-billion-parameter AI model required a data center with a large GPU cluster. Today, models with 7-billion-parameters run on a mobile phone.

One of the shifts that has made this possible is quantization. Compress a large model, trade a little accuracy for dramatically less compute, and you get intelligence that runs locally on the device without a cloud round-trip. This is the foundation of what is being called the AI of Things, or AIoT. And while it sounds like a hardware story, it has immediate implications for the tools your organization buys, the systems you depend on, and the AI architecture decisions you are making right now.

Why regulated industries should pay close attention

Cloud-first AI has always carried a tension for industries that operate in the field or under regulatory constraints.

A construction site with intermittent connectivity cannot depend on a cloud round-trip for real-time monitoring. A field agent taking measurements and running an AI-assisted scope estimate on-device, syncing back to the office once connectivity returns then completes a full productive workflow without the internet ever being load-bearing. We are seeing this pattern emerge in real engagements, and it changes what a field visit actually produces. An industrial technician in a restricted facility cannot route operational data through a third-party inference endpoint. A life sciences organization handling clinical data has regulatory requirements, including HIPAA and 21 CFR Part 11, that make cloud-hosted AI legally complicated and operationally heavy with BAA and audit requirements.

Local inference collapses the chain. When the model runs on the device, the data does not leave. There is no third-party cloud provider in the mix. For organizations navigating these constraints, AIoT is not a future consideration. It is a present one.

What to ask your vendors right now

Most enterprise buyers are not yet thinking about on-device inference as an evaluation criterion. They should be. A few questions worth bringing into your next procurement conversation:

• Where does inference happen: on the device, on-premise, or in a third-party cloud?

• What functionality degrades when network connectivity is unavailable?

• Who has access to the data generated during inference, and where is it stored?

• Is the AI component modular, or tightly coupled to a proprietary cloud platform?

Vendors with clear answers are thinking about this. Vendors who cannot answer clearly are not.

There is also a cost argument worth making explicitly. A large construction or industrial operation with dozens of field workers running AI queries throughout the day is looking at cloud inference costs that are hard to predict and easy to exceed. On-device inference has no per-query charge once the model is deployed. For ops leaders thinking about scaling AI access to frontline workers, that math gets compelling fast.

The architecture decisions that matter now

You do not need to be deploying on-device models today to position yourself well for when they arrive. Avoid over-centralizing data in platforms with limited portability. Know where inference is happening across your stack. Build for modularity so your AI components can be swapped as better options emerge. And start paying attention to the AI architecture baked into the equipment and systems you procure, because those procurement cycles are long and those decisions are hard to reverse.

Thinking things

IoT gave us connected things. AIoT gives us thinking things.

The organizations that will be well-positioned are the ones asking better questions of their vendors and making deliberate architecture decisions now, not after the transition is obvious. If that work is in front of you, we would be glad to help you think it through.