The default assumption in enterprise AI has been that intelligence lives in the cloud. You send data up, the model processes it, and you get results back. This architecture works well for many applications, but it has fundamental limitations that are becoming more apparent as AI moves from experimental projects into critical business operations.

Latency is one problem. Round-trip time to a cloud API adds 50-500 milliseconds depending on location and load. For many applications this is fine. For real-time decision making in manufacturing, autonomous systems, or interactive user experiences, it’s not.

Connectivity is another. Cloud-dependent AI stops working when the internet connection drops. For applications in remote locations, mobile environments, or critical infrastructure, this dependency is unacceptable.

Cost at scale is the third. Every API call costs money. When you’re running AI inference millions of times per day, the API costs become a significant and growing expense line.

Edge computing — running AI models on local hardware close to where the data is generated — addresses all three of these problems. And the hardware and software required to do it practically has improved dramatically in the past 18 months.

What’s Changed in the Hardware

The critical enabler for edge AI is purpose-built inference hardware that’s small enough, cheap enough, and power-efficient enough to deploy outside of data centres.

NVIDIA’s Jetson platform has been the dominant edge AI hardware for several years. The latest Jetson Orin modules deliver AI inference performance that would have required a full server rack five years ago, in a module roughly the size of a credit card. The Jetson Orin Nano, aimed at cost-sensitive deployments, delivers roughly 40 TOPS (tera operations per second) while consuming under 15 watts.

Apple’s M-series chips have brought powerful neural engine capabilities to laptops and desktops. A current MacBook Pro can run a 7-billion parameter language model locally at usable speed. This wasn’t possible on consumer hardware two years ago.

Qualcomm’s AI Engine and Intel’s NPU (neural processing unit) integrated into their latest processors are bringing AI acceleration to standard business hardware. This means that ordinary laptops and workstations can run modest AI models without dedicated AI hardware.

For enterprises, this hardware evolution means that edge AI deployment no longer requires exotic, expensive equipment. In many cases, existing business hardware already has the capability to run useful AI models locally.

Where Edge AI Makes Business Sense

Manufacturing Quality Inspection

Visual quality inspection is one of the most compelling edge AI applications. Cameras on a production line capture images of every product, and an AI model running on local hardware identifies defects in real time.

The speed requirement makes cloud processing impractical for high-throughput lines. A bottling line running 1,000 bottles per minute needs inspection results in milliseconds, not the hundreds of milliseconds that a cloud round-trip requires.

The volume makes cloud processing expensive. A single inspection camera generating 30 frames per second produces 2.6 million images per day. Sending all of those to a cloud API for analysis would cost far more than running the model on a $500 edge device.

Retail Analytics

In-store analytics — foot traffic counting, queue length monitoring, heat mapping of customer movement — are natural edge AI applications. The cameras are already in the store, the processing can run on local hardware, and there’s no need to send video streams to the cloud.

Privacy considerations make edge processing particularly attractive for retail. Analysing video locally and extracting only aggregate statistics (customer count, average dwell time) means that no identifying imagery needs to leave the premises.

Remote and Mobile Operations

Mining operations, agricultural machinery, offshore platforms, and field service operations all involve locations where internet connectivity is unreliable or unavailable. Edge AI enables these operations to run AI-powered monitoring, diagnostics, and decision support without depending on connectivity.

A haul truck running autonomous guidance needs its AI to work regardless of network conditions. A remote weather station analysing satellite imagery for crop health needs to function independently. Edge deployment makes these applications feasible.

The Software Stack for Edge AI

Running AI models on edge hardware requires a different software approach than cloud deployment.

Model Optimisation

Large models designed for cloud deployment don’t run efficiently on edge hardware. They need to be compressed, quantised, and optimised for the specific target hardware. Techniques like INT8 quantisation, pruning, and knowledge distillation can reduce model size by 4-10x while retaining most of the accuracy.

Frameworks like TensorRT (NVIDIA), Core ML (Apple), and ONNX Runtime provide tools for optimising models for specific hardware targets. The optimisation process is increasingly automated, but understanding the trade-offs between model size, speed, and accuracy still requires expertise.

Model Management

Deploying AI models to hundreds or thousands of edge devices creates a management challenge that doesn’t exist with centralised cloud models. Each device needs model updates, performance monitoring, and health checking.

Edge AI management platforms handle the deployment, versioning, and monitoring of models across distributed fleets of edge devices. This is critical infrastructure for any edge AI deployment at scale.

Hybrid Architectures

In practice, most edge AI deployments aren’t purely edge or purely cloud — they’re hybrid. The edge device handles real-time inference locally, while the cloud handles model training, model updates, aggregate analytics, and exception handling for cases where the edge model is uncertain.

This hybrid architecture gives you the latency and reliability benefits of edge processing with the compute power and data aggregation benefits of the cloud.

Practical Advice for Getting Started

If you’re considering edge AI for your business, start with a clear understanding of why cloud processing isn’t sufficient for your use case. If cloud latency, connectivity, cost, or data privacy are genuine constraints, edge AI is worth investigating. If cloud processing works fine for your application, the additional complexity of edge deployment isn’t justified.

Identify a specific, contained use case for your first edge AI project. Quality inspection on a single production line, analytics at a single location, or monitoring on a specific piece of equipment. Don’t try to deploy edge AI across your entire operation at once.

Choose hardware that matches your performance requirements without massive over-specification. You don’t need an enterprise GPU for a model that runs comfortably on a $200 edge module.

Invest in the management infrastructure from the start. Even a small edge deployment of 10-20 devices needs systematic model deployment, monitoring, and update capabilities. Building this infrastructure during a pilot makes scaling to hundreds of devices much more manageable.

The shift toward edge AI isn’t about abandoning the cloud. It’s about putting intelligence where it’s most effective — sometimes in the cloud, sometimes on the device, and increasingly often, in both places simultaneously.