The headlines all go to the big models. GPT-5 rumours, Claude’s latest capabilities, Gemini’s multimodal updates — these dominate the conversation. But there’s a parallel revolution happening that might end up mattering more for most enterprises: small language models (SLMs) that run locally, on-device, without ever calling a cloud API.

I’ve been tracking this space closely since Microsoft released Phi-3 in early 2024, and the progress over the past year has been remarkable. Models in the 1-7 billion parameter range are now handling tasks that would have required 70B+ parameter models two years ago. And they’re doing it on hardware that sits in an office, a factory floor, or a mobile device.

Why Size Matters (In Reverse)

The case for small models in enterprise settings is straightforward and compelling.

Cost. Running a 3B parameter model on a local GPU costs effectively nothing per query after the hardware investment. Compare that to API calls that add up fast at scale — an enterprise processing millions of documents annually can save hundreds of thousands of dollars by moving appropriate workloads to local models.

Latency. Local inference eliminates network round trips. For real-time applications — quality inspection on a production line, point-of-sale product recommendations, field technician assistance — the difference between 50ms local inference and 500ms cloud API calls is the difference between usable and frustrating.

Privacy. This is the big one for regulated industries. When patient records, financial data, or classified government information never leaves the local network, an entire category of compliance headaches disappears. No data processing agreements with cloud providers, no cross-border data transfer concerns, no risk of training data contamination.

Reliability. No internet connection required. No API outages. No rate limits. For mining operations in remote Australia, manufacturing plants with spotty connectivity, or defence applications where network access isn’t guaranteed, local models are the only viable option.

What’s Actually Working

Let me be specific about where I’m seeing small models deployed successfully in enterprise settings.

Document classification and routing. A fine-tuned 3B parameter model can classify incoming documents by type, urgency, and department with accuracy that matches larger models for this specific task. Several Australian financial services firms are running this kind of pipeline entirely on-premises.

Structured data extraction. Pulling specific fields from invoices, contracts, receipts, and forms. Models like Phi-3 and Mistral 7B, fine-tuned on domain-specific documents, handle this reliably. The key is that the task is narrow and well-defined — exactly where small, specialised models outperform general-purpose giants.

Code assistance in secure environments. Defence contractors and government agencies can’t send their code to external APIs. Local code completion models, based on architectures like CodeLlama or StarCoder derivatives, are filling this gap. They’re not as capable as Copilot or Claude for code, but they’re good enough for autocomplete and boilerplate generation in environments where “good enough and private” beats “excellent but requires external API access.”

Edge inference for IoT and manufacturing. This is where I think the biggest growth will happen. Small models running on edge devices can process sensor data, camera feeds, and equipment telemetry in real time. The team at Team400 has been building interesting implementations here — deploying compact AI agents on edge hardware for Australian manufacturers who need real-time quality inspection without cloud dependencies.

The Fine-Tuning Advantage

Here’s what makes small models especially interesting for enterprises: they’re dramatically easier and cheaper to fine-tune. You can fine-tune a 3B parameter model on a single consumer GPU in hours. A 7B model takes a day or two on modest hardware.

This means enterprises can create highly specialised models tailored to their exact use case, their exact data, and their exact terminology. A mining company can fine-tune a model on their geological reports. A law firm can train on their specific document formats. A hospital can build a model that understands their particular clinical workflow.

The result is often a small model that outperforms a much larger general-purpose model on the specific task it was trained for. It’s the difference between hiring a brilliant generalist and hiring a domain expert.

What’s Still Missing

I don’t want to oversell this. Small models have real limitations. Complex reasoning, creative writing, nuanced analysis of ambiguous situations, multi-step planning — these still require larger models. And maintaining a fleet of fine-tuned small models creates its own operational overhead: version management, retraining pipelines, performance monitoring across dozens of specialised models.

The smart approach is hybrid — route complex tasks to cloud-based frontier models and handle everything else locally. The architectural challenge is building that routing layer cleanly, which is a non-trivial engineering problem but a solvable one.

Small language models won’t replace frontier models. But for the majority of enterprise AI workloads — the repetitive, well-defined, privacy-sensitive, latency-critical tasks that make up the bulk of actual business processes — they’re increasingly the right tool for the job.