The AI narrative fixates on scale. Bigger models, more parameters, more training data. GPT-4, Gemini Ultra, Claude Opus—the largest models dominate discussion.

Meanwhile, in actual enterprise deployment, smaller models are winning.

The Deployment Reality

Here’s what happens when an enterprise tries to deploy a frontier model in production:

Latency. Large models take time to respond. For real-time applications—customer service, recommendation engines, document processing at scale—that latency breaks user experience.

Cost. API calls to large models add up quickly at enterprise volume. Processing millions of documents or millions of customer interactions generates substantial bills.

Privacy. Many enterprises can’t send data to third-party APIs. Deploying large models on-premises requires infrastructure most organizations don’t have.

Reliability. External API dependency introduces failure modes. When the AI service is down, business processes stop.

These constraints push enterprises toward smaller, deployable models.

What Small Models Can Do

“Small” is relative—models in the 7B-30B parameter range are still impressively capable.

Current small models handle:

• Document classification with high accuracy
• Entity extraction from structured and semi-structured text
• Summarisation of reasonable-length content
• Question answering over defined knowledge bases
• Translation between common language pairs
• Sentiment analysis and content moderation
• Code completion and simple generation

For many enterprise applications, this capability is sufficient. You don’t need GPT-4 to categorize support tickets or extract dates from contracts.

The Quantisation Revolution

Quantisation techniques have dramatically reduced model resource requirements. Running a 7B model on consumer hardware was impractical two years ago. Now it’s routine.

This matters for:

• On-device deployment (mobile, IoT, embedded systems)
• Edge computing close to data sources
• Cost-effective cloud deployment
• Air-gapped environments with no external connectivity

Quantised small models run where large models can’t.

Distillation and Fine-Tuning

Large models are increasingly used to train smaller models rather than serve production traffic directly.

The workflow:

1. Use frontier model to generate high-quality training data
2. Fine-tune small model on that data for specific tasks
3. Deploy small model in production

The small model inherits task-specific capability from the large model without the deployment constraints.

This approach is particularly effective for narrow applications. A small model fine-tuned exclusively for invoice processing outperforms a general large model on that specific task.

Enterprise Deployment Patterns

Organizations deploying small language models typically follow certain patterns:

Gateway architecture. Simple routing layer directs requests to appropriate models. Complex queries go to larger models (possibly external APIs), simple queries to local small models.

Task-specific models. Rather than one model doing everything, multiple small models optimised for different functions. The email classifier is separate from the document summariser.

Edge deployment. Models run on device or at edge locations, with larger models available for fallback when needed.

Hybrid inference. Initial processing by small local model, with escalation to large model for uncertain cases.

What’s Driving This

Several trends reinforce small model adoption:

Open source momentum. Llama, Mistral, and other open models give enterprises options beyond API dependency. The best open models are competitive with commercial offerings for many tasks.

Hardware evolution. Consumer and enterprise hardware increasingly includes AI accelerators. Deploying models locally gets easier every year.

Regulatory pressure. Data sovereignty requirements in Europe, Australia, and elsewhere push against cloud AI dependency.

Cost pressure. CFOs eventually ask about the AI API bill. Moving inference to owned infrastructure reduces ongoing costs.

The Emerging Stack

The small model enterprise stack is maturing:

• Model serving: vLLM, TGI, Triton
• Quantisation: GPTQ, AWQ, GGML
• Fine-tuning: LoRA, QLoRA
• Orchestration: LangChain, LlamaIndex
• Vector stores: Pinecone, Milvus, Weaviate
• Monitoring: LangSmith, Arize

This infrastructure enables sophisticated AI applications without frontier model dependency.

Limitations To Acknowledge

Small models aren’t universally superior. They struggle with:

Complex reasoning. Multi-step logical problems still favour larger models.

Broad knowledge. General knowledge questions expose training gaps in smaller models.

Long context. Large context windows in frontier models enable applications small models can’t match.

Instruction following. Complex prompts with multiple requirements challenge smaller models.

The decision isn’t “small vs large” but “what’s appropriate for this specific application?”

Looking Forward

Expect the gap between frontier and deployable models to narrow. Techniques that improve small model capability—better training data, improved architectures, more efficient quantisation—advance continuously.

Within a few years, models deployable on commodity hardware will handle tasks currently requiring frontier models. The frontier will advance too, but the “good enough” bar for most enterprise applications will be met by smaller systems.

For AI consultants Brisbane or Melbourne firms helping enterprises with AI strategy, small model deployment expertise is increasingly valuable. The flashy frontier model demos are one thing; running AI reliably in production at reasonable cost is quite another.

The enterprises that figure out efficient AI deployment—not just impressive AI demos—will have significant operational advantages. Small models are the path to that deployment efficiency for most use cases.