The global shortage of high-performance GPUs for AI workloads is hitting Australian businesses harder than the general tech press is reporting. Lead times for NVIDIA H100 and A100 systems are running 6-9 months, and that’s if you can get an order accepted at all. Prices have increased 40-60% over list price through the secondary market.

I’ve been working with companies trying to deploy machine learning systems, and GPU availability has become a critical constraint on timelines and budgets. Projects that were scoped assuming normal hardware procurement are now stalled or being redesigned to work within available resources.

The shortage isn’t just affecting the obvious players like tech companies and research institutions. Businesses in mining, agriculture, manufacturing, and logistics that want to implement computer vision or predictive analytics are finding they can’t get the compute infrastructure they need.

Why the Shortage Exists

Demand for AI compute has exploded over the past two years. Every major company is exploring AI applications, and training large models requires massive amounts of GPU compute. The infrastructure build-out for AI services by cloud providers has absorbed most of the available supply.

NVIDIA dominates the AI chip market with probably 85-90% market share. AMD and Intel are shipping AI-focused chips, but NVIDIA’s CUDA ecosystem and software stack have a huge incumbent advantage. Most AI frameworks and tools are optimized for NVIDIA hardware.

Manufacturing capacity for advanced chips is constrained. TSMC produces most of the high-end GPUs, and their fabrication capacity is fully allocated years in advance. Adding new fab capacity takes 3-5 years and billions of dollars in investment.

Geopolitical factors are creating additional supply chain friction. Export controls on advanced chips to China have reduced the global available market but haven’t freed up supply for other regions. The chips just aren’t being produced at sufficient volume for global demand.

Impact on Australian Businesses

Cloud GPU instances are easier to access than physical hardware but they’re expensive and sometimes unavailable. I’ve seen projects budgeted at $500/month for GPU compute now running $2,000-$3,000/month for equivalent capacity.

AWS, Google Cloud, and Azure all have GPU instance types, but availability varies by region. The Sydney and Melbourne regions frequently show GPU instances as unavailable or limited to short-term spot instances that can be terminated with minimal notice.

For training large models, you need sustained GPU access for days or weeks. Spot instances don’t work for that. Reserved instances provide guaranteed access but require 12-month commitments at prices that have increased significantly over the past year.

Some Australian companies are using US or Singapore cloud regions for GPU access, which introduces latency issues if your data’s in Australia and regulatory concerns if you’re handling sensitive information that needs to stay onshore.

Alternatives to Cloud GPUs

On-premise GPU servers make sense for some workloads, but purchasing hardware is even more difficult than cloud access. Suppliers are prioritizing large enterprise orders. If you’re trying to buy two or three GPU servers for a mid-sized business, you’re at the back of the queue.

I know companies that have purchased consumer gaming GPUs (RTX 4090s) as a stopgap. They’re not designed for datacenter use and lack features like NVLink for multi-GPU scaling, but they’re available and can run inference workloads reasonably well.

The performance per dollar on gaming GPUs is actually better for some inference tasks. An RTX 4090 costs around $3,500 and delivers decent performance for running pre-trained models. An A100 costs $15,000+ (if you can find one) and is designed for training, which many businesses don’t actually need.

Some companies are working with Team400 and similar AI consultancies that have their own GPU infrastructure. Outsourcing the compute to a specialist who’s already solved the hardware procurement problem can be faster and cheaper than trying to build your own capacity.

Optimizing for Limited Compute

Model efficiency has become critical. Six months ago, companies were defaulting to the largest models they could run. Now there’s real focus on using the smallest model that achieves acceptable accuracy.

Model distillation, where you train a smaller model to mimic a large one, is getting more attention. If you can get 90% of GPT-4’s performance from a model that’s one-tenth the size, you save enormously on compute costs.

Inference optimization techniques like quantization, pruning, and efficient serving frameworks (like NVIDIA’s TensorRT or AWS Inferentia chips) let you run models on less expensive hardware. A quantized model might run on a CPU or edge device instead of requiring a GPU.

Some workloads can use older or less powerful GPUs. If you’re doing batch processing overnight rather than real-time inference, a T4 GPU from a few generations ago might suffice. They’re more available and much cheaper than current-generation hardware.

Open Source Model Deployment

The rise of high-quality open-source models (Llama, Mistral, and others) has helped with the compute shortage. Instead of paying for API access to proprietary models hosted on someone else’s GPUs, you can run open models on your own infrastructure.

The trade-off is you still need GPUs to run them at scale. A 70-billion parameter model requires significant GPU memory to run efficiently. But smaller models in the 7B to 13B range can run on consumer hardware or modest cloud instances.

Hugging Face and similar platforms provide model hosting that abstracts away GPU management. You upload your model or use a pre-trained one, and they handle the infrastructure. It’s more expensive per inference than running your own hardware but eliminates the procurement problem.

Regional Differences in Access

Access to AI compute infrastructure in Australia lags behind the US and parts of Asia. Cloud providers prioritize their largest markets, and Australia’s relatively small. We get capacity after North America, Europe, and major Asian markets are served.

There’s discussion about building sovereign AI compute capability in Australia, but that’s years away if it happens at all. The capital investment required is enormous, and it’s not clear there’s sufficient local demand to justify it commercially.

Some Australian universities and research institutions have GPU clusters for AI research, and there are partnership programs that let businesses access that compute. CSIRO and some universities offer commercial research collaborations that include compute access.

Where This Is Heading

GPU supply should improve over the next 12-18 months as new TSMC fab capacity comes online and as alternative chip providers gain market share. AMD’s MI300 series and Intel’s Gaudi chips are competitive technically; they just need software ecosystem maturity.

Specialized inference chips optimized for running AI models rather than training them are becoming more common. Google’s TPUs, AWS Inferentia, and Groq’s LPU are examples. They’re more efficient for production deployments, which should ease demand for general-purpose GPUs.

On-device AI processing is improving. Apple’s M-series chips, Qualcomm’s Snapdragon platforms, and dedicated edge AI accelerators can run smaller models locally. For some applications, that eliminates the need for datacenter GPU compute entirely.

Model efficiency improvements continue. Research into mixture-of-experts architectures, sparse models, and other techniques is reducing the compute required to achieve given performance levels. This won’t eliminate demand for GPUs but will stretch available capacity further.

Practical Steps for Businesses

If you’re planning AI projects, factor GPU availability into your timeline. Don’t assume you can provision compute on-demand. Build in 3-6 months of lead time if you need dedicated hardware, and have cloud backup plans.

Start with the smallest viable model for your use case. Don’t default to the largest available model without testing whether smaller ones meet your needs. The compute savings can be substantial.

Consider inference optimization from the beginning rather than as an afterthought. Designing your system to run efficiently on available hardware is easier than trying to optimize later when you’re already deployed.

Look into managed AI services where appropriate. If your use case fits a pre-built solution like computer vision APIs or natural language processing services, the compute’s someone else’s problem and they’ve already solved GPU procurement.

Build relationships with hardware vendors and cloud providers if you’re likely to need significant AI compute in future. Being a known customer with a track record can help when allocating limited resources.

The GPU shortage is frustrating and expensive, but it’s forcing companies to be more thoughtful about AI deployments rather than just throwing compute at problems. In that sense, the constraint might actually be beneficial long-term, even though it’s painful short-term.