Evals don’t get much press. Models get press. Agents get press. Whatever the new platform-of-the-month is gets press. But if you talk to anyone running an AI workload in production in 2026, evals are quietly eating a much bigger share of the budget than anyone expected eighteen months ago. It’s worth looking at where that money is actually going.

I’ve been pulling apart the cost structure of evals at a few different scales — from small SaaS teams running maybe a hundred test cases per model release to large enterprises running tens of thousands of evaluations across multiple models, multiple prompts, and multiple regulatory regimes. The numbers are interesting, and the line items are not what you’d expect.

Evals used to be free

Here’s the funny thing. In 2023 and into 2024, evals were essentially a side project. You’d write a spreadsheet of test cases, run them by hand, eyeball the outputs, and either ship or not ship. The cost was your engineer’s time and maybe a few dollars in inference. That was it.

What changed is that production AI workloads stopped tolerating regressions. If your support agent suddenly starts confusing two of your products because the new model release changed something subtle, you’ve got a real business problem. If your legal review agent starts missing a clause type it used to catch, you’ve got an even bigger one. So teams started running evals on every prompt change, every model upgrade, every deployment. Continuous integration for AI behavior. And those eval runs cost real money.

The four lines of spend

When you decompose what teams actually spend on evals in 2026, four cost categories dominate.

Inference cost. Running your eval set isn’t free. Every test case is a model call. If you’ve got five thousand test cases, two prompts, three model variants you’re comparing, and you run the full grid on every change, you’re talking thirty thousand model calls per eval pass. At frontier model pricing — even with the cost curves coming down — that’s not nothing. A team running eval passes nightly can easily spend more on evals than they spend on production traffic in their early stages.

Judge inference cost. Most teams have moved past pure string-match evals and are using model-based judges to score outputs. Coherence judges, factuality judges, safety judges, task-completion judges. Each judge call is itself a model call. The judges are often using a different model than the system being tested — sometimes a more expensive one — and you typically run multiple judges per output. So judge inference can be two to four times the cost of the actual system inference.

Human review. Despite all the automation, every serious team has some level of human-in-the-loop review. Either spot-checking judge outputs, reviewing edge cases the judges disagreed on, or doing periodic quality calibration where humans grade a sample to make sure the judges haven’t drifted. Human review is expensive per output but you don’t do much of it, so it tends to settle at maybe ten to twenty percent of the eval budget for mature teams.

Eval infrastructure. This is the sneaky one. Storing eval results. Versioning eval sets. Running CI pipelines that gate deployments on eval pass rates. Building dashboards so product managers can see whether quality is trending up or down. None of this is exotic but it adds up. Teams that started with notebook-based evals have all migrated to proper eval platforms by 2026, and those platforms — whether self-built or commercial — cost real money to run.

The model-comparison tax

One pattern I keep seeing: teams that want to compare multiple model providers spend two to three times more on evals than teams that lock in to a single provider. The reason is obvious in retrospect. If you’re trying to decide whether to move from Claude to GPT or to Gemini, you have to run your full eval suite on every candidate. And if you want that comparison to be statistically meaningful you need to run it more than once, ideally across a varied test set.

For a serious provider switch, I’ve seen teams run something like fifty thousand eval calls just to make a confident decision. At current frontier pricing that’s a few thousand dollars of inference cost. Not catastrophic. But if you’re doing this every quarter because the model landscape keeps shifting, it adds up. MIT Technology Review covered this dynamic earlier in 2026 and the framing has stuck with me — they called it the “model-comparison tax” and noted it’s one of the underdiscussed costs of staying competitive in the AI tooling space.

The hidden win: cheap judges

The single biggest cost optimization I’ve seen this year is teams switching from frontier-model judges to small-model judges. Two years ago using a cheaper model as a judge would have been unthinkable — they weren’t reliable enough. In 2026 that’s changed. The small models from each of the major labs are competent enough for most judge roles, especially when you give them a well-structured rubric.

Teams that have made this switch are reporting fifty to seventy percent reductions in their eval costs without measurable quality loss. The trick is calibration. You can’t just swap the judge model and call it a day. You need a calibration set where you’ve graded outputs by hand, and you need to verify that the cheap judge’s grades correlate well with human grades on that set. Most teams that do this calibration end up shipping the cheap judge with confidence.

The eval-set drift problem

The other hidden cost is eval-set maintenance. Your eval set is supposed to represent your real production distribution. But your production distribution drifts. New use cases emerge, old ones fade, customers change how they ask questions. If you don’t refresh your eval set, you end up shipping models that pass your evals brilliantly but fail in production because production no longer looks like your evals.

Teams I respect are spending real engineering time on eval-set curation. Reviewing production logs to identify new failure modes. Pulling representative samples from the latest customer queries. Pruning eval cases that have stopped being meaningful. This isn’t glamorous work but it’s load-bearing. Without it, your eval scores stop predicting your actual quality.

Where the spend is heading

A few predictions. First, eval costs as a share of total AI budget are going to keep growing through 2026 and into 2027. As more workloads move to production, the fraction of compute that goes to evaluation is going to climb. I wouldn’t be surprised to see mature teams running eval-to-production compute ratios of one-to-five by the end of the year. That sounds like a lot until you remember that a single quality regression in production can cost vastly more than a year of eval compute.

Second, judge models are going to keep getting cheaper faster than production models. The small-model improvements have been outpacing frontier-model improvements on judge tasks. So the cost-per-eval should keep dropping even as the volume goes up.

Third, evals will become a board-level topic at companies running serious AI in production. The way uptime and security are board topics now, AI quality will be too. That means more investment in eval infrastructure, more rigor, and more spend.

Evals are boring. They’re also the thing that determines whether your AI investment turns into a reliable product or a constant fire drill. Worth paying attention to where the money is going.