Two years ago evaluating AI systems was a side activity that the data science team did before shipping. In 2026 it is a continuous production cost that some enterprises are still treating as a one-off. The bills are catching up.

What evaluation actually involves now

The eval stack has three layers. Offline evaluation against a held-out test set, run on every model or prompt change. Online evaluation through A/B testing or shadow traffic, run continuously. Production monitoring with sampled human review and automated regression detection.

All three are needed. None of them are cheap.

The cost shape

For a production system handling moderate traffic — a customer service deflection bot, say — the eval infrastructure typically costs 20% to 40% of the inference cost itself. For systems with higher consequences the ratio gets worse. Healthcare and finance teams are reporting eval costs that match or exceed their inference spend.

The cost is in three places. Synthetic eval traffic that exercises the system without affecting users. Human annotation of model outputs, sampled at a rate that catches drift. Infrastructure to store and analyse the eval data, which gets large fast.

What is being underestimated

Two things. First, that eval is continuous, not one-off. Models drift, prompts change, the upstream data distribution shifts. The eval suite needs maintenance and the data needs to be refreshed every few months.

Second, that the human review piece is the bottleneck. Annotation contractors are reliable for some types of review and not others. In-house subject matter experts are expensive and slow. The teams getting this right have built tooling that makes human review fast for the experts whose judgement is needed.

The unsexy line item

The CFO conversation about AI in 2026 is shifting from “what does inference cost” to “what does maintaining the system cost over a year.” Eval is most of the answer.

For organisations building their first serious production AI system, AI evals consulting firms that have built and maintained these systems are worth the engagement. The eval architecture decisions made early are hard to walk back later.