Why AI TCO is so tricky — and how to start calculating it

Achieving return on investment is impossible without knowing the total cost of ownership (TCO) of an initiative — and when it comes to AI, CIOs are finding cost calculations anything but straightforward.

Subscription and token costs are a big part of the calculus, but several other factors go into the cost of AI projects, says Ben Schein, chief AI and analytics officer at AI data platform provider Domo. Chief among those are cloud infrastructure costs and the human time involved in guiding or correcting AI outputs, he notes.

In addition, many organizations have multiple divisions using different AI tools for vastly different purposes.

“There’s not like a single ledger,” Schein says. “Right now, and maybe for the foreseeable future, there’s sort of like a multiple ledger approach to how all this works.”

A shifting paradigm

While token costs have dropped significantly in the past two years, costs vary wildly between models and AI providers, and the price drops are often offset by increased usage. And AI providers have also explored other kinds of consumption-based pricing, including API calls, compute time, or documents processed.

All this makes it difficult to measure TCO, Schein says.

“You have sort of these subscriptions, you have the consumption and the tokenization, you have some of the infrastructure you might be paying for,” he says. “There’s also a human tax that introduces new time for verification and review, and if the AI is sloppy or creating slop, you might be inadvertently adding to your costs without knowing it.”

It’s difficult to measure TCO because AI doesn’t have a single cost center, agrees Shane Cronin, head of FinOps and ITAM services at systems integrator SHI.

“By the time you’re looking at the bill, you’re dealing with token consumption, cloud infrastructure, multiple AI models, governance tooling, integration work and, increasingly, autonomous agents making decisions across systems,” he says.

IT leaders at many organizations still define AI success through narrow technical metrics instead of prioritizing business outcomes, Cronin adds.

“Calculating token costs is relatively straightforward,” he adds. “Calculating whether those tokens actually created measurable business value is much harder. That’s where most CIOs are today.”

Unpredictability and hidden costs

Michael Moran, chief technology and information officer at contact center outsourcing provider NQX, sees several other factors leading to further unpredictability over AI costs.

For example, data center costs are rising, AI vendors are starting to shift from subsidized pricing to profitability, and organizations have increasingly complex AI use cases, he says.

“IT leaders should temper expectations that AI inherently reduces costs,” he adds. “Instead, it’s important to understand that full automation is likely to be prohibitively expensive for most enterprises, and that brands will need to balance AI investment with human engagement strategies that improve long-term value rather than cut costs in the short term.”

If AI implementations work exactly as expected right out of the grate, TCO should be relatively easy to calculate, he says. But agentic AI implementations often require much more human training and intervention than expected.

“These are the hidden costs that are often underestimated or ignored altogether when initially calculating TCO,” Moran adds.

Visibility is the first step

Chris Cagnazzi, chief innovation officer at IT solutions provider Presidio, is one IT leaders seeking to get a handle on the complexity of calculating AI TCO by applying playbooks from the cloud migration era.

Cagnazzi has adapted Presidio’s cloud cost optimization platform, PRISM, to track AI costs internally and to help customers do the same.

The first step toward tracking AI costs is visibility, he says. IT leaders should know every model running across their organizations, the cost per user per month, and what kinds of prompts each user is writing, he explains, adding that Presidio is using real telemetry to track internal AI use, as well as internal tools to direct prompts to cost-efficient AI models.

The second, more difficult, step is turning visibility into action, he adds. “You have to think about mapping the usage back to the owners, whether it’s users or groups,” he says. “Then you look at, what are some of the anomalies? And if you’re looking at those anomalies, do you have governance in place around overspend?”

What Presidio has found is that the bill for AI services represents only about 30% of the total cost, Cagnazzi notes.

“The other costs really lie in areas around the hidden AI stack,” he says. “Those things around orchestration or retrieval, observability of the guardrails, or the rereads and the redos. There’s a lot of cost that people are missing.”

While traditional IT costs can be fairly predictable, AI costs are driven by usage and can increase because employees are repeatedly using inefficient prompts, Cagnazzi says.

“The spend is hard to forecast; it’s hard to see the true hidden costs behind the bill,” he adds. “If that prompt is less efficient, it might produce a bill that’s 100% higher than what it should be.”

The good news, says Domo’s Schein, is that IT leaders have a lot of variables to play with to control AI spending. They can encourage users to use more cost-efficient AI models, they can track employee usage of AI, and they can test different prompts and other interactions for cost effectiveness, he says.

“The price spread on the different models is crazy,” he says. “You could say, ‘I have no ROI on this investment; if I could get the same outcome with a model that costs one-30th as much, I may have ROI.’”