AI cost overruns are adding up — with major implications for CIOs

IT and finance leaders are struggling to estimate and contain the cost of AI deployments, with most AI projects exceeding spending targets, according to a new survey.

Despite mass AI adoption among enterprises, many IT leaders are finding it difficult to achieve ROI for AI projects, with inaccurate budgeting a significant part of the problem.

About 85% of organizations misestimate AI costs by more than 10%, and nearly a quarter are off by 50% or more, according to a recent survey from SaaS benchmarking firm Benchmarkit and cost governance platform Mavvrik. The cost estimates are nearly always too low.

Several factors contribute to this issue, with data platforms being the top driver of unexpected AI costs, followed by network access to AI models, according to survey respondents. Large-language model (LLM) token costs are the fifth highest driver of unexpected AI expenditures.

CIOs take the blame

While budget authority often rests with the CFO, CIOs overseeing unexpectedly expensive AI projects can expect major implications, experts say.

“Underestimating AI projects by this much doesn’t just blow budgets, it risks stakeholder confidence and adoption,” says Sergii Opanasenko, cofounder of AI-focused web development firm Greenice. “The best path forward is to treat AI like any other core infrastructure investment: scope realistically, account for integration and iteration, and build in a buffer for experimentation.”

Many organizations appear to be “flying blind” while deploying AI, adds John Pettit, CTO at Google Workspace professional services firm Promevo. If a CIO-led AI project misses budget by a huge margin, it reflects on the CIO’s credibility, he adds.

“Trust is your most important currency when leading projects and organizations,” he says. “If your AI initiative costs 50% more than forecast, the CFO and board will hesitate before approving the next one.”

Cost overruns are often associated with hidden layers beneath the cost of an AI model, including data preparation, security, integration, and compliance, Pettit adds.

Dan Stradtman, CMO at AI-powered knowledge management vendor Bloomfire, has seen the effects of project cost overruns. “I’ve led teams at Fortune 500 companies where missed forecasts set off a chain reaction: delayed roadmaps, frozen headcount, and CFOs pulling back on strategic bets,” he says. “For IT leaders, governance isn’t optional. Without disciplined visibility into costs and a clean foundation of knowledge feeding AI, you’re left with unpredictable bills and frustrated executives.”

Beyond creating distrust in IT leadership, missed cost estimates also hurt the company’s bottom line, notes Farai Alleyne, SVP of IT operations at accounts payable software vendor Billtrust. “It is not just an IT spending issue, but it could materialize into an overall business financials issue,” he says.

The survey results bear this concern out. More than eight in 10 companies reported that AI costs eroded gross margins by more than 6%, with more than a quarter of companies seeing drops of 16% or more.

Tiny budgets, unrealistic expectations

To demonstrate the AI budget disconnect, Greenice recently examined more than 500 agentic AI projects posted on freelance marketplace Upwork and found that more than 60% of them had budgets of less than $1,000, with expected delivery timelines of less than three months. Most of those cost estimates and timelines appeared unrealistic, Opanasenko says.

In his experience, enterprise leaders often assume AI coding assistants or no-code/low-code tools can take care of most of the software development needed to roll out a new AI tool. These tools can be used to create small prototypes, but for enterprise-grade integrations or multi-agent systems, the complexity creates additional costs, he says.

“We often see clients who proudly say they’ve built 80% to 90% of their system in a week with AI, but the remaining 10% to 20% is where the real complexity hides,” Opanasenko adds. “That last mile often takes far longer — either because integrations are complex or because AI assistants struggle as projects grow larger and contexts get too big.”

In addition, organizations often underestimate the cost of operating an AI project, he says. Token usage for vectorization and LLM calls can cost tens of thousands of dollars per month, but hosting your own models isn’t cheap, either, with on-premises infrastructure costs potentially running into the thousands of dollars per month.

And then there’s scope creep. “When clients change their minds or grow their appetite mid-project, the only way forward is to re-evaluate the scope, cut back to what’s realistic for the budget, and move extra features into later phases,” Opanasenko says.

Best practices for containing costs

Promevo’s Pettit recommends CIOs embrace observability technologies to track AI costs and focus on outcome-based ROI metrics.

“Without observability, you’re guessing,” he says. “With it, you can spot a runaway training job or ballooning API usage before it wrecks your budget.”

CIOs should use FinOps practices and treat AI cost as a primary metric, complete with dashboards and alerts, he recommends. IT leaders should also encourage cross-team transparency, with marketing, R&D, and IT teams all sharing their AI use and costs.

IT leaders can also start with AI pilot projects to measure real consumption, Pettit suggests, and they should realize that small AI solutions sometimes work best.

“Not every use case needs the biggest model,” Pettit says. “Sometimes a lighter approach delivers 90% of the value at 10% of the cost.”

Like Pettit, Billtrust’s Alleyne recommends cost management and observability tools to control AI spending. While cost management tools from cloud vendors can be useful, they aren’t as deep as third-party solutions and don’t cover every potential AI cost, he says.

“These other third-party tools can make recommendations like, ‘Hey, you’re actually overspending on this feature, and you can resize,’” he says. “We’ve done a lot of resizing, taking the recommendations on the contract commitments, and changing our backups to be more mindful of the overall cloud spend.”

The key to using cost management and observability tools is to check them regularly, Alleyne adds. Billtrust IT leaders check the AI cost management software at least a couple of times a month.

“You can have the tools, but don’t let them be shelfware,” he says. “Be diligent about a cost management program.”

IT leaders should also communicate regularly with other departments about their AI use, Alleyne recommends.

“A lot of the misses can be attributed to some other department doing something with AI, and then, it ends up showing up in your cost,” he says. “You’ve got some department that’s using the capabilities with the ease of cloud onboarding, and they’re running on your corporate cloud account, and you get the bill.”