CIOs should beware the AI confidence trap

Facing an AI make-or-break moment, IT leaders are confident of success, but most aren’t tracking the impact of AI projects to prove it.

According to a survey by Economist Impact, 84% of IT leaders say their AI returns are beating original estimates, but only 43% require teams to track the impact of AI projects, begging a vital question: How do IT leaders really know what value they’re getting from AI?

Moreover, only 39% of respondents say their organizations review AI projects for safety risks after systems go live — a major problem, says Eddie Milev, editorial lead for the Tech Frontiers program at Economist Impact.

“AI systems are not like conventional enterprise software,” he says. “They actually change after use, and they are systems that respond to the context that they’re provided. If companies don’t sustain governance after they deploy AI systems, they run a massive risk to have these systems to go rogue.”

Overall, the survey results suggest a misplaced confidence in the returns companies are achieving from AI projects, he says.

“It really shows that these capacities, the ability to measure, and the pure deployment are actually pretty disjointed,” Milev says. “That’s a pretty fundamental disconnect.”

How to succeed

The Economist Impact report points to several operational strategies that separate companies successfully deploying AI from the pack. Success starts with measurement, and organizations should focus on team-based output improvements by using AI rather than incremental time savings, Milev recommends.

“In the report, there’s this broad understanding that you shouldn’t focus on particular time savings here or time savings there when it comes to measurement, but rather you should look at an output of a team that uses AI versus the output of a team that doesn’t use AI,” he explains. “This is a much more accurate way to understand how much the technology is actually contributing to the performance of companies that are ahead of it.”

Successful AI rollouts share several attributes, according to Economist Impact, including:

• Strong data foundations: Companies achieving solid AI results treat data architecture as a binding constraint on AI.
• A disciplined route from idea to deployment: Only 40% of firms have a fully established AI development life cycle, but leading companies quickly scale what works and retire what doesn’t, instead of allowing experiments to pile up into pilot purgatory.
• Governance of AI over the long run: While about three in five organizations review AI systems during development, that number drops after deployment. The most successful organizations layer automated monitoring, human review, and drift detection across the full AI project lifecycle.
• Linking AI to the bottom line: Top companies connect AI to specific business outcomes and have the discipline to cut what isn’t working.
• Rewiring the organization to embed AI: Organizational change can be one of the most difficult elements of deploying AI, and the businesses reaping the most benefits embed AI into existing tools, routines, and decisions that already shape daily work.

Experiments outpace measurement

Several AI experts weren’t surprised about the report’s conclusion that measuring AI results still has a long way to go at most companies.

Carter Busse, CIO at AI integration provider Workato, says that many organizations moved quickly into AI experimentation before they built the operational discipline to consistently measure business impact. As a result, many companies are still measuring AI adoption instead of business outcomes.

“It’s relatively easy to launch a pilot or deploy a copilot, but much harder to tie AI directly to KPIs, revenue impact, efficiency gains, or workflow improvements across the business,” he says. “The organizations seeing the strongest results are the ones defining success metrics upfront and deploying AI against specific operational problems.”

One of the biggest hurdles to overcome is connecting AI to existing systems and workflows that run the business, like the report recommends, Busse says.

“A lot of organizations still have AI living at the edges in disconnected copilots and standalone tools instead of embedding it into core operational processes,” he adds.

IT leaders need to focus on internal adoption of AI, he recommends. “One thing that is missing is AI adoption internally, across your organization,” he says. “This is a crucial piece in driving AI’s success within an organization and often one that is an afterthought.”

Another problem is that too many companies still roll out AI without a vision of where it will have an impact, adds Andrew Sales, chief methodologist at agile methodology vendor Scaled Agile.

“Many start with the technology and go looking for a business problem to solve, rather than the other way around,” he says.

Successful companies deploy AI using analytical and rule-based tools where measurement playbooks are mature, and returns are well understood, Sales adds. While metrics for older automation technologies are well understood, newer technologies such as generative AI still give organizations fits when they try to understand their value, he notes.

“A more disciplined approach starts by identifying where AI can address specific struggles or inefficiencies, aligns implementation with concrete business objectives like cost reduction or customer satisfaction, and tracks performance through meaningful metrics rather than just adoption numbers,” Sales says.

IT leaders should create structured frameworks that track real impact, he says.

“This creates the accountability loop that makes measurement both possible and purposeful,” he adds. “That kind of structured framework is what separates organizations that can draw a clear line from AI activity to business outcomes from those that are left counting tasks completed and time saved, with no clear connection to what actually matters.”

Excitement for new tools takes you only so far

Benchmarking AI success hasn’t caught up yet with the enthusiasm of deploying a new technology, adds Darren Cassidy, CIO at AI-based CMS provider Sitecore.

“Most companies are still treating AI as a technology experiment rather than a business transformation,” he says. “They get excited about deploying something new, but they don’t do the harder work of wiring it into outcomes — revenue, cost, speed, or customer impact.”

IT leaders should anchor AI initiatives to clear business owners and outcomes, he recommends. AI should move metrics that company leaders actually care about.

“The hardest part is mindset and operating rhythm,” he adds. “It’s relatively easy to deploy a model; it’s much harder to change how teams work, make decisions faster, and trust AI‑driven recommendations. That’s an organizational challenge, not a technical one.”