AI’s big payoff hinges on fixing fragmented data: Study

In a rush to capitalize on AI, CEOs are hitting a wall. Their data is a mess. A new study revealed that while 61% of CEOs are actively deploying AI agents and planning to scale them, 50% admit their rapid tech investments have left their organizations with fragmented, disconnected systems. This data disarray is choking AI’s potential, with only 25% of AI initiatives delivering expected ROI in recent years.

Years of piecemeal tech adoption, according to the IBM survey of 2,000 CEOs across 30 countries, has created siloed systems that threaten to derail AI investments without a unified data foundation.

“CEOs are balancing the pressures of short-term ROI and investing in long-term innovation when it comes to adopting AI,” Mohamad Ali, senior vice president and head of IBM Consulting, said in a press note. Yet, the study shows that 68% of CEOs see an integrated, enterprise-wide data architecture as critical for cross-functional collaboration, and 72% view their proprietary data as the key to unlocking generative AI’s value.

“There’s no long-term AI ROI in layering models over broken foundations,” said Amandeep Singh, practice director at QKS Group. Surface-level AI integrations only add to the growing technical debt, he warned.

Not leveraging AI and your data is a choice to fall behind, IBM Vice Chairman Gary Cohn warned in a press note.

The data disconnect isn’t just a technical issue — it’s a strategic one. The study found that 59% of CEOs struggle to balance funding for existing operations with investments in innovation, especially when unexpected changes hit.

Driven by the fear of being left behind, 64% of CEOs admit to investing in AI without fully grasping its value. In 2025, Chief AI Officers reported a modest average ROI of 14%, even as AI programs scale beyond initial pilots. The study noted that while groundbreaking proofs-of-concept grab attention, they don’t consistently translate to business results.

CEOs know AI could drive significant growth — 85% expect positive ROI from scaled AI efficiency by 2027 — but they’re building on shaky foundations.

Data issues are a top AI bottleneck, according to a Deloitte research. Only half of the CEOs surveyed reported sufficient data integration for scaling AI. Untangling legacy systems, inconsistent formats, and governance gaps requires both technical and cultural fixes for CIOs.

CEOs recognize that strategic leadership and talent are key to AI value, yet skills gaps persist. Roughly a third (31%) of the workforce needs AI retraining within three years, and 54% of CEOs are hiring for new AI roles, highlighting the talent scramble.

The study also highlighted cautious optimism. While only 52% of CEOs report realizing value from generative AI beyond cost reduction, 68% say their organizations have clear metrics to measure innovation ROI. By 2027, 77% expect positive returns from AI investments focused on growth and expansion. But these gains depend on CIOs untangling the data mess now.

Getting the data stack right

CIOs need to start by auditing their data, spotting gaps, cleaning inconsistencies, and making sure it’s usable. The goal isn’t to lock everything into a central system, but to ensure data is structured, governed, and easy to access. With smart tools like data virtualization and system integration, teams can work with a unified view without added complexity, suggested the study.

Singh urged CIOs to embrace “data product thinking” — treating high-quality, reusable data sets as business assets. When done right, this powers AI use cases that actually move the needle, like predicting local stock needs or reducing travel spend.

To make AI work in real time, CIOs should build a data fabric that connects systems and embeds intelligence into day-to-day operations. Cloud-native platforms help teams collaborate across silos, while event-driven architecture lets AI respond the moment new data comes in.

AI also needs to be trained on clean, enterprise-specific data, with business rules, ethics, and security baked in. A strong training framework, coupled with feedback loops, helps AI spot issues, improve processes, and stay relevant, added the study.

“No AI model should hit production without plugging into real business workflows,” Singh said. “Done right, this rewiring turns data chaos into a competitive advantage.”