CIOs drop shotgun approach to get more strategic with AI pilots

Over the past two years, many organizations have launched dozens of AI proof-of-concept projects, with a high failure rate and disappointing ROI, but there’s a new trend emerging, marked by a major reevaluation of the shotgun approach to AI experimentation.

Some IT observers now see many organizations pulling back on the number of AI POCs they launch, with some IT leaders turning to commercial AI tools and many others focused on a limited number of strategic and targeted use cases.

After an era of widespread experimentation, when companies were exploring the potential of AI, many have now zoned in on a handful of use cases, says Bhrugu Pange, managing director for AI, digital, and technology solutions at IT and management consulting firm AArete.

“We’re seeing a marked shift from high-volume experimentation to more focused, outcome-driven AI deployment,” he says. “Instead of launching dozens of proofs of concept in parallel, organizations are prioritizing a few use cases where AI can be embedded deeply into operational workflows and drive measurable results.”

For example, the finance department at one AArete customer identified invoicing as a high-friction workflow, then created an AI-driven fix that included generative AI, natural language processing, and optical character recognition, Pange says.

“This effort, sourced from within the [finance] function itself, delivered measurable improvements in cycle time and accuracy — outperforming several parallel experiments that lacked operational anchoring,” he adds. “This focused approach reflects a practical shift: AI delivers the most value when data, business context, and operational urgency come together in a few well-defined initiatives that span the enterprise.”

The days of dozens of AI POCs are dwindling

An April 2024 survey by IDC found organizations running an average of 37 AI POCs at the time, but some AI experts suggest the number was significantly underestimated, with some large companies running hundreds of pilots.

Many of the organizations that were running hundreds of POCs in previous years are now down to about 30, Pange says, with separate business units focusing on three to five experiments.

Other observers see the same trend, although they don’t offer estimates about the drop in the number of POCs being launched today. Even in July 2024, when data management provider Hitachi Vantara released its latest State of Data Infrastructure Global Report, most organizations were beginning to treat AI projects like traditional R&D and expecting returns over a two- to three-year horizon instead of rapid ROI, says Jason Hardy, CTO for AI at the company.

With continued insistence from company executives to experiment with AI, IT leaders are forced to balance speed and cost, Hardy says. Despite the pressure, nearly every conversation he has had recently with customers includes talk about running fewer, more strategic POCs, he adds.

“While the pressure to move fast has led many early adopters to deploy AI before they’re fully ready, we are starting to see successful adoption, especially with the expansion of focus towards agentic AI,” he says.

More IT leaders now appear to be resisting the pressure caused by the fear of missing out, and the “Wild West” of AI experimentation seems to be ending, Hardy says.

“While a shotgun approach to identifying possible outcomes through POCs and pilots does cast a wider net, it is easier for customers to get distracted by chasing after the science experiment or low-value outcomes,” he says. “Without a blueprint to execute against, customers could find themselves trying to solve the same foundational problem many different ways, emphasizing inefficiencies.”

Change the conversation

CIOs who are still being encouraged to launch dozens of AI POCs should steer the conversation elsewhere, says Chandra Venkataramani, CIO of IT outsourcing provider TaskUs. His company has created a strategic framework for AI adoption, with the emphasis on a limited number of high-value POCs.

“Don’t mistake speed for progress,” he says. “When there’s pressure to launch a lot of AI projects, CIOs should steer the conversation toward impact. Anchor your decisions to clear business goals and employee outcomes.”

He advises CIOs to take a long-term approach to AI deployment. AI projects should fit with operational needs, and IT leaders should focus on building trust in deployed AI tools with both teams and customers.

“Be selective,” Venkataramani adds. “Don’t run POCs just to check a box.”

Value in failing fast

Some AI experts, however, urged CIOs to leave room for AI experimentation. While many organizations have launched copilots and other “low-hanging fruit” AI tools, IT leaders can drive competitive advantages when they find specialized and greenfield AI use cases, says Nancy Gohring, senior research director in the generative AI strategies program at IDC.

CIOs should still focus on the “fail-fast” approach to AI experimentation and find a balance between 1,000 potential use cases and a handful, she adds.

“Rather than worry about the rate of movement from POC to production, put in place the systems that let you very quickly try out new ideas and determine whether or not they’re worth pushing to production,” Gohring says. “You really don’t want fewer POCs per se. You want to be able to experiment quickly.”

Past POC failures may be more of a problem with a lack of governance than the number of POCs launched, adds Thomas Robinson, COO at AI platform vendor Domino Data Lab. IT leaders should build in milestones for AI projects, including effective project management with checkpoints to assess progress, he recommends.

Still, there’s value in AI POCs, he adds. “Slowing down isn’t necessarily smarter,” he says. “What we’ve seen consistently is that accelerating the AI lifecycle with governance built in actually leads to better outcomes.”

Rapid iteration combined with responsible oversight is the key to success, Robinson adds.

“The key isn’t fewer proof-of-concepts, it’s governed velocity — being able to experiment quickly, learn fast, and scale what works, all while maintaining compliance and control,” he says.