CIOs increasingly dump in-house POCs for commercial AI

After thousands of AI proof-of-concept projects have died on the vine, many organizations are scaling back internal efforts in favor of adopting commercial, off-the-shelf AI tools.

About half of companies surveyed by Gartner in late 2023 were developing their own AI tools, but the number fell to about 20% at the end of 2024, says John-David Lovelock, a vice president and analyst at Gartner.

Many organizations are still running a few POCs, but cooling hype surrounding generative AI has many CIOs turning to vendors, whether they be large language model (LLM) providers or traditional software sellers with AI built into their products, Lovelock says.

Many ambitious AI projects started in 2024 or earlier now face internal scrutiny because of the high POC failure rates, Lovelock adds. Recent research by IDC found that 88% of POCs didn’t make it into widescale deployment, while CIOs struggle to define POC success.

“Last year, when CIOs were doing proof of concept work and getting lots of help from service providers and internal resources, there was a fairly high failure rate among companies that already had a good pedigree in artificial intelligence,” Lovelock says. “But for companies that didn’t have that same pedigree, their failure rates were over 50% higher.”

Not enough brain power

The “vast majority” of customers at cloud- and automation-focused Asperitas Consulting are now using off-the-shelf AI tools, says Scott Wheeler, the company’s cloud practice lead. Many Asperitas customers are in the financial services and insurance sectors, often seen as potential beneficiaries of gen AI functionality.

Many companies that have tried to build their own AI tools have run into problems with a lack of expertise and budget, he says.

“For most people, the juice isn’t worth the squeeze,” Wheeler adds. “You’ve got to have the brain power to do it right, and those people are in super high demand. And then, you’ve got to have the time and other resources in addition to the brain power.”

Lovelock has observed several variations of POCs, with some companies attempting to build their own AI models from scratch and others focused on adding functionality on top of existing LLMs. Still, many of those less ambitious POCs either failed or delivered underwhelming results, some AI experts say.

Heavy pressure from top executives and board members to launch gen AI POCs has led to many ill-fated projects, says Eamonn O’Neill, CTO at managed services provider Lemongrass.

“You had this initial impetus to try out, which, in itself, is not a bad thing,” he says. “But then, the reaction once you did these POCs was, ‘Well, that’s not actually very useful,’ because the quality was quite low. Nobody seems to understand how to use it properly.”

With the initial excitement over gen AI, many business and IT leaders had inflated expectations for their POCs, adds Carmel Wynkoop, partner in charge of AI, analytics, and automation at accounting and business consulting firm Armanino.

“The attitude was, ‘Let’s get to the big hairy problems and sick AI on it,’” she says. “What we’re seeing is something slightly different, with incremental improvements in process and in time, but they’re not the big hairy problems.”

Many organizations jumped into huge AI projects with a long time to market and to ROI, she says. “If I’m tackling my largest issue with AI to begin with, that could be a year-long project,” Wynkoop adds. “I may have to go clean up a bunch of data, and I may have to modify the GPT code.”

The market has changed

After the AI POC gold rush of the past two years, the market dynamic has shifted in 2025, Gartner’s Lovelock says. Instead of CIOs going out and looking to build or buy AI tools, software vendors are pushing their add-on AI products onto CIOs, who themselves are seeking to get more practical about generative AI results.

“We’re used to CIOs going out and buying software, and this year, they’re going to be sold [AI] software,” he says. “In the past, they had an idea in mind, a problem to solve, and it is directional, intentional. They are in control.”

In some cases, the CIOs won’t have a choice about whether to purchase the add-on AI, Lovelock says.

“This year, virtually every software company, for virtually every product, will have a gen AI feature this year, if they don’t already,” he says. “The salespeople are going to be calling their customers and saying, ‘We have gen AI,’ and in some cases, you come in one morning, and you have a slightly higher bill and a new button.”

At the same time, much of the incentive for companies to build their own AI tools has gone away, with software vendors and AI covering much of the targeted functionality, Armanino’s Wynkoop says.

Aiming smaller

Instead of solving big problems with AI, companies lacking in-depth expertise may benefit from aiming for something smaller, she adds.

“If I’m starting with initiatives that I can get some quick turnarounds on, build momentum, and get some efficiencies in a lot of processes across my organization, then I can build the reputation about what AI can do and start seeing those efficiencies take hold,” Wynkoop says. “Then I can look at how people change their behavior and how they’re working with AI.”

Going forward, many internal development projects are likely to focus on training an AI model using in-house data to create niche functionality, says Asperitas’ Wheeler.

“The AI model seems to be a commodity thing,” he says. “But a lot of companies have these customized data sets, and they’ll train the AI model on their proprietary data.”

These AI models trained on proprietary data have the potential to create great value for the companies that embrace this approach, adds Daniel Avancini, chief data officer at Indicium, an AI and data consultancy.

“It’s a very niche product, but there’s a lot of value for our company, if we can get that right,” he says. “Instead of having 20 POCs, we can have just this one product that’s a really big ROI for us.”