88% of AI pilots fail to reach production — but that’s not all on IT

The proof of concept (POC) has become a key facet of CIOs’ AI strategies, providing a low-stakes way to test AI use cases without full commitment.

But as enterprises increasingly experience pilot fatigue and pivot toward seeking practical results from their efforts, learnings from these experiments won’t be enough — the process itself may need to produce more targeted success rates.

Recent research from IDC, undertaken in partnership with Lenovo, found that 88% of observed POCs don’t make the cut to widescale deployment. For every 33 AI POCs a company launched, only four graduated to production, IDC found.

“The high number of Al POCs but low conversion to production indicates the low level of organizational readiness in terms of data, processes and IT infrastructure,” IDC’s authors report. “Half of the organizations have adopted Al, but most are still in the early stages of implementation or experimentation, testing the technologies on a small scale or in specific use-cases, as they work to overcome challenges of unclear ROI, insufficient Al-ready data and a lack of in-house Al expertise.”

Analysts tracking generative AI have found a similar pattern, noting a strong desire among companies to leverage gen AI, but worries about the various errors that make it difficult to take the technology to the next level. 

Moreover, Jason Andersen, a vice president and principal analyst for Moor Insights & Strategy, sees undemanding greenlighting of gen AI POCs contributing to the glut of failed experiments. 

“Gen AI POCs in the enterprise are getting approved much more easily than other technologies in general,” mostly because of CEO and board pressure to do as much experimentation as possible with gen AI, Andersen said. 

“They are saying, ‘I want to know more so I will sign off on a [POC] project to see if and how this applies to my business,’” Andersen said, adding that he sees enterprises deploying “a much larger crop of POCs.”

IT managers are leveraging this trend to try to get greenlights for broader technology efforts, Andersen says. “A lot of efforts are not gen AI, but they are trying to inject some gen AI things into it,” he explains. “They figure, ‘If we throw some gen AI into it, we’ll get it approved.’”

A question of ROI

Reece Hayden, a principal analyst at ABI Research, said his research has also found a soaring number of enterprise POC approvals. “The bar for POCs [for generative AI] has gotten a lot lower” and that is partly because “the cost of developing that gen AI POC is now much lower,” Hayden says.

Companies’ pilot-to-production rates can vary based on how each enterprise calculates ROI — especially if they have differing risk appetites around AI.

“It’s going to vary dramatically. It’s about the risk and their willingness to accept that risk — and the potential lack of accuracy,” Hayden says. “The level of risk is significantly higher [than non-AI projects] and it’s fairly rare that the reward outweighs the risk.”

Brian Jackson, a principal research director at Info-Tech Research Group, is more sanguine about low production rates for AI pilots.

“The whole point of POCs is to experiment. Don’t be afraid to fail the first time. It’s not a waste,” he says. “Even the failures are not failures if there are good lessons learned.”

But while IDC sees systemic IT issues at the root of high pilot failure rates, including insufficient data operations and AI talent, the research firm admits corporate politics is also at play.

“Most of these gen AI initiatives are born at the board level. And a lot of this panic-driven thinking is what caused a lot of these initiatives,” says Ashish Nadkarni, group VP at IDC. “These POCs are highly underfunded or not funded at all. Most of the time the POC happens not because of a strong business case. It’s trickle-down economics to me.”

AI marketing campaigns have caused boards and CEOs to put undue pressure on IT executives to do something with AI now.

“ROI [calculations] are being influenced by a certain level of urgency, a certain level of existential threat. Alarm bells are going off and people are willing to bend the rules on what ROI means. They are freaking out,” Nadkarni says.