What CIOs need to know about measuring AI value

There’s a wealth of information about the high failure rates of AI projects while, simultaneously, an equally large number of surveys show that many, if not most, companies are already reporting positive business benefits and ROI from their AI investments.

For example, a global study of 2,000 CEOs IBM released in May says only 25% of AI initiatives delivered expected ROI, and only 16% have scaled enterprise-wide. In another, Wakefield Research, on behalf of Informatica, found 67% of 600 business leaders say they’re unable to successfully transition even half of their gen AI pilots to production, and 97% say they’re having trouble demonstrating the business value of their pilots. But other surveys seem to show the opposite.

In April, Enterprise Strategy Group surveyed 1,900 business and IT leaders, and, of those who deployed AI solutions, 92% said their investments have already paid for themselves. And for the over 1,200 respondents who have quantified the ROI of their gen AI initiatives, the average return is 41% through increased revenues, reduced costs, or both.

Then there are surveys that show both things simultaneously. In February, an IDC survey of nearly 3,000 IT and business decision makers, conducted on behalf of Lenovo, showed that 68% say that their AI initiatives have met expectations, and an additional 26% said they exceeded them. But only four out of 33 AI projects reached production, for a failure rate of 88%.

So what’s actually happening? The problem lies in the issue of how we measure success or failure of AI projects. And since AI is rapidly becoming a top priority for CIOs, if it isn’t already, it’s critical for CIOs to understand where the value lies, how to measure it, and which metrics have meaning for the company. When it comes to business disruption, numbers don’t always rule.

Pilots are supposed to fail

Let’s start with the headline-making data point that major AI projects fail. The implication is that this is a bad thing. But POCs and pilot projects aren’t supposed to succeed every time. Their objective is to weed out the least-promising ideas so companies can focus on the most promising ones. High failure rates mean companies are experimenting and trying out a lot of different possibilities, so failure rates are a good thing.

The more projects that fail in the early stages, the more likely it is that when projects do go into full production, they’ll provide positive business value. And the more companies find success with particular projects, the more likely they are to run even more experiments, and find more opportunities to create value.

In fact, if all of a company’s POCs and pilot projects go into production, that could just mean the company isn’t being creative, innovative, or experimental enough with their AI ideas. Looking at the number of projects that go into production is not a good metric of success, no matter how much pressure is coming from the board and other senior executives, business unit leaders, employees, partners, or customers.

So it’s not a contraction that most pilots fail while companies also report positive ROI. This isn’t a bug. It’s a feature. The goal is to have the right AI projects in production, not the most. And the biggest mistake a company can make is to skip straight to full deployment, without a pilot project or adequately vetting the results of the POC, says Rosha Pokharel, chief AI architect at UST Healthproof, a healthcare operations company.

“Proof of concept usually only works on a few hundred data points,” she says. “But when we scale that to production, we need to prove it’s actually working and delivering the level of accuracy that meets business standards and user requirements in millions of data points.”

If that data problem isn’t solved, the project fails, and the company wasted all the money they spent scaling it up. Similarly, if the AI designers didn’t fully consider user needs in their POC, and they roll it out at scale and nobody uses it, that’s also a huge cost.

Companies also need to be careful about where they start to measure ROI, she says. “We shouldn’t factor in the investment that was done on the proof of concept,” she says. “POC isn’t the place to compute ROI. It’s the fail fast zone. We want to understand what is feasible, viable, usable, and valuable, and what can scale. That’s the goal of the POC, and the budget for it comes from R&D or innovation labs.”

That, at least, is how it’s been done at the companies she’s worked, she says. But she’s not the only one who believes in rapid experimentation, and cutting losses early.

“The mantra was ‘fail fast,’” says Eric Johnson, CIO at PagerDuty, an incident response company, which has deployed gen AI and AI agents for both internal operations and in its products and services. “We’d spin things up, and things that worked well — great, let’s keep moving with them.”

But there were also ideas that didn’t work. “A lot of it was picking poor use cases, or we didn’t fully understand the amount of work it took to build something, or it didn’t result in a tremendous amount of impact.”

Build vs. buy

To reduce pilot project costs, PagerDuty decided not to build the technology from scratch. In fact, commercial solutions, instead of having an up-front cost, often come with free trials. Or free trials can be negotiated.

“If we want to spin up a new AI agent, we’ll say we’re not paying for it until it actually proves out to be of value,” says  Johnson. “And many of these vendors are willing to have that conversation, because they know if they can’t prove the value, you’re not going to continue to renew the contract. So they want to make sure you’re getting the value you expect.”

Another company that didn’t build from scratch and found immediate value in commercial products was Flexential, a colocation company. “I’m using off-the-shelf services that adhere to our AI policy, which means they’re not training public models,” says Ryan Mallory, the firm’s COO.

Plus, using AI features of existing products, ones which are part of well-established workflows, makes it easier to calculate ROI since there’s already an existing baseline to measure against, especially when it comes to customer support or sales.

“Everything we’re doing is providing us direct ROI, meeting our predictions, or exceeding them,” Mallory says.

In particular, most of the new AI tools are plug-in modules to the company’s existing tech stack. That makes them very easy to deploy and scale. “We’ll eventually get to our own model,” he adds. “We’re probably 18 months away from that.”

The key metric: the customer

The best ROI metric is, of course, impact to the bottom line. However, when there are multiple AI projects, as well as many other initiatives going on, plus changes to the broader business landscape and the economy, it can be hard to tease out the effect of any one individual project.

According to a March report from McKinsey, only 17% of companies say that 5% or more of their EBIDA is attributable to gen AI, while more than 80% report no tangible impact from gen AI. So companies are forced to use proxy metrics. The best one? Customer satisfaction, retention, and word-of-mouth recommendations.

Flexential, for example, has been deploying gen AI for multiple sales and customer support functions. “It’s absolutely bottom-line,” says Mallory. “We’re seeing a reduction in churn. When your customer satisfaction goes up, and your ability to address asks, requirements, or challenges is quicker, then your customers don’t have as high a propensity to leave.”

And since the company added AI-powered customer support agents to its workflow, response times fell by an average of 20%, and there was a 25% increase in the mean time required to resolve tickets.

They’re not the only company to find AI value in customer satisfaction. According to the IBM survey, 65% of CEOs say establishing and maintaining customer trust will have a greater impact on their organization’s success than any specific product and service features. And, across industries, customer loyalty is the key differentiating factor that drives ROI.

Cost reduction

But in the short term, reducing costs is a quick win for many companies, and the main priority of early adopters cited by 51% in the ESG survey. And it’s working: 88% say they’ve already seen material improvements in efficiency.

“Some clients are especially focused on cost right now because there are a lot of pressures in various sectors,” says David Martin, senior partner at Boston Consulting Group. “And they do have an obligation to return both near-term shareholder value, as well as invest in the future.”

There are many areas where companies can find productivity improvements and translate them directly into dollar value, he says.

“In something like customer service, if you can deflect more calls, empower your agents to do their work faster, or if you’re paying a third party for the call center costs, then that immediately translates to dollars,” Martin says.

According to the BCG survey, 47% of employees say they save more than an hour a day with gen AI, and most use that time to get more work done. In addition, 44% say they work on strategic tasks and 34% pursue professional development. But many companies are doing both at the same time, says Martin, reducing costs and finding productivity benefits while also investing in the future. “They’re really thinking about the implications of AI in the enterprise,” he says.

Business growth

There’s only so much value a company can gain when cutting costs. But there’s a nearly unlimited upside if a company can find ways to generate new business or enter new markets.

In the ESG survey, the third-biggest motivator for companies adopting gen AI is improving innovation outcomes. In fact, 84% say AI is already accelerating their pace of innovation. This can result in new business opportunities, which can be quantifiable in some cases, even in the short term. For example, Flexential is able to generate more leads by having AI chatbots on the company website that are associated with particular types of content.

“We’re seeing about five times the number of meetings booked for prospective customers,” says Mallory. “Then we have that tied into the pipeline and close ratio for the opportunities. So we have very quantifiable data and it’s working very well for our teams.”

Flexential was also quick to roll out AI sales support functionality for its sales staff, who now have access to meeting notes, revenue intelligence, and other resources they need to get new business.

“We’re getting a higher productivity level from our reps when they have the right information,” he says. And the close ratios are also higher when reps are able to use AI to better anticipate how they can serve the customer.

Overall, more companies should focus on potential growth, says Todd Lohr, head of ecosystems at KPMG, and about the risk of AI-powered disruption.

“AI is coming for your business model,” he says. “It’s disrupting your business and barriers to entry. You should think about it more on the business model side of the equation than the operating model for long term sustainability, success, and viability of the organization.”

According to the IBM survey, 68% of CEOs say AI changes aspects of their business that they consider core. These changes aren’t necessarily reflected in immediate ROI, but are critical to the long-term survival of the company as business models change. And a lot of those changes are caused by AI. So it’s no surprise that 61% of CEOs say competitive advantage depends on who has the most advanced gen AI.

“Our clients are no longer asking if AI will transform their business,” says Lohr. “They’re asking how fast it can be deployed. This isn’t just about technology adoption, it’s about fundamental business transformation that requires reimagining how work gets done and how it’s measured.”

Embrace uncertainty

The truth is, CIOs often don’t know what the ROI of their AI projects is, says Jackson Ader, equity research analyst at KeyBanc Capital Markets. In his research on CIOs, he found that ROI expectations were all over the place.

“I don’t think CIOs are spending many millions of dollars on artificial intelligence and expecting an ROI in the single digits,” he says. “I think what’s probably more likely happening is we’re still in an experimental phase.”