Where’s the ROI for AI? CIOs struggle to find it

Finding the return on investment for AI remains elusive for many organizations, even as they rush to adopt the technology.

Difficulties estimating or demonstrating the value of AI technologies to the business have been or will be a top barrier to their implementation, according to a recent Gartner survey of more than 700 IT leaders at organizations that have adopted or plan to adopt AI. Nearly half of respondents pointed to challenges with showing its value.

Other top concerns include a lack of talent and skills among employees and a lack of confidence in AI technologies. Even with the questions, adoption is widespread, with more than 95% of IT leaders saying AI has been implemented in at least one business process. More than a quarter of those responding have adopted AI for multiple business processes across several business units.

Experimenting with the novelty

Despite the heavy adoption, CIOs’ concerns about the value of AI doesn’t surprise Ryan Kane, owner of IT managed services provider Soaring Towers. While Kane shows clients how to save time and money using AI tools like Microsoft Copilot, many SMB customers still don’t see the value of generative AI in tasks like writing a newsletter, when the AI doesn’t have access to their internal data.

“They need to be shown how to really use the tool in order to use it effectively,” he says. “A chisel in the hands of a trained professional can create amazing things; a chisel in the hands of an amateur can be a lost opportunity.”

The AI market is still developing, and some companies are adopting the technology without a specific use case in mind, he adds. Kane has seen companies roll out Microsoft Copilot, for example, without any employee training about its uses.

“I have found very few companies who have found ROI with AI at all thus far,” he adds. “Most companies are simply playing with the novelty of AI still.”

The concern about calculating the ROI also rings true to Stuart King, CTO of cybersecurity consulting firm AnzenSage and developer of an AI-powered risk assessment tool for industrial facilities. With the recent red-hot hype over AI, many IT leaders are adopting the technology before they know what to do with it, he says.

“I think back to the first discussions that we had within the organizations that are working with, and it was a case of, ‘Here’s this great new thing that we can use now, let’s go out and find a use for it,’” he says. “What you really want to be doing is finding a problem to solve with it first.”

As a developer who has integrated AI into his own software, King is not an AI skeptic. He believes there can be a return on investment if organizations are smart about where they focus their AI efforts.

“Ultimately, like any new tool that you implement in any environment, essentially it comes down to learning it and knowing how to get the best out of it,” he says. “Once you’ve gotten over that first hurdle and that first learning curve, there are a lot of problems that AI can solve for you.”

First, find the use case

The Gartner survey found generative AI to be the most adopted AI-related technology, with 28% of the respondents saying their organizations had rolled it out. Close behind were machine learning and natural language processing.

Gartner expected the concern about value, given that it was also a top barrier to entry in a similar 2021 survey, says Leinar Ramos, a Gartner analyst focused on generative AI. Tracking and demonstrating ROI can be difficult for many use cases, he adds.

Gartner recommends CIOs thinking about deploying AI to first consider potential use cases, establish metrics to measure value, and run pilot programs before launching large-scale projects. The second-highest barrier to adopting generative AI specifically was the ongoing cost, Ramos notes.

As the number of users of generative AI increases, some organizations are surprised by the increasing costs.

“We, in our conversations with clients, are hearing the question around value very, very often,” Ramos adds. “There’s a big difference between piloting some of the generative AI use cases with a few users and in a more controlled environment, to then scaling that up.”

And with researching and implementing AI having shot up to No. 4 on CEOs’ priority lists for IT leaders, according to CIO.com’s State of the CIO Survey, the pressure on CIOs to flip that switch will only rise.

Tracking the value

One company that has embraced AI in a thoughtful way is CNH Industrial, a farming and construction equipment manufacturer.

CNH uses AI in equipment health monitoring systems, enabling equipment owners to get notices about possible maintenance issues. The company also uses AI to assist in code development and to build autonomous driving systems, says Marc Kermisch, global chief digital and information officer. CNH first unveiled an autonomous tractor concept in 2016.

A small number of CNH employees are using Microsoft Copilot to record and summarize meetings, and the company has also built an AI-driven chatbot that gives service technicians instant access to equipment manuals and repair information.

“Now instead of having to go to a PDF file, a three-ring binder, or pull out the big book, they can literally say, ‘Hey, I’m working on a T7 tractor and I need to replace the oil filter,’” Kermisch says. “They can automatically pull up the page with the illustrated instructions, and they can either do it on their phone, or they can send it to a printer, however they want to do it.”

Even though CNH has fully embraced AI, the company has wrestled with finding the ROI with AI, he adds. Employees may save time by reading a summary of meetings they missed, but unless they manually note it, that time savings may be lost in the ether.

For example, Kermisch and the CNH tech team have set a goal of saving 10,000 hours by using GitHub Copilot, the AI-driven coding tool. But to capture that time savings, CNH is asking developers to manually note any time they saved.

The value of the chatbot tool for service technicians will be easier to measure, he says. CNH will look to its Net Promotor Scores to see whether customer satisfaction improves with the wider rollout of the chatbot.

“With that service technician tool, we’ve had a very high NPS score for those that have utilized it,” Kermisch says.  “As AI tool gets broader adoption, we would expect to see the global NPS from our dealers to go up.”

These efforts to measure the value of AI are important, he adds. “It’s very easy for organizations to be enamored with new tech, when they don’t have a hard outcome they’re trying to strive for,” he says.

Echoing Kane and King, Kermisch recommends that organizations first educate themselves about AI and its potential benefits before diving in.

“Increase the rigor of the use case you’re going after, and make sure that you can identify it,” he says. “If I believe I can reduce 10,000 man-hours, that’s a hypothesis. What experiments am I going to run to prove that or disprove that?”