Rushing for AI ROI? Chances are it will cost you

Many organizations have struggled to find the ROI after launching AI projects, but there’s a danger in demanding too much too soon, according to IT research and advisory firm Forrester.

Enterprises fixated on ROI will scale back prematurely, the firm predicts, as some IT leaders begin to realize that giving experiments more time to bloom may be more important than expecting a quick turnaround to value.

Nearly half of AI decision-makers say their organizations expect ROI on AI investments within one to three years, while another 44% expect a longer timeframe, according to Forrester’s Q2 AI Pulse Survey, 2024.

“Enterprises are achieving improved customer experience, employee productivity, and even new revenue streams with AI use,” Forrester says in its latest round of AI predictions. “But an AI reset is underway. Obvious use cases that enterprises experimented with last year are now table stakes and embedded in business software.”

Measure everything

Looking for ROI too soon is often a product of poor planning, says Rowan Curran, an AI and data science analyst at Forrester. Organizations rolling out AI tools first need to set reasonable expectations and establish key metrics to measure the value of the deployment, he says.

In many cases, organizations using generative AI for generic, general-purpose tasks aren’t seeing the productivity gains or ROI they expected, Curran says. Targeted AI projects solving problems specific to the deploying organization tend to show more promise.

“There’s a lot of folks who looked at some potential use cases around AI that, when you look at them at a high level, might be very appealing with the increases in productivity or some of other kind of broader impacts,” he says. “Folks were very excited about things like copilots, and for those things, it’s been hard to nail down some specific ROI numbers that can tie directly into business outcomes.”

Curran’s comments on copilots ring true with many CIOs who are not entirely sold on generative AI copilots even as they attempt to differentiate between what’s hype and where to drive results. For others, agentic AI, which focuses more on decision-making than content generation, holds promise as a use of AI that could impact business outcomes.

To demonstrate a specific use case, Curran uses a call center as an example. By adopting an AI agent to assist with calls, a call center may be able to cut call times by 30 to 40 seconds, a huge increase in productivity when employees are handling large volumes of calls per day.

The goal of cutting the time of an average call is measurable and trackable over time, he notes.

Many paths to ROI will take longer, Curran says. “You might have your initial rollout, let’s say it is an internal employee support chatbot responding with 75% accuracy,” he says. “Your target ROI may sit around 85% or 90% accuracy, but there’s no way to get to that without rolling it out and slowly achieving it over time by having people giving feedback and refining the responses.”

ROI will come in step-by-step increments, he adds. “It’s not going to come all at once,” he says.

One challenge for CIOs is deciding when to finally pull the plug on an AI project. The choice depends on the unique circumstances and needs of each organization, Curran says. There’s no formula for CIOs and other IT leaders to follow.

Fear of being left behind

But first, organizations need to understand when AI is the right fit. Part of the problem with abandoned AI projects is that many organizations are jumping in out of the fear of missing out, says Tony Fernandes, chief AI experience officer at HumanFocused.AI, an AI strategy and design consulting firm.

CIOs and other IT leaders are often forced into adopting AI by their boards, and the projects then fail because of a lack of due diligence, he adds. That same pressure can also push CIOs to overstate AI progress when perhaps little promise has been demonstrated.

“Rather than taking the crawl-walk-run approach, I see organizations trying to go from 0 to 60 in microseconds,” says Fernandes, also CEO of UEGroup, a strategic design and insights provider. “It isn’t that they are abandoning AI too early, it is that they are riding into dead ends at full speed because they didn’t take the time to get the lay of the land first and do the methodical experimentation that is needed.”

Organizations achieve ROI with AI when it’s the best tool for the job, he adds. Many companies now rushing to adopt AI will end up going back to more traditional tech solutions in the next five to seven years, he predicts.

“The organizations joining the herd at this stage are using AI as a solution looking for a problem,” Fernandes says. “Most will never see an ROI.”

Like Fernandes, Rob Owen, CIO at accounting and IT advisory firm Sax, has seen some organizations scramble to adopt AI, with some early adopters paying a premium to rent GPUs and other infrastructure.

“We saw a lot of projects stop and start where they would say, ‘These costs are getting out of control,’ because they’ve underestimated the time and the resources, from a technical perspective, that it would take to get it done,” he says. “We saw a lot of projects get abandoned.”

Start small — and cost-effective

Since the early days of commercially available gen AI, many AI services have become available, giving companies reasonably priced options, Owen says. Sax has deployed AI on several internal projects, including its help desk functions, with the company training and customizing AI models itself.

“My approach is tinker, leverage AI in the most cost-effective way you can,” he says. “And once you have a proof of concept, a working model, then expand. Don’t start big and hope for the best, unless you’ve seen someone else do it, or you have a proven model to start with.”

While Sax measures its AI projects by tracking KPIs, Owen believes it’s often irrational to expect ROI immediately. Most AI projects will take 18 to 24 months to achieve ROI, he says.

“If you have a good IT staff, they’re going to solve problems and figure things out,” he says. “The projects can’t all solve a business problem for me immediately. You have to come up with fun ways to get your smartest people to start playing with the stuff.”