When is the right time to dump an AI project?

Many organizations running AI pilots will inevitably encounter a key IT dilemma: when to pull the plug and move on. If they dump a pilot that’s not meeting expectations too soon, they may miss out on huge benefits down the line, but if they hang on too long, they can waste huge amounts of time, money, and resources.

On the one side, Forrester recently warned organizations not to look for AI ROI too soon, because they could miss out on AI’s benefits.

But AI isn’t cheap, and pilot projects that produce no value can be money pits. For example, a retrieval-augmented generation (RAG) AI document search project can cost up to $1 million to deploy, with recurring per-user costs of up to $11,000 a year, according to Gartner. A medical, insurance, or financial large language model (LLM) AI, built from scratch, can cost up to $20 million.

Of course, AI pilots are panning out at higher rates of late, according to Gartner. In 2022, nearly half of all AI pilot projects failed to make it to production, the analyst firm says, while it expects only about 30% of AI projects to fail next year.

Still, a 30% failure rate represents a huge amount of time and money, given how widespread AI experimentation is today. An EY survey published in July found 95% of senior executives saying their organizations were currently investing in AI.

The question then becomes: When do CIOs or chief AI officers know it’s time to dump an AI project? While there’s no one-size-fits-all answer, AI experts say IT leaders can take steps to ensure they keep the AI projects that make sense and drop the ones that don’t.

Defining success

One of the first steps IT leaders should take in launching an AI pilot project is to define metrics for success — beyond ROI — and set a timeline to check on progress, says Andreas Welsch, an AI consultant and former vice president and head of marketing for AI at SAP.

“The challenge is people keep going down the path of not pulling the plug, because they always hope that there’s a next breakthrough right around the corner,” Welsch says. “A lot of times people don’t have proper goals in place.”

Some KPIs for an AI project could, for example, include increasing customer satisfaction by 10%, reducing the time to fill out a request for proposal (RFP) by 30%, or spending four fewer hours per month paying invoices.

During predetermined checkpoints, IT and business teams can then ascertain progress toward those goals. If a project isn’t hitting the metrics, the teams can decide whether to dump it or give it more time. If a customer service bot improves satisfaction by 7% instead of 10%, maybe it’s worth a continued investment.

In some cases, however, organizations can recover from an unfocused start, adds Adam Lieberman, chief AI officer of banking technology firm Finastra. Sometimes a CIO or CAIO can help a failing project recover by defining a viable roadmap.

“When the aims are open-ended, the project will lack focus and unravel,” he says. “This is the earliest sign that a project won’t work, but it’s also early enough in the process to refocus and establish a more specific end goal.”

One challenge with setting KPIs is measuring the results, adds Kathy Gersch, chief growth and commercial officer at Kotter International, a change management consulting firm. Measuring customer sentiment may not be difficult, for example, but measuring the time an employee saves by using a copilot to draft an email can be more evasive.

“The ROI may be coming from many of these less tangible things,” she says. “You could throw out a project too quickly by saying, ‘Oh, we’re not getting our ROI,’ if you’re not measuring all those things.”

Connect to business needs

In addition to clearly defined KPIs, organizations should tie AI projects to specific business needs, Welsch adds. In some cases, organizations appear to launch AI projects just to do something with the technology. Successful projects, however, address organizational pain points.

“What is that business problem that that we are trying to solve?” he says. “You should be working with your business stakeholders very closely, ideally at the beginning, to say, ‘What are we trying to solve?’”

Many abandoned AI projects fail this basic requirement, Gersch adds.

“If you get a little skunk-works team on the side working on what AI should do, and they don’t connect into the business, it becomes difficult to get adoption,” she says. “That’s where things can get abandoned, either too late or too early, depending on your perspective, because they’re really not connected to the business.”

While Gersch recommends tying AI projects to business goals, she also encourages experimentation. When a project is tied to a business goal, employees may be more likely to embrace it.

“You’re probably going to learn a lot more of what’s possible with the AI once you can get people to actually adopt it, to use it and leverage it,” she says.

Limit the damage

One approach to launching AI projects is to set a limited timeframe, says Arijit Sengupta, CEO of Aible, an AI solutions provider. In many cases, Aible and its customers decide together within two business days whether a project can be viable, he says.

While it’s important to have metrics, IT and business leaders also shouldn’t be too tied to hitting exact goals, Sengupta adds. In some cases, the original goal is too grandiose or isn’t really what the business needs.

“You’re basically imagining what you might want, and you say, ‘Well, if you give me a flying car, I’m going to love it,’” he says. “And then, six months later, somebody comes in with a car that doesn’t quite fly, and you’re like, ‘Well, that’s not what I wanted.’ But that actually wasn’t what you needed; you actually needed a faster boat.”

Six- to nine-month AI pilot projects can be dangerous, he adds. “If you’re going to dump it, you don’t want to dump it in nine months, because then you get into some sunk cost fallacy where people are going try to really make it work, and they’re putting a lot of effort in.”

Some failed projects aren’t total losses, and in some cases, putting a project on hold may be a better alternative than dumping it, adds Lieberman, from Finastra.

“It’s important to note that failure is a necessary part of innovation,” he says. “Progress in the field of AI is rapid. This is why it’s better to pause projects, rather than abandon them entirely, as new capabilities and techniques are emerging all the time.”