6 hard truths of generative AI in the enterprise

With the adoption of generative AI skyrocketing, the technology is eliciting a myriad of feelings — from excitement and anticipation to stress and concern — inside organizations.

A May global McKinsey survey puts generative AI usage at 65% of organizations, nearly double the firm’s previous survey 10 months ago. And with that rise, use case are proliferating.

As with past technology adoption patterns, most are starting to use genAI in areas that provide tactical benefits, such as improving their existing processes and reducing costs, says Jim Rowan, a principal at Deloitte Consulting. This approach helps derive value from “low-hanging fruit” while building knowledge, experience, and confidence with a new technology, Rowan says.

Still, organizations are at vastly different phases of genAI usage. Some early adopters are now scaling pilots to production by linking multiple use cases together to show tangible efficiency benefits. Others have been investing in fluency, trying some proofs-of-concept, and largely looking to adopt AI technology embedded with third-party software, Rowan says. Another camp has taken the wait-and-see approach.

While organizations figure out how to go from learning about generative AI to starting pilots to deploying full-scale implementations that can be truly transformational, here are six hard truths IT leaders have gleaned so far.

1. Tech talent remains the No. 1 barrier to adoption

Organizations that reported a high level of AI expertise (33%) tended to feel more positive about generative AI, yet they too feel more pressure to adopt the technology, seeing genAI as more of a threat to their business model, according to a Deloitte report on genAI.

“This reveals that even organizations who consider themselves well-versed in AI are preparing for potential headwinds,’’ says Rowan.

As a result, IT leaders, even those who report high levels of AI expertise on their teams, are rethinking talent strategies around AI, with upskilling playing a vital role in attempting to bridge AI skills gaps.

2. GenAI’s bottom-line impact is a ways away

While noting that “the honeymoon phase is over,” there are likely few genAI projects generating bottom-line impact right now, according to Aamer Baig, a senior partner and global leader at McKinsey, who spoke about the hard truths of GenAI at the MIT CIO Consortium in May.

Only 15% of the companies McKinsey surveyed have a line of sight into earning improvements from genAI initiatives, Baig said. “Not all use cases are equal. Not a lot add value.” He suggested that organizations become focused on “initiatives that solve real business problems that are technologically feasible and have little risk.”

The Deloitte report found that 48% of organizations do not expect to see a transformation from genAI for one to three years.

“I kind of feel like we’re in a big science fair and designing hypotheses, and eight out of 10 are failing.”

— Marc Kermisch, global chief digital and information officer, CNH Industrial

UK-based CNH Industrial was an early adopter of Microsoft’s Copilot and IT began creating its own large language models (LLMs) in November 2023, says Marc Kermisch, global chief digital and information officer.

“We definitely have learned there’s plenty of constraints,” Kermisch says, adding that “it’s not the panacea we all thought it would be.”

While there have been some productivity gains, Kermisch expected that they would be more significant in areas such as invoice processing. GenAI is “primarily what I’m going to call an information synthesizing tool,” he says. Off-the-shelf genAI tools have “not been effective at all for any financial or number analysis at this time,” he says.

Another issue is a proliferation of tools and technologies, which Baig said needs to be tamped down. “One of the top obstacles to achieving generative AI at scale is too many genAI platforms,’’ he said.

“Not a week goes by without another new tool that is mind-blowing in its abilities and potential future impact,’’ agrees David Higginson, chief innovation officer and executive vice president of Phoenix Children’s Hospital. But right now genAI “can really only be executed by a small number of technology giants rather than being tinkered with at a local skunkworks level within a healthcare organization,’’ he says. “Therefore, it feels as if we are in a bit of a paused state waiting for established vendors to deliver mature solutions that can provide the tangible value we all anticipated.”

Higginson says effective tools will emerge to solve real problems for healthcare providers. When that happens, it “will force people to make uncomfortable decisions between risk tolerance, cost, provider satisfaction, and patient outcomes,” he says.

3. Legal questions put IT in a state of limbo

The two factors Higginson believes are slowing down the delivery of platforms at scale are the legal and regulatory unknowns of relying on the technology and the high cost of being an early adopter.

“While the legal jeopardy of using this technology remains murky, the vendor costs and the media-fed paranoia about ‘getting something wrong’ remains high,’’ he says. “As a result, many administrators are electing against being the first test case for this brave new frontier of technology — especially when they have an unclear notion of how the technology functions and cannot provide attorneys any guarantees on how it actually works.”

The Deloitte study found that compliance (28%) and governance issues (27%) are considered barriers to AI adoption. Less than half (42%) of respondents agreed they have done enough to govern generative AI adoption and mitigate its potential risks, according to the report.

“This shows a lot of uncertainty in terms of how AI will be regulated over the coming year, especially for global organizations operating in multiple regions,’’ Rowan observes. “Looking at the bigger picture, the challenges that generative AI poses in corporate governance and risk parallel those in society governance and risk.”

More than half of the study’s respondents expressed concern that the widespread use of generative AI will centralize global economic power (52%) and increase economic inequality (51%).

“In both realms, the technology’s potential benefits and potential harms, are high,’’ Rowan says. “National organizations and governments will need to strike a balance in terms of making sure generative AI benefits are broadly and fairly distributed, without overly hindering innovation or providing an unfair advantage to companies with different rules.”  

4. Managing costs is a huge issue

Echoing Higginson’s concern about the cost of implementing genAI, McKinsey’s Baig stressed that organizations must manage costs — before costs manage them. This is due to the high compute intensity and high change management genAI requires.

He recommended that organizations invest as much in these areas as they do to bring about digital transformation because genAI will require changes in workflows, business processes, and new KPIs. Organizations also have to factor in risk and hallucination training, and budget for ongoing maintenance, Baig said.

The fundamental barriers to adopting genAI are the scarcity and cost of the hardware, power, and data needed to train models, Higginson says. “With such scarcity comes the need to prioritize which solutions have the broadest appeal to the population and can generate the most long-term revenue,’’ he says.

CNH’s Kermisch had high hopes that genAI “could help us bend our cost curves,’’ but that hasn’t happened. Platforms are very expensive to implement and the more genAI is utilized, the more expensive it becomes, he says.

“If I could gain the investment back in productivity, great, but we’re not seeing it translate one for one,’’ he says. CNH Industrial has a limited number of Copilot licenses and IT rotates them among the employees who are typically excited about using new technology.

This tends to peter out. “Usually within 30 days, that utilization drops off dramatically,’’ Kermisch notes. “There’s definitely the [phenomenon of] the shiny new tool. They’ll open up an Excel spreadsheet and try to use Copilot and realize quickly they can’t use it.”

Copilot is effective when the right prompt is used, he adds. “The more precise your prompt the better.” However, “We have yet to find the power user who uses it.”

Brian Olsson, vice president and CIO at Briggs & Stratton, has done a sizable rollout of Google Gemini and says that while the cost will be somewhat of an issue, more concerning is figuring out how to monitor the adoption and use of it. Like Kermisch, Olsson says this will help IT determine whether the tool is a good investment.

“The thing about the AI space is it’s evolving so fast, we’re trying to be nimble and we’re rolling it out trying to see efficiencies,’’ he says. “We’re looking at the financials and the adoption [rate] but we’re not putting up so many roadblocks that we get left behind.”

5. Getting the right data is hard

Right now, many organizations are challenged by the need to use high-quality data, because genAI models require vast amounts of accurate, relevant data to perform effectively.

Organizations should not focus on finding the perfect data because that is a “huge, daunting challenge,’’ said McKinsey’s Baig. Many organizations are taking a top-down look at data, and “that ends up looking like a massive spreadsheet,’’ he said. “Focus on data that helps you with multiple use cases.”

While research and development continue to push the needle on what genAI can do, “we know that data is a critical aspect to enabling AI solutions and we also recognize that many organizations are uncovering the work it will take to build the right data foundations to support scaled AI deployments,” says Deloitte’s Rowan. 

CNH’s Kermisch says they have learned that creating LLMs using documents or text-based repositories of data has “a relatively high value and is relatively easy to turn around quickly.”

Eventually, he says, CNH executives believe genAI will have a significant impact on their business, but it is early days. “We thought we’d see huge improvements in our ability to design vehicles and reduce costs by automating tasks in a few months,’’ Kermisch says. “None of that has really come to fruition. We’re seeing positive outputs and positive outcomes — but in limited use cases.”

Consequently, the approach Kermisch is taking is to train a group of tech leaders on the technology and partner with Microsoft to provide training for the broader employee base. “We’re encouraging experimentation and allowing business teams to get out and learn,’’ he says. “I kind of feel like we’re in a big science fair and designing hypotheses, and eight out of 10 are failing. As long as they fail quickly, that’s okay for us right now.”

6. It’s here to stay

Despite the growing pains, IT leaders recognize that genAI isn’t going anywhere.

“One thing is for sure, it’s going to be a bumpy ride with some spectacular successes and failures, and I cannot wait to see how the industry changes over the next five to 10 years,” Phoenix Hospital’s Higginson says.

Michael Corrigan, CIO of World Insurance, says that while genAI is powerful and evolving very rapidly, it is maturing very slowly. There is also a lot of hype and misnomers about it, he says.

“It definitely requires a strategy and roadmap to be implemented properly for it to have a positive impact on your business, to be able to enhance your capabilities, and reach your business goals,’’ Corrigan says.

It also requires organizations to establish use cases and the tools they want to use because shadow genAI is creeping in.

“Even if a company hasn’t rolled out a particular AI tool, employees are out there using ChatGPT and all sorts of third-party AI tools because it is making them more efficient,’’ says Briggs & Stratton’s Olsson. “The hard truth is, if you don’t start giving them tools, they’re going to find them; … even if they’re not doing anything with AI, the data risk is there. It’s a new information security risk.”

Dave Pawlak, executive director of IT at Consumers Energy, agrees, saying genAI must be implemented securely “and it’s not as easy as what the public is experiencing with OpenAI or other [open] generative AI tools.”

Still, even with all these hard truths, Kermisch, Pawlak, Baig, and others say there is value in implementing genAI quickly, safely, and at scale.

“It will allow you to go from pilot to scale,’’ Baig said. “Unlike other digital disruptions, I believe we’re in a phase of genAI where a level of investment is needed,” as well as a better understanding of the technology. “You have an incredible opportunity to take advantage of this and it makes it even more important for CIOs to play a leading role in leading organizations forward.”