CIOs rethink public cloud as AI strategies mature

CIOs have begun to rethink their reliance on the public cloud for AI and other workloads, with resurgent interest in private cloud and on-premises environments accelerating.

While the public cloud provides the flexibility to spin up large numbers of GPUs for AI experimentation, CIOs are looking to private cloud or on-premises environments as their AI strategies mature, and settle into predictable AI workloads, to limit spending and protect data privacy, says Greg Whalen, CTO of data observability firm Prove AI.

A recent Prove AI survey of 1,000 enterprise leaders in the US and Canada found that 67% of them plan to migrate some AI data to non-cloud environments within the next 12 months. In addition to cost predictability and data privacy concerns, the top reasons for the move include security issues and cloud integration challenges with SaaS environments, according to the survey.

Organizations running consistent AI workloads can save money by buying a couple of GPUs or installing a few at their private cloud provider instead of renting time in the public cloud, Whalen says. If IT leaders can accurately estimate their needs, in-house GPUs will get plenty of use, with little down time, he contends.

“If you’re really doing fine-tuning, or even if you’re just trying to customize a RAG [retrieval augmented generation] model, you probably need hours of continuous GPU compute,” he says. “Your workloads are not very spiky, even with the actual evaluation of the model, the running of the model.”

Whalen doesn’t see many organizations running their own GPUs with the problem of underuse, he says.

“If somebody says, ‘You have a GPU, and you’re probably only going to use it 10% of the time,’ in our experience, that hasn’t been the case,” he adds. “You’re going to find things to do with it, and often most of the workload is training, which is very continuous. It’s something you run in predictable amounts of time.”

Growth in private cloud spending

While the Prove AI survey shows interest in on-prem computing, a second survey shows significant growth in private cloud spending, even as spending on the public cloud also increases at a lower rate.

The survey, for networking and security provider GTT Communications, shows 12% growth between 2024 and 2025 in the number of organizations planning to spend more than $10 million on the public cloud.

But the percentage of respondents planning to spend more than $10 million on private cloud services grew even faster, from 36% in 2023 to 43% in 2024 and to 54% in 2025, according to the survey. That’s double the growth rate of big spenders on the public cloud.

GTT found that more than half of all AI workloads now reside in a combination of private cloud and on-premises environments, with security, compliance, and the specific needs of AI workloads as the major reasons for seeking alternatives to the public cloud.

Regulatory and compliance concerns are a big driver toward the private cloud or on-premises solutions, says Bastien Aerni, vice president of strategy and technology adoption at GTT. Many companies are shifting their sensitive workloads to private clouds as a piece of broader multicloud and hybrid strategies to support agentic AI and other complex AI initiatives, he adds.

“Most of the time, AI is touching confidential data or business-critical data,” Aerni says. “Then the thinking about the architecture and what the workload should be public vs. private, or even on-prem, is becoming a true question.”

The public cloud still provides maximum scalability for AI projects, and in recent years, CIOs have been persuaded by the number of extra capabilities available there, he says.

“In some of the conversations I had with CIOs, let’s say five years ago, they were mentioning, ‘There are so many features, so many tools,’” Aerni adds. “Now when I’m having the same conversation, they say, ‘Actually, I’m not using those tools that much now.’ They are all looking for stability and predictability.”

A minor exodus

Other cloud and AI experts don’t see a huge exit from the public cloud, with growth still happening because of the high computing demands of AI. A huge percentage of enterprises are still using hybrid cloud models, says Danilo Kirschner, managing director of cloud consultancy Zoi North America.

Repatriation is happening, but organizations aren’t completely abandoning the public cloud, he says.

“The paradox is clear: AI workloads are driving both massive cloud growth and selective repatriation simultaneously, because the market is expanding so rapidly it’s accommodating multiple deployment models at once,” Kirschner says. “What we are seeing is the maturation from a naive ‘everything-to-the-cloud’ strategy toward intelligent, workload-specific decisions.”

Zac Engler, chief AI officer at IT staffing consulting firm C4 Technology Services, sees the same trend.

“We’re not watching a mass exodus from the cloud,” he says. “It’s more like companies are quietly sneaking out the side door with their most valuable AI workloads.”

Trust, cost, and control of data are back on the boardroom agenda and influencing decisions about where AI workloads run and data is stored, he adds.

“The public cloud is still great for experimentation, scaling fast, and looking impressive in a board deck,” Engler says. “But when it comes to proprietary data, compliance, or not burning cash needlessly, on-prem and private setups start to make a lot more sense.”