CIOs are rethinking how they use public cloud services. Here’s why.

Over the past few years, enterprises have strived to move as much as possible as quickly as possible to the public cloud to minimize CapEx and save money. Increasingly, however, CIOs are reviewing and rationalizing those investments. Are they truly enhancing productivity and reducing costs?

“In the rush to the public cloud, a lot of people didn’t think about pricing,” says Tracy Woo, principal analyst at Forrester. And for some organizations, annual cloud spend has increased dramatically. “Cloud spending is going up and budgets are tightening, so they’re asking what’s going on and how do we right this ship.”

In 2025, the plan, according to Ron Hollowell, SVP and CTO at Reinsurance Group of America (RGA), is to focus on right-sizing their public cloud footprint by maturing processes around work intake, distribution criteria, and implementation practices across private and public clouds. “Expense optimization and clearly defined workload selection criteria will determine which go to the public cloud and which to private cloud,” he says.

As VP of cloud capabilities at software company Endava, Radu Vunvulea consults with many CIOs in large enterprises. “This will be a year when we talk more about hybrid cloud, multi cloud, and repatriation to on-premises,” he says. The reasons include higher than expected costs, but also performance and latency issues; security, data privacy, and compliance concerns; and regional digital sovereignty regulations that affect where data can be located, transported, and processed.

“The primary driver for leveraging private cloud over public cloud is cost,” Hollowell says. He sees public cloud as the most cost efficient for seasonal or bursty, on-demand workloads. “For workloads with more consistent capacity demands, the economics can be more attractive for private cloud and fixed-capacity solutions.”

For many other CIOs, the primary motivator is cost as well, says Vunvulea. While up to 80% of the enterprise-scale systems Endava works on use the public cloud partially or fully, about 60% of those companies are migrating back at least one system. “We see this more as a trend,” he says.

Where are those workloads going? “There’s a renewed focus on on-premises, on-premises private cloud, or hosted private cloud versus public cloud, especially as data-heavy workloads such as generative AI have started to push cloud spend up astronomically,” adds Woo. “By moving applications back on premises, or using on-premises or hosted private cloud services, CIOs can avoid multi-tenancy while ensuring data privacy.” That’s one reason why Forrester predicts four out of five so called cloud leaders will increase their investments in private cloud by 20% this year.

That said, 2025 is not just about repatriation. “Private cloud investment is increasing due to gen AI, costs, sovereignty issues, and performance requirements, but public cloud investment is also increasing because of more adoption, generative AI services, lower infrastructure footprint, access to new infrastructure, and so on,” Woo says.

Hidden costs of public cloud

For St. Jude’s Research Hospital, the public cloud is a good way to get knowledge into the hands of researchers who aren’t part of their ecosystem today, says SVP and CIO Keith Perry. The hospital uses on-prem supercomputers to generate much of its research data, and the movement of that data into and out of the public cloud can become expensive. “The academic community expects data to be close to its high-performance compute resources, so they struggle with these egress fees pretty regularly,” he says.

But data-heavy workloads can be expensive, especially if constant, high-compute is required. “Another driver is data movement, not only in terms of dollars but in performance,” Hollowell says. “So we carefully manage our data lifecycle to minimize transfers between clouds.”

Woo adds that public cloud is costly for workloads that are data-heavy because organizations are charged both for data stored and data transferred between availability zones (AZ), regions, and clouds. Vendors also charge egress fees for data leaving as well as data entering a given AZ. “So for transfers between AZs, you essentially get charged twice, and those hidden transfer fees can really rack up,” she says. And Vunvulea says the cost of data transfer, especially in terms of petabytes, is high, and data transfer and synchronization can be complex. “We’ve seen AI projects where around 45% of cloud costs are generated by moving data from the public cloud to another location,” he says. “And if you put the full systems in place with everything you need around the service, you can have a solution that costs three or four times more than the initial estimation.”

For example, organizations that build an AI solution using Open AI need to consider more than the AI service. Adding vaults is needed to secure secrets. Security appliances and policies also need to be defined and configured to ensure that access is allowed only to qualified people and services. Secure storage, together with data transformation, monitoring, auditing, and a compliance layer, increase the complexity of the system. Around the AI service, you need to build a solution with an additional 10 to 12 different cloud services that fulfill the needs of an enterprise system.

Jeff Wysocki, CIO at mining firm Mosaic Company, acknowledges those budget-busting concerns, but he says CIOs may be able to work with their public cloud provider to get those costs under control. For example, Mosaic recently created a data-heavy Mosaic GPT safety model for mining operations on Microsoft’s Bing platform, and is about to roll that out in a pilot. It contains years of safety information that Mosaic built into the model, so contractors working at a mining site can enter questions around safety and see how to handle a given situation.

“We made changes to our architecture to get around the cost issues,” he says. How Mosaic’s team built the models, as well as how Microsoft architected the solution, helped to keep the project within budget. “We made some changes with Microsoft to get the cost down to something we can consider a reasonable return.”

Mosaic’s ERP system initially resided in a private cloud but now runs in an SAP private cloud, says Wysocki. But, he adds, some servers will always be on premises, and that’s unlikely to change, although there may be edge server solutions with cloud synchronization. “I don’t see that evolving too much beyond where we are today.” Between 80 and 85% of the company’s IT operations are in the cloud, and he expects it to stay that way.

AI projects can break budgets

Because AI and machine learning are data intensive, these projects can greatly increase cloud costs. Organizations don’t have much choice when it comes to using the larger foundation models such as ChatGPT 3.5 and 4.0 because the scale of compute power required would be too costly to reproduce in house, says Sid Nag, VP, cloud, edge, and AI infrastructure services and technologies at Gartner.

By 2027, however, more than 50% of the gen AI LLMs enterprises use will be industry-specific, Gartner predicts. These will be a much smaller carve-out of the very large-scale general-purpose foundation models, and could be run elsewhere. Even after organizations use tools such as RedHat’s InstructLab to augment those industry-specific models with company-specific data, they’re still small by comparison. “Industry-specific models…require fewer resources to train, and so could conceivably run on on-premises, in a private cloud, or in a hosted private cloud infrastructure,” says Nag.

But, says Vunvulea, the computation power and infrastructure needed to train or optimize the model isn’t easy to find or buy on prem. “Computation needs are one of the most important factors,” he says. Fortunately, cloud vendors also offer off-the-shelf AI platforms that enterprises can use to train their models against their own data. “So you don’t need to configure the on-premises system, even if you decide to run it there.”

But should you? “I’d be cautious about going down the path of private cloud hosting or on premises,” says Nag. “Decision makers with fiduciary responsibility are going to balk at the idea of going back to the days of CapEx unless there are compelling reasons to do so.”

Cloud vendors continue to provide more AI and ML services as part of their platform-as-a-service offerings, Vunvulea says. You start with a pretrained model, bring your own data, and just use the service without any problems. “We’re getting close to the point when the models available from public cloud vendors are mature enough to cover up to 90% of the standard needs of most companies,” he says. The question as to whether to use those services or not will come down to cost: Do the numbers make sense for your business model?

Inexpensive but underperforming

At first, says Woo, CIOs focused on reducing cost, but that doesn’t always align with performance considerations or end goals. Even when the public cloud is the less costly option, it may not be the best fit if potential latency or other performance issues are factored in. That’s particularly true for industries that can’t tolerate latency, such as in payment processing and financial services, says Vunvulea.

“The latency between the instrument producing the data and the compute power that processes it is an important variable in determining data location,” says St. Jude’s Perry. In some cases, that instrument needs an almost instantaneous connection to high-performance compute resources. “Due to the latency between research instrumentation and our high-performance computers on premises and the public cloud, using the public cloud to perform real-time checks doesn’t make sense.” And as more public cloud hyperscalers build large-scale GPU clusters that can handle high-performance computing, you also have to factor in the cost, he says.

Genomic sequencing is one area where offloading some processing from local supercomputers to the public cloud might make sense — if the price is right. Some of the workflows associated with genomic sequencing become somewhat standardized over time, Perry says. In those cases it may make more sense to optimize the pipelines for scale and run them in the cloud, depending on the cost. “We’ve worked on moving some of our genomic sequencing pipelines into the cloud to free up cycles on our on-prem high-performance compute,” he says.

Performance is certainly important, but not the deciding factor when choosing whether to host an application in the public cloud — with the exception of some that run on edge servers at Mosaic’s mining operations sites, says Wysocki. “For us there’ll always be a need for edge computing that needs to be on the device or near it to be effective.”

A question of location

Security, privacy, and cost are the three main factors for us,” adds Wysocki. But so far, security and privacy haven’t been major issues with public cloud services.

Hollowell says RGA is satisfied with the security of its public cloud service. “We’re utilizing foundation models from Anthropic, Mitral, and others through AWS’s Bedrock service, which provides data isolation and security,” he says, enabling the company to provide ChatGPT-like functionality in a secure environment.

But digital sovereignty issues are a different matter, says Woo. In countries with strict localization rules, public cloud may be a non-starter. “You can opt for on-premises private cloud or hosted private cloud where you manage it or someone else does,” she says. “Either way you have control over where your data resides.”

But the regulatory landscape isn’t the only factor, Hollowell says. “In some geographies, data localization and privacy requirements are embedded directly into customer contracts,” he says. In such cases, a private cloud may offer a more flexible solution. So a hybrid approach between on prem and cloud is the best choice for large organizations running in multiple countries, says Vunvulea. And with respect to regional regulations, the choice of public cloud provider matters. “For example, Oracle cloud is one of the best options if you want to run workloads from inside a specific location in the Middle East,” he says, where each country has its own regulations with regard to handling data. No single cloud provider has a presence in all of those countries, but Oracle has a big footprint there, so you can run on-prem workloads in conjunction with Oracle and other cloud vendors.

But there’s a downside to hybrid cloud, says Hollowell. “Managing interoperability and performance for large data sets across public and hybrid cloud environments remains a key challenge to address, he says.

Maintain your flexibility — and be ready to adjust

Going forward, says Hollowell, “our strategic intent is to evaluate hosting decisions through the lens of evolving business requirements for new features, combined with natural application lifecycle management practices, rather than simply moving everything to the public cloud.” Applications with consistent capacity requirements that can be satisfied with traditional converged infrastructure will run in a private cloud, while those that don’t consistently require high compute will remain candidates for public cloud.

For Perry, constructing the right IT infrastructure for his organization’s applications is all about using the right building materials. “Public cloud is just one of the materials we need to build an architectural solution,” he says, and you have to strike the right balance.

Unfortunately, optimizing the mix of on-prem, private cloud, and public cloud services is a moving target. “I can’t say that everything is in the right place because the technology is evolving constantly,” Perry says. Cloud technologies are always changing, so be ready and able to change with the times, he advises. Making sure you have the right tools to do that is extremely important since the tools you have today might not be the ones you’ll need tomorrow.

That need to change things up as the technology advances is also a reason why you should avoid vendor lock-in, says Vunvulea. That’s a conundrum because to run cloud workloads in the most optimized way, you may need to use the vendors’ most advanced, proprietary features.

But in the end, he says, you want to avoid lock-in to have the flexibility to move more easily between on-prem, public cloud, and private cloud.