Why enterprise CIOs need to plan for Microsoft gen AI

Between building gen AI features into almost every enterprise tool it offers, adding the most popular gen AI developer tool to GitHub — GitHub Copilot is already bigger than GitHub when Microsoft bought it — and running the cloud powering OpenAI, Microsoft has taken a commanding lead in enterprise gen AI.

Beyond the ubiquity of ChatGPT, CIOs will find obvious advantages working with a familiar enterprise supplier that understands their needs better than many AI startups, and promises integrations with existing enterprise tools. But those close integrations also have implications for data management since new functionality often means increased cloud bills, not to mention the sheer popularity of gen AI running on Azure, leading to concerns about availability of both services and staff who know how to get the most from them.

In fact, many similar advantages and disadvantages will likely apply to any AI platform provider that enterprises choose, and CIOs need to consider these wider questions in their gen AI strategy.

Ahead in a broad market

In Morgan Stanley’s quarterly CIO survey, 38% of CIOs expected to adopt Microsoft Copilot tools over the next 12 months. Microsoft itself claims half of Fortune 500 companies use its Copilot tools and the number of daily users doubled in Q4 2023, although without saying how widely they’re deployed in those organizations.

Existing investments and relationships with Microsoft play a significant role here. “Copilot winds up being a bigger part of the pie than even we expected because it’s with a vendor most companies already work closely with,” says Forrester lead analyst on Copilot for Microsoft 365 J.P. Gownder.  “It’s embedded in the applications we use every day and the security model overall is pretty airtight. CIOs would rather have employees using a sanctioned tool than bring your own AI. That’s risky.”

All OpenAI usage accretes to Microsoft because ChatGPT runs on Azure infrastructure, even when not branded as Microsoft OpenAI Services (although not all the LLMs Microsoft uses for AI services in its own products are from OpenAI; others are created by Microsoft Research). The cost of OpenAI is the same whether you buy it directly or through Azure. New models roll out at the same time, and buying from Microsoft offers safety and governance advantages like every other Azure service, with access to Azure OpenAI services segmented by subscription and tenant, and each enterprise getting its own instance.

Microsoft CTO Kevin Scott compared the company’s Copilot stack to the LAMP stack of Linux, Apache, MySQL and PHP, enabling organizations to build at scale on the internet, and there’s clear enterprise interest in building solutions with these services.

CVP of Microsoft’s AI platform Eric Boyd says the 60,000-plus organizations using Azure AI services include 65% of the Fortune 500, over a third of which are new Azure customers drawn by OpenAI that go on to use more Azure AI services. For example, half use Azure AI Search to make enterprise data available to gen AI applications and copilots they build.

Microsoft has also made investments beyond OpenAI, for example in Mistral and Meta’s LLAMA models, in its own small language models like Phi, and by partnering with providers like Cohere, Hugging Face, and Nvidia. Its model catalog has over 1,600 options, some of which are also available through GitHub Models.

Although competitors have similar model gardens, at 13.8% of the market according to IDC, Microsoft 2023 revenue from its AI platform services was more than double Google (5.3%) and AWS (5.1%) combined.

Plan ahead for heavy usage

The popularity of OpenAI raises questions about gen AI availability. As in Q3, demand for Microsoft’s AI services remains higher than available capacity. That’s an industry-wide problem. “Generative AI and the specific workloads needed for inference introduce more complexity to their supply chain and how they load balance compute and inference workloads across data center regions and different geographies,” says distinguished VP analyst at Gartner Jason Wong. While availability is not a fait accompli from Microsoft, he notes it’s an issue for many hyperscalers.

Continuing its existing approach of investing in more regional and country-specific data centers for Azure improves performance and data sovereignty, and will also help Microsoft avoid gen AI bottlenecks, with half the substantial increase in CapEx earmarked for server CPUs and GPUs.

Specifically for OpenAI, Microsoft also designed its own Maia AI accelerator for much greater rack density than the Nvidia and AMD GPUs usually used for frontier models. It’s aggressively deploying those to Azure data centers, which won’t require any changes by customers, and expects these investments to come closer to meeting demand by mid 2025.

It’s also creating tools to help customers pick from a wide range of models, Wong adds. “There’s small models and multimodal models with vision, voice, and text,” he says. “There’s a lot of new advancements and small models that might help to reduce workloads.” Organizations typically start with the most capable model for their workload, then optimize for speed and cost. Azure AI customers are starting to experiment with smaller models, using Azure AI Studio to benchmark and compare them in their application scenario.

“If I can effectively fine tune one of those models to my specific use case and get a better performing model than perhaps a larger model that wasn’t fine tuned on that, then I get the benefits of the smaller model in terms of costs and lower latency with the quality of a larger model, at least on this specific task,” Boyd says.

He encourages CIOs expecting to scale up to significant gen AI usage to approach Microsoft in advance. “Surprises are always the hard thing,” he says. “If a company wants to go from zero to a million GPUs overnight, that’s probably going to be hard. But if there’s some plan for how that’s going to go, I think in general, we’ve been able to meet all of those [requests].”

Enterprises with data gravity and infrastructure gravity in other clouds naturally start by evaluating gen AI functionality from their existing providers — who are also cautious about capacity — before considering options like Azure, notes Rowan Curran, senor analyst at Forrester. “Microsoft and, by proxy, Open AI have been in the forefront of the public perception around this, but when we look at the enterprise buying landscape, what we see is there’s a lot to go around.”

Platform familiarity has advantages for data connectivity, permissions management, and cost control. “Implementation is a huge portion of actually achieving success,” he adds.

Start where your data is

Using your own enterprise data is the major differentiator from open access gen AI chat tools, so it makes sense to start with the provider already hosting your enterprise data. “It’s the contextual information supporting the use of these tools,” Curran says.

Organizations with experience building enterprise data lakes connecting to many different data sources have AI advantages. “They’re more able to connect to a diverse set of data sources, build more complex context around queries going through the model, and do retrieval augmented generation,” he adds.

For many enterprises, Microsoft provides not just document and email storage, but also the root of enterprise identity for those data sources, as Vadim Vladimirskiy, CEO of software developer Nerdio, points out. “For most of our customers, Azure is where their identity is, where people’s credentials are, where people’s data already is,” he says. “From an authentication perspective, making those data connections is easier when you’re in the same cloud that shares the same identity plan.”

That’s especially true for Copilot. “The big advantage of Microsoft 365 Copilot, compared to some non-Microsoft alternatives, is it’s integrated into the Microsoft 365 ecosystem of products, which means all the data an enterprise has in its data repositories can be surfaced within Copilot,” he adds.

With enterprises new to Azure choosing it for the OpenAI services, Boyd also claims they come for the AI applications, but stay for the complete platform. Using Azure for both data and gen AI means both Copilot and any applications organizations build themselves inherit the security, permissions, and data access already in place. The gen AI tools in the Microsoft stack respect access permissions and sensitivity labels, too. “When the data is coming from SharePoint and the like, Azure AI Search will honor the permissions the preceding data had,” says Boyd.

But the disadvantage of using gen AI that can connect to all your data comes when you’ve cut corners in privacy and data management. “If you pull your data from a document with no permission set on it, then there’s no information to be had,” he adds. “People need to consider this when building their applications.

This isn’t a new issue. “Good data hygiene, and information architecture has always been a problem,” Wong says. Gen AI just makes it more obvious when enterprises aren’t on top of managing permissions or curating their data. “When you talk about the embedded AI capabilities of tools, whether it’s Microsoft 365 or Google Workspace or Slack AI, inherently it feeds on the data in the system — the quality and relevance of the data, if it’s redundant or obsolete, if it’s overly shared and has sensitive data, that’s going to cause quality issues as well as compliance and safety risks.”

Concerns over exposing data to staff who shouldn’t have access has delayed some Copilot deployments, Wong says. “For Microsoft 365 Copilot, we see a significant portion of clients who say they’ve extended the pilot or delayed production rollout by three months or more because of security concerns.”

To help, the Microsoft Purview data governance service now includes an AI hub organizations can use to find and secure data, track the usage of that data in Copilot and other gen AI tools, and manage compliance, retention, and deletion, but it takes time and expertise.

“Microsoft provides all the tools to lock that down, but it takes a good amount of expertise to understand what allows data flows, how they work, and how permissions are set up within the Microsoft Graph system to properly lock it down and set it up exactly as the CIO wants for the organization,” says Vladimirskiy.

Virgin Atlantic VP technology and transformation Gary Walker suggests organizations start there. Though eager to get on the Copilot beta, the airline spent 10 weeks analyzing data security using tools like Purview and Sharegate to look at every document and artefact in their Office 365 tenant, documenting what permissions were set on them in a data leakage report before enabling Copilot.

“Data privacy, data control, and data access management has changed massively even over the last five years,” he says. “A lot of us inherited estates that are 20 to 30 years mature and you can bet documents, which may just be as sensitive as the stuff being created today, may not have been treated with data privacy in mind at that time.”

Even if SharePoint permissions were initially correctly set, changes to job roles and group membership can introduce issues as people move around the organization and documents get misclassified. The report covered not only the documents and people involved in the proof of concept, but the actions required to safely extend Copilot usage to other parts of the business.

Evaluating that much data to clean up any existing problems and create the right structure to enforce future data hygiene is a valuable exercise few organizations will have attempted but many would benefit from. Business functions like legal and HR proved to have the expected tight controls but there were surprises like business access to IT systems.

“My biggest recommendation would be to concentrate on data security and classification, create a data leakage report and make sure you understand the bounding of the user group you’re enabling this function on,” he says. “Don’t do it straight across the enterprise. Don’t view Copilot as just an extra bolt on to Office 365. It’s a very different beast.”

Measuring costs and value

The other major issue with gen AI is the price. Organizations expect using gen AI to increase costs by almost a quarter over the next two years, according to IDC.

Many IT budgets are fixed, and saving time doesn’t automatically mean more money available to spend. Organizations typically start with a small number of users to assess where to deploy to more broadly, and even look at budgets beyond IT to allocate the cost of productivity improvements. Microsoft calls this ‘land and expand’ and it’s very different from established Office adoption, or other familiar software costs.

Some Microsoft gen AI tools are included in the price of existing products, like Copilot Studio in the Power platform, or Copilot in Dynamics 365 for sales, which also works against other CRM systems like Salesforce. But as well as the price of whatever models and services enterprises pick to build custom tools, the cost of Copilot often doubles the costs of Microsoft 365, says Vladimirskiy. Not all licences are the base $30 per user per month for Copilot. For more specific skills, like the new Copilots for financial services or customer service, there’s an extra cost for domain expertise on core tasks like variance reports.

Microsoft says the majority of enterprise customers are coming back to buy more Copilot seats. But most licences are for trials, not large scale deployments — usually less than 20% of employees according to Gartner, with early adopters looking at the familiar cost versus ROI equation before expanding.

“CIOs can have a hard time building a business case to show how you can double the cost of your Microsoft 365 licence and get an equal amount of productivity gain,” Vladimirskiy says. “But Microsoft is working on providing insights and utilization visibility, which will help with that justification exercise.”

Enterprises are increasingly interested in creating custom copilots in Copilot Studio, Wong says. “Getting very specific data that’s curated, vetted, authoritative for specific roles and functions is ultimately going to move the needle, instead of just indexing the entirety of SharePoint and hoping for the right answer,” he says.

That may also save money. “Often customers have very specific needs like summarizing document libraries,” Vladimirskiy suggests. “Copilot can do that but then I pay $30 per user per month for lots of other things I don’t really find valuable. Can I build something that still has access to my data, still runs within the Microsoft cloud, still uses the same foundational models like OpenAI, without paying for things I’m not using? Assuming you have the software development talents to build a product using these AI building blocks, you can get something much more robust, customized, and secure than ChatGPT.”

Enterprises with strong experience in open source may look to open foundation models as an option to reduce costs, but Curran cautions against equating open weight models with the more familiar open source ecosystem. He predicts enterprises will adopt them though, including using them in curated environments provided like Azure.

“I’ve seen a lot of interest in the open source models, but not many in production,” says Boyd, although customers are starting to use small models like Microsoft’s own Phi series. “But it’s largely early days. I haven’t seen mass adoption.”

Beyond simplifying setting up and running open weight models, using them on a platform like Azure has an added benefit: Microsoft’s model content safety service is “on and integrated by default with Azure Open AI Services, but it’s also on by default with all our open source models as well,” he adds.

After the excitement and experimentation of last year, CIOs are more deliberate about how they implement gen AI, making familiar ROI decisions, and often starting with customer support. “It’s a cost most organizations have but don’t like paying for, yet they still want to provide a quality experience,” he says. Reduced call times and escalations are obvious benefits as well.

These more vertical, task specific, integrated gen AI offerings may contribute more than generalist productivity copilots because people won’t need to find uses and then remember to include them in their workflow. But the most popular copilots can perform strongly: Virgin Atlantic, for instance, reports efficiency gains of 14 minutes per day.

But not all copilots necessarily provide the same value. Curran suggests security copilots may not provide significant extra value on top of existing tools in Microsoft Defender, at least without extra training. But the Excel Copilot was a surprise hit at Virgin Atlantic. “People absolutely love that Copilot will automatically tell you if you have data inconsistencies in the way you’re filling out forms,” says Walker. He describes how Copilot can warn if, say, you’re adding a duplicate filter in lower case instead of upper case, and fix it. “It’s like having someone look over your shoulder as you’re doing it.”

The Teams Copilot to summarize meetings and provide next steps is almost universally popular, too. “You get into a room with 15 people and you’re not focusing on who’s taking the minutes or whether you need to be clear enough in allocation and make sure everyone understands what the output is,” says Walker. “You’re focusing on the meeting itself, and you’re more present in the room because you know Copilot is behind you recording and transcribing.”

Even this pre-built Copilot needs preparation before enabling. Multilingual organizations where staff speak in both their native languages and a common language like English or Mandarin will need to monitor quality of transcriptions and translations more carefully. And if recording meetings isn’t already common in the organization, CIOs need to consult with legal and data protection teams on retention, auditing, and deletion policies because of potential issues around discovery.

A data leakage plan helps here too. “As soon as you record something, it becomes a form of data and needs classification and a place in the organization,” Walker says. “But equally, you need to know whether it’s appropriate to create that data in the first place.”

While CIOs need to maintain financial discipline and track usage of gen APIs with the now familiar ‘pay as you go’ model, especially with September budgeting season looming, they also need to play a long game warns Mickey North Rizza, group VP, Enterprise Software at IDC. “It’s going to cost you a lot of money,” she says. “CIOs may complain they’re not getting enough out of it, but the first time you got an iPhone, nobody knew what to do with it.”

Whether used as an assistant, advisor, or an agent, she expects gen AI’s optimized access to information to reduce multi-step business processes to real-time systems with far fewer steps. But experimentation to achieve significant results takes time.

In the meantime, Boyd notes, OpenAI prices have significantly reduced. “In the year and a half since Azure OpenAI Services has been available, ChatGPT 4 has fallen by 12 times while being six times faster,” he says.

Make training specific

Phased deployments aren’t just about cost, security, or compliance concerns, but capturing the right feedback to manage them well and support users properly. Training is key, even when considering gen AI skills in hiring, as is being willing to accept the simplified processes gen AI can produce. Troublingly, there’s a considerable disconnect between what leaders think their employees are ready for with gen AI and what staff feel prepared for.

Forrester found 59% of leaders believe they’ve given staff sufficient training, but only 45% of employees say they’ve had any formal training. The most successful training covers not just staff roles but their workflows. There’s enormous enthusiasm for gen AI but engagement quickly drops off if they don’t have the time to explore it and learn how to get useful results for their work, Wong says.

“If you don’t use the technology to fundamentally rethink processes, and you just layer more AI work over existing processes, you don’t get the best benefit out of it,” he says. “You have to rethink the underlying processes, and have training and ongoing education because these technologies are moving very quickly. The paradox is employees still want it despite the fact it’s hard for them to ingrain generative AI into their work routines, and that in some cases it’s underwhelming based on their expectations.”

CIOs may then want to consider how organizations adopt low code tools, where encouraging bottom up enthusiasm, experimentation, and sharing of growing expertise helps spread usage across the business. Both Microsoft and Virgin Atlantic report good results from structured training that includes time to experiment. Walker refers to “guided play sessions” and users were encouraged to share what worked with their peers. “They can go out as trusted users into the environment and say to people this isn’t scary,” he says.

CIOs should also remember gen AI is just one of many changes organizations are asking staff to absorb. The rate of change enterprise workers are expected to adapt to is up to three times what it was in 2010, Curran warns. “Businesses have not increased their ability to support those changes with the same speed,” he says. Adding resources to support employees through these changes will be as important for succeeding with gen AI as getting the technology right.

That includes IT teams themselves, who need to prepare for gen AI to continue developing at this speed. Vladimirskiy passes on Microsoft’s advice to software partners creating their own gen AI products. “Everyone should have the expectation that by the time you build something, you’re going to have to scrap it and start again,” he says. “The value for companies is maybe not so much the outcome of the product they’re building, but the creation of the expertise within the organization, to be able to leverage it in the future when AI becomes much more capable than it is today.”