Building an AI CoE: Why you need one and how to make it work

Artificial intelligence (AI) is no longer the playground of hobbyists and programmers. From automating customer‑service transactions to optimizing supply‑chain decisions, AI is rapidly becoming the central nervous system of today’s enterprises. McKinsey surveys have found that nearly nine in ten organizations are now using AI regularly in at least one business function, compared with 78% the previous year.

But adoption rates are far lower when it comes to scaling AI programs throughout the enterprise. Only about one‑third of companies have advanced past the pilot stage. Two‑thirds of organizations use AI technologies in multiple functions and 64% believe AI has had a positive impact on innovation. Just 39% say they’ve seen a significant impact on the bottom line.

This research shows that AI has gone mainstream, but its benefits are still concentrated in the hands of a relative few. This reality makes even more compelling the case for establishing a formal center of excellence (CoE).

Per Microsoft’s cloud‑adoption framework, a CoE is a centralized team responsible for standardizing best practices across the organization. An AI CoE functions as an internal team of experts that helps business units realize valuable and successful AI outcomes while avoiding pockets of AI solutions built without governance or standards. A well‑managed AI CoE builds consensus for standards and pilots and provides business and technical guidance to help convert AI excitement into measurable business value.

An artificial intelligence Center of Excellence is a strategic capability, not an ivory tower nor a vanity project for the few AI early adopters. An AI CoE can dramatically improve the quality and safety of your organization’s AI adoption efforts. In this article, we will discuss why every organization needs an AI hub like a CoE. We’ll explore how to staff and structure your center and provide recommendations to help it adapt over time.

Why create an AI CoE?

An AI CoE “acts as the central nervous system” for your company’s AI strategy. Without a centralized AI organization:

• Siloed initiatives. Separate business units kick off AI pilots that aren’t centrally coordinated, leading to duplicated effort and fragmented results.
• Inconsistent governance. Data governance, model security and compliance efforts differ from project to project putting your organization at risk of regulatory infractions or biased models.
• Lack of standards. Groups spend time reinventing the wheel instead of building on reusable assets and shared toolkits.
• Difficulty scaling. Smaller pilots become roadmaps unto themselves because there’s no standardized process for prioritizing and scaling AI solutions.

An AI CoE helps solve these issues by setting forth an AI strategy, aligning AI efforts with your business objectives and offering governance, policy and expertise. Additionally, an AI CoE promotes responsible and ethical AI principles through fairness, privacy and transparency policies. According to IDC research, CoEs enable cross‑functional collaboration between data scientists, domain experts and chief finance officers to help translate AI prototypes into repeatable solutions that align AI spending with business strategy. IDC analysts also find that effective CoEs can provide talent development, knowledge sharing, partnerships and an innovation mindset.

The business case only strengthens when looking at the economics of generative AI specifically. IDC’s 2024 “Business Opportunity of AI” study, sponsored by Microsoft, revealed generative‑AI adoption rose from 55% in 2023 to 75% in 2024. Companies that deployed generative AI were also found to experience significant returns. Organizations see an average ROI of $3.7 realized for every dollar spent on generative AI. For high performers, ROI reached $10.3 for every dollar spent. The majority of companies spend less than eight months deploying generative AI solutions, with ROI seen after just thirteen months. The research highlights how well‑managed AI initiatives can produce exceptional returns and why centralized governance and expertise are critical.

In addition to mitigating risks, a properly established CoE also creates opportunities. According to IDC, successful COEs can provide workforce enablement, knowledge dissemination, strategic alliances, certification and training and a culture that fosters innovation and creativity. Leaders at Hitachi Vantara call their COE “enabling AI to go from theoretical exploration to practical implementation that optimizes processes, supercharges efficiency and unlocks data‑driven insights.”

When it comes to AI in financial services, BizTech Magazine found that while 81% of executives say they are currently using AI and will invest more money in it, only 25% of them have completely deployed monitoring and management tools even though 87% said they already have governance in place. This is a clear example of how businesses are adopting new technologies but lack operational maturity and how a centralized CoE can help bridge that final mile and make governance frameworks a reality.

Business impacts achieved through CoEs are faster AI adoption, more efficient resource utilization, better decision‑making, lower risk, stronger strategic alignment and better collaboration. Boards and investors are demanding it; according to IDC analysts, organizations with mature AI governance have seen increased returns on invested capital (ROIC) and view CoEs as critical source of competitive differentiation.

How to build and structure the CoE

An AI CoE should have an executive sponsor who can offer budget, authority and credibility to ensure standards are upheld. Create a steering committee of business and IT leaders and schedule frequent reviews to monitor progress. Then, identify a leader for your CoE who is devoted to its success and has deep AI skills, as well as reach across the enterprise. Hire a diverse team of business leaders, data scientists, machine‑learning engineers, governance and security professionals. This variety of backgrounds will help ensure AI initiatives meet technical and business needs, as well as regulatory and ethical standards.

Define the operating model and responsibilities

IDC analysts note that an important objective of a CoE should be to close the supply‑and‑demand gap for AI skills. This can be accomplished by creating a centralized team of employees from different business units or geographies who pool their collective knowledge and then disburse back out into the business. Hardy recommends that this cross‑functional team include not only deep technical experts such as data scientists, AI engineers and machine‑learning specialists but also business leaders as well as IT and cybersecurity professionals. These members can help ensure AI initiatives are applied to business problems and integrated into production environments securely. Team members may include data scientists, software engineers, business analysts, subject‑matter experts and project managers. Common skills include domain knowledge, programming and data skills, problem solving, communications and a team mindset. Skills required of the Center of Excellence leader include a strong knowledge of AI, along with a visionary but execution focused approach to work. Additional leadership traits include practicing radical candor with your teams and colleagues while staying agile and flexible in your decisions due to the rapidly changing nature of AI.

Skills gaps are another common challenge to AI scaling. According to Microsoft’s 2024 IDC-commissioned survey, 30 % of respondents stated their organizations don’t have specialized AI skills and another 26 % said their organization has too few employees with the skills necessary to learn and work with AI. By tapping into talent from across the business, the CoE can centralize hard-to-find expertise. It can also administer training initiatives to fill these gaps. The CoE should partner with HR leaders to create learning journeys, certification initiatives and mentorship programs to ensure the talent pool continues to grow with advancing AI technology.

Determine where the CoE sits in your company’s hierarchy. A centralized hub makes sense early in your AI journey to centralize knowledge and ensure consistent practices. But as your organization adopts AI, the CoE can evolve into a decentralized, enablement model that provides guardrails and allows product teams to own their own AI applications. Clearly establish your CoE’s primary functions. These may include:

• AI strategy and use‑case identification. Partnering with business leaders to identify and prioritize AI opportunities that will provide the most value to the organization.
• Skills development. Determining your current level of AI skills and implementing learning and hands‑on experimentation programs.
• Pilot projects. Leading targeted pilots to prove out AI methodologies and provide proof of business value.
• Standards and governance. Establishing governance frameworks and security standards, monitoring usage to ensure AIs are being used ethically and performing routine data security and compliance audits.
• Intake and prioritization. Establishing a formal process to accept requests and evaluate them based on potential business value, feasibility and resource demands.
• Reusable assets. Creating checklists, templates and code libraries to speed up future initiatives.
• Metrics and reporting. Measuring adoption, compliance and business value and using that information to foster continual improvement.

Integrate ethics and responsible AI

AI solutions should be ethical, which means they should be helpful and unbiased. The CoE should develop policies around responsible AI use, so models are transparent, unbiased and reflect company values. Teams should conduct audits of training data and model outputs to identify and reduce bias and the potential for inadvertent harm. AI governance should include privacy and data‑security principles.

Avoid bureaucratic bottlenecks

One pitfall of CoEs is that they tend to turn into gatekeepers and slow down innovation. Instead, Microsoft recommends shifting the CoE from being a gatekeeper to playing an advisory role once your AI adoption becomes more established. Build AI delivery into platform teams and allow product teams to execute against AI solutions under guardrails. The CoE can concentrate on things like setting standards, sharing knowledge and mentorship while teams on the frontline own the execution.

How to ensure continuous improvement

AI is still an emerging technology and science, which means that a one‑time standing CoE will become irrelevant almost immediately. It should evolve constantly, leveraging three primary levers:

• Feedback and learning loops. Establish processes to capture input from users and stakeholders in production and pilot environments. This feedback should be used to update models, training datasets, documentation and governance processes.
• Investment in skills and culture. Embed AI literacy into your culture by providing regular training and building communities of practice. Forums like these allow employees to share failures and best practices. Focus change‑management efforts on employees’ misperceptions about how AI will replace their jobs. Communicate how AI tools will make their jobs easier instead.
• Metrics‑driven evolution. Define a measurement framework that encompasses adoption, compliance and ROI metrics. Use these measurements to surface bottlenecks and opportunities for improvement. If your metrics indicate your central governance is slowing adoption, consider a more federated approach.

Researchers from IDC stress that “unlocking the power of frontier AI … requires building a culture of continuous learning.” COEs should implement processes like internal training initiatives, communities of practice and sandbox spaces for experimentation so that knowledge can continue flowing to employees as fast-changing AI capabilities develop. Benchmarking your progress is also key, say the analysts in their suggestions for how to measure COE success with AI. IDC recommends defining clear goals, building KPIs into projects, tracking completed initiatives, gathering feedback on customer satisfaction and looking at indicators for revenue growth and innovation.

Conclusion: A strategic capability, not a side project

An AI Center of Excellence won’t solve every challenge. It needs ongoing senior leadership sponsorship, cross‑functional teamwork and an openness to shift your operating model as your organization matures in its AI journey. However, when done right, it provides a framework for enterprise‑level AI enablement. It ensures AI initiatives are aligned to business priorities, sets standards and governance, develops your workforce and speeds up the responsible delivery of AI solutions.

AI Centers of Excellence are already helping organizations drive value across industries by scaling AI adoption, strengthening governance, accelerating experimentation and moving AI use cases from ideation to production.

• Financial services. Major banks are forming federated AI CoEs made up of divisional CoEs within business units like retail banking, wealth management and asset management. These cross‑functional teams customize AI for their functions, whether that’s portfolio optimization or customer support automation, backed by a centralized GenAI layer that provides unified governance, tools and evaluation frameworks. The federated approach limits redundancy, fosters collaboration and scales AI more broadly.
• Professional services and technology. With Microsoft’s guidance, NTT DATA developed an agentic AI CoE. The center offers a centralized environment for customers to design, deploy and operate AI agents spanning different cloud environments. Highlights include unified governance that’s aligned with its cloud center of excellence (CCoE) architecture, shared infrastructure to build agent‑based applications and coordination with Microsoft subject matter experts. The AI CoE serves as an engine for delivery, helping accelerate the path from experimentation to production at scale with security built in.
• Consulting firms. Capgemini applies the principles of an AI CoE to its suite of offerings to ensure consistent AI governance, reuse assets and tools and link AI projects to quantifiable business outcomes. Standardizing the how behind project execution allows Capgemini to decrease variation across customer projects and empower organizations to operate at speed without losing sight of enterprise needs.
• Enterprise experimentation. EY created an AI CoE focused on providing a secure sandbox environment. Teams can experiment with AI use cases, validate their feasibility and associated risk and fast‑track promising use cases to production. Centralized visibility and governance allow for consistent security and compliance standards, shortening the time between ideation and execution and preventing siloed adoption.

Taken together, these industry examples highlight how a CoE is less of a technology endeavor and more focused on creating institutional trust and capacity. Want more proof points on how COEs make an impact? Consider how at ECS, a provider of cloud, cybersecurity and artificial intelligence (AI) services, the data and AI COE unify more than 200 data professionals across the business, shares their collective expertise across town halls and events and manages strategic partnerships. The COE enables proposals, solutions and fosters a culture of creativity and innovation.

Over at Hitachi Vantara, the AI COE is tasked with transforming ideas into production‑ready prototypes and is already being recognized for improving efficiency across operations and creating new revenue streams from advanced machine‑learning models. The board wants ROI and organizations with mature AI governance and COEs can increase returns on invested capital and create new sources of revenue. These are just a few examples of how a well architected CoE turns strategy into tangible business value.

The race for AI supremacy is picking up speed. Organizations that invest time and resources into building an effective AI Center of Excellence will be best positioned to turn innovation into competitive advantage. The CoE is how your biggest ideas go from concept to business solution–helping leaders do their best work.

This article is published as part of the Foundry Expert Contributor Network.
Want to join?