AI agent orchestration: The CIO’s crucial next step

In the near agentic AI future, enterprises will run dozens if not hundreds of AI agents, with CIOs in charge of orchestrating and connecting them to help employees navigate a range of interconnected business processes.

Take onboarding a new employee. In the not distant future, an HR rep might ask a chatbot to set up the new employee. By coordinating several agents, the chatbot would in turn enter the employee into the payroll system, walk her through health insurance options, and set up her email and videoconferencing services. Further agent coordination could also deliver the new employee training, issue a building entry badge, and even ship her a laptop or assign her a desk, all with minimal human input.

This vision of AI orchestration and integration is already being rolled out by some companies, and as organizations deploy multiple large language models (LLMs) and dozens of AI agents in the coming years, mass adoption of AI integration and orchestration tools is likely.

By 2028, 70% of organizations building multi-LLM applications and AI agents will use integration platforms to optimize and orchestrate connectivity and data access, Gartner predicted in a recent report. Less than 5% of similar organizations were using AI integration platforms in 2024.

Some AI experts see that orchestration function, where many AI agents are tied together to create wide-ranging and autonomous workflows, as the point when agents become agentic.

Connecting data and decision-makers

AI integration and orchestration tools will be vital for most enterprises because AI agents and LLMs need to be connected to fully reach their potential, says Andrew Humphreys, a senior director and analyst at Gartner. Recent releases of several agent protocols take the first step toward this widespread integration, he notes.

“AI has to have access to data in order to make decisions, and it has to have the ability to actually take some kind of action,” he says. “An agent is useless if it can’t have access to data, and it can’t make a decision.”

Agent integration and orchestration will help CIOs address several emerging questions about the coming agentic AI world, he adds. “How do I make it easier for my AI developer or my agent to be able to connect to things and get data?” Humphreys says. “How can I observe what’s actually happening within my IT architecture?”

As CIOs’ AI strategies become more complex, the need for agent orchestration platforms becomes readily apparent, adds Beth Scagnoli, vice president of product management at data readiness solution provider Redpoint Global.

“Most organizations today aren’t just experimenting with a single AI model; they’re working across multiple LLMs, legacy systems, and newer AI agents simultaneously,” she says. “Without orchestration, that ecosystem risks quickly becoming fragmented, redundant, and inefficient.”

The orchestration layer will manage how AI tools access, move, and act on data across systems, she adds. “This is not just for outputs, but also for maintaining enterprise standards around governance and trust,” Scagnoli says. “AI orchestration is quickly becoming the critical link between data strategy and AI execution.”

A market emerges

CIOs adding orchestration tools should look for interoperability, Scagnoli says.

“Orchestration layers that are flexible and AI-agnostic will be where the true value shines,” she says. “Tools that can sit above any LLM or agent, allowing for organizations to plug in the right model for the job all while managing security, versioning, and data lineage behind the scenes,” will be IT leaders’ best bet.

Gartner’s Humphreys sees a burgeoning marketplace for AI integration and orchestration platforms, with many small players currently offering out-of-the-box solutions. As the market becomes more profitable, larger IT and AI players will get into the game.

Some companies will also build orchestration tools themselves, Humphreys says, but he urges IT leaders to take the lessons learned from past integration efforts, including orchestration of API calls.

“You’ve probably already put together your existing API integrations,” he says. “Tweak that rather than thinking that you’ve got to completely reinvent the whole bit. Just adjust that to meet the way that your AI needs to talk to it, rather than thinking, ‘Oh, it’s AI, I’d better rewrite everything I’ve learned in the past.’”

Experimenting with orchestration

IBM is one company that’s taking on agent integration in house. The tech giant began experimenting with agent-like tools eight years ago, and it now has agents deployed in several workflows, including IBM’s sales and IT departments, says Suzanne Livingston, vice president at IBM watsonx Orchestrate Agent Domains. HR was the early test case, and now, agents operate many HR functions.

“There are a lot of processes in HR, and it’s difficult for employees to understand how to work with the HR system,” she says. “We use one [app] at the time, and if you’ve ever used one of these enterprise HR systems, you had to find the instructions to know exactly what you wanted to do. And, by the way, those instructions changed every month.”

Now, IBM employees can interact with an AI agent to create salary increase requests, transfer employees between departments, create job descriptions, and accomplish a range of other HR functions, she says. An AI orchestration layer is needed to interact with the agents operating all those individual HR tools to accomplish multistep workflows, such as employee onboarding, she notes.

Still, for most companies, a vendor-supported out-of-the-box tool may be an easier way to get started, Livingston says.

“It’s a great way to not have to start by building something from scratch,” she says. “It’s a great way to trial it out and get a feel for it, and then, it may lead you to bigger projects as a result, but it is useful on its own.”

She also suggests that CIOs look for low-hanging fruit where there are a lot of employee pain points, for example, time-off requests. “Everyone has to submit time off, and no, no one wants to do it because it is clunky,” she says. “But it’s an easy one to get started with, and you get immediate value.”

The future will be agents everywhere, Livingston adds.

“It’s like a never-ending realm of underlying agents,” she says. “As companies evolve and bring on board new processes, or reduce complexity of processes, it’s its own positive loop of an experience.

“It’s getting companies to adopt that mindset, ‘I don’t have to train my employees on 500 different systems, I can help them understand how to utilize the benefit of these systems through an agent.’”