CIOs plagued by growing AI accountability gap

IT leaders are getting a sneak preview of governance in the agentic era, and it’s shaping up to be a horror show.

Two-thirds of CIOs and CTO surveyed by the IBM Institute for Business Value say they’re accountable for AI systems they don’t fully control as employees and other business units spin up new agents.

In addition, 70% of the IT leaders surveyed say their organizations are deploying tech systems faster than their IT teams can track. Moreover, CIOs expect a 38% increase in the number of AI agents deployed at their organizations by next year, with just one in 10 IT leaders saying they’re prepared for the anticipated scale of agent deployment.

The study reinforces growing concerns about the state of AI governance, with many IT organizations challenged to track output, security, and value as employees spin up new agents without IT’s input, says Matt Lyteson, CIO of technology transformation at IBM.

“A lot of enterprises have policies that make it easier for more people to develop agents, and it’s not just the only two people in the IT department to be able to develop these solutions,” he says.

Danger for IT leaders

While there’s value in encouraging employees to experiment with AI, doing so can create major problems for IT leaders, experts say.

When CIOs and CTOs are held accountable for AI tools they don’t control, it creates real tension in the enterprise, says Ben Schein, chief AI and analytics officer at data platform vendor Domo.

AI tools at many organizations are being deployed outside of IT teams faster than they can be inventoried, he adds. “The pace problem isn’t usually that AI is being shipped recklessly,” he says. “It’s that AI is being adopted faster than governance models can adapt.”

In many cases, the AI deployments are reasonable actions taken by employees, but CIOs and CTOs don’t see it happening, Schein says.

“Someone in marketing connects an LLM to a content workflow,” he adds. “Someone in finance pastes a forecast into ChatGPT to clean it up. Someone in product gives a new agent access to a customer dataset. The aggregate is invisible to the CIO.”

AI governance and observability are huge issues that need IT leader attention at many organizations, he says.

“Can you see what AI is doing?” Schein adds. “Treat your AI agents like employees: who are they, what data did they touch, what did they produce, what did it cost, what went wrong?”

AI governance isn’t a posture; it’s basic plumbing, he says. “The CIOs who’ll succeed in the next 24 months are the ones who build observability and policy enforcement into the same layer where data already lives — not a separate AI governance workstream bolted on after the fact,” he says.

Governance gap

Aatish Salvi, CTO at software testing vendor Applause, agrees that there’s an AI governance gap.

“We’re seeing that people are developing agentic workflows and AIs and products at a rapid scale across all industries, and they’re developing them faster than they understand how to govern, control, or evaluate them when CIOs or CTOs don’t have full control over AI systems,” he says.

Salvi sees the same challenge that Schein does: that many organizations encourage or permit employees outside the IT team to deploy agents, without letting the CIO or the CTO know. Control rests with whoever happened to build the agent.

“They may or may not have the technical expertise to manage, govern, or evaluate it,” he adds. “So someone somewhere in the company built something in order to help them do work, and now they’re getting their work done at some cost or expense to tokens, security, compliance, and all sorts of other concerns.”

Salvi sees a huge AI governance challenge for many organizations. “When people are building agentic systems by the dozen, democratizing the tools to build them throughout their organizations, and have absolutely no evaluation frameworks, they do not understand that they are compromising the quality of the work their employees produce in exchange for getting that work done significantly faster,” he says. “They’re producing mediocrity at great speed — and probably defects as well.”

There’s a danger in agent control sitting with someone outside the IT team, Salvi adds. “That someone does not have the tools, know-how, or technical experience to be exercising that control intelligently,” he adds.

In many cases, there’s no control at all over the agents deployed outside the IT team, counters Itai Schwartz, cofounder and CTO at data security vendor MIND. Without agent guardrails in place, agents often run without supervision, he suggests.

“Every AI tool should have a business owner who’s accountable, and usually there’s a name on paper,” he says. “But most of these systems are autonomous and non-deterministic. They don’t follow a fixed script. So in practice, the tool is controlling itself.”

The answer isn’t to slow down AI deployments in enterprises, but to give IT leaders visibility and enforcement tools that can keep up with the adoption curve, Schwartz says.

“No technology leader I talk to wants to slow AI down,” he says. “They want better tools to move fast and stay safe at the same time. The answer is better tooling, not more caution.”

Sanctioned AI platform

One possible approach is what IBM did: The company created an AI agent platform that allows employees to create their own tools within a controlled environment, Lyteson notes. New agents are checked for security and privacy, as well as whether they already replicate existing IBM tools.

“Very early on, we built an enterprise platform and invited people to build these solutions in a way that I feel comfortable working with my CISO on, by protecting the data by only using certain models that we feel comfortable with,” he says. “I want to know what that thing is doing, whether it is an individual productivity capability or whether this is supporting a full workflow, and I want to know the cost and value it brings to the organization.”