Ways CIOs can maintain control amid changes brought by AI

It took nine seconds for an AI agent to destroy PocketOS’s production database. At work on a routine task in April, the coding agent, a variant of Cursor running on Claude Opus 4.6, ran into a credential mismatch and decided to fix the problem by triggering an API token. Little did PocketOS founder Jer Crane know that its activation would also delete its production database. “Had we known,” Crane later wrote on X, “we would never have stored it.”

The consequences of the agent’s actions were immediately apparent. Not only were recent backups belonging to PocketOS’ infrastructure provider contained in the production database — the recoverable versions were at least three months old — but so were those belonging to its infrastructure provider, Railway, which at press time still couldn’t tell Crane whether full infrastructure-level recovery was possible. Crane couldn’t fathom why the agent did this. So he asked it.

What he got back was an apology, of sorts. “I guessed that deleting a staging volume via the API would be scoped to staging only,” the agent said. “I didn’t verify. I didn’t check if the volume ID was shared across environments. I didn’t read Railway’s documentation on how volumes work across environments before running a destructive command.”

Ignoring built-in safety guardrails is hardly unique to agents operating on Claude Opus 4.6. In July, a Brazilian software engineer claimed an agent powered by OpenAI’s GPT-5.6 Sol model also deleted his production database, while in February, a Meta AI security and safety researcher claimed she had to switch off her computer to prevent an experimental agent deleting her entire inbox.

It wasn’t meant to be like this. Agentic AI was intended to be the culmination of millions of hours of research and development in gen AI to perform hyper-qualified acts of pattern recognition in the real world, and truly live up to their labor-saving promise. Their apparent predilection for destruction, however, has revived multiple debates about exactly how they should be restrained, and who, ultimately, is responsible for doing so.

Ultimately, the answer is those who green-lit the offending system. But as the pace of AI development puts greater daylight between companies pressured to adopt it, and those very tools capable of wreaking havoc across their internal databases, are CIOs now out of their depth?

Setting the pace

There’s no question the emergence of gen AI has changed the CIO role. “A few years ago, most of my time went to infrastructure decisions, including what to build, what to buy, and how to sequence the roadmap,” says Mike Trkay, CIO at data analytics company FICO. “Now, a growing share goes to questions of trust, verifying that when AI writes code, makes recommendations, or acts on behalf of a system, those actions can be explained and traced back to someone accountable for them.”

So the CIO has become the enterprise’s technological organizer du jour. “AI is accelerating software development, decision automation, and organizational experimentation at a pace that can outstrip institutional coherence,” says Edosa Odaro, executive advisor for data and AI at consulting firm VDS Global. “As AI becomes embedded across every business function, CIOs are increasingly responsible for ensuring that technical capability, governance, data quality, cybersecurity, human capability, and business strategy continue to evolve together rather than fragment.”

Day to day, that’s led to an exponential change of pace. “Things have always been fast,” says Zach Lewis, CIO and CISO of the University of Health Sciences and Pharmacy in St. Louis. “But now that speed of change is quicker, and you have to adapt.” And the need to catch up is constant. There’s no other option because then any competitor or co-collaborator can jump ahead, adds Lewis.

The rapid pace of change in AI also threatens to diminish the authority of individual CIOs who fail to keep up or set effective guardrails on those individuals who like to experiment with the newest models with loose regard for corporate security. “There’s all these AI tools that employees can now just go out and adopt,” says Lewis. And at the moment, a paid subscription to Claude or ChatGPT isn’t required to capitalize on its abilities. Consequently, staff are just a click away from asking LLMs to perform various tasks and expose sensitive corporate information in the process. “Everyone wants to play with the new thing,” he says. “And when they find benefit there, they’re going to want to bring it to their work lives.”

Agentic AI poses an entirely new set of problems. For one thing, says Odaro, the next phase of application adoption will be defined less by the capabilities of individual models, and more on what you allow their agents to do. “As AI becomes increasingly capable of generating software, coordinating workflows, and making recommendations across functions,” he says, “the challenge shifts from building AI to continuously governing evolving AI systems.”

This, Odaro continues, means that the CIO’s current approach to governance isn’t sustainable. “Static policies, annual reviews, and isolated oversight will struggle to keep pace with dynamic AI environments,” he says. “CIOs will increasingly need continuous governance capabilities that provide ongoing visibility into AI performance, value creation, risk, trust, and organizational adoption.”

Falling over the guardrails

How, then, should CIOs approach writing these new guardrails? Traditionally, this would be perfect fodder for so-called alignment researchers investigating how to instil a sense of morality and propriety into agents. According to analysts at Google DeepMind, however, it’s best to assume the agent will always be a potentially chaotic force within the company, and set parameters on its conduct from there.

“We borrow a lot from security, which already deals with the threat of internal employees who might be malicious, and we can apply these to a new setting,” Rohin Shah, Google DeepMind’s AGI safety and alignment team lead, told Fortunein June. Even so, he added, “AI is systematically different from humans.”

That difference primarily pertains to authority and speed. For agentic AI to live up to its full potential, it requires the freedom to access multiple systems simultaneously — an uncomfortable fact for CIOs hoping to align agent responsibility across the enterprise. In a time when workflows are becoming ever-more automated, however, that aspiration may prove unrealistic. In that case, Google DeepMind theorises that yet another monitoring layer for agentic AI may be required to make sure these free-roaming agents don’t cause too much trouble.

If that sounds daunting, you’re not alone. According to recent research by Gartner, up to 40% of enterprises using agentic AI will either demote or decommission these applications because their guardrails have proven inadequate. Preventing this, the research organization advises companies will need to adopt a graded approach to access, with autonomy for AI agents governed by the level of authority actually determined by the task they’ve been assigned.

Trkay is doing something similar at FICO. “Rather than chase every new model or capability, I focus control on the decisioning layer beneath it,” he says. “That includes the rules for what data AI can access, what it can act on autonomously, and where a human must sign off.”

All this, he adds, is defined from the start by a cross-functional governance committee, clear RACI ownership across standards and monitoring for the application, and a platform approach that enforces responsible AI usage. “Built well, that layer doesn’t need to be rebuilt every time the technology shifts,” says Trkay. “New capabilities plug into an existing structure of accountability, which is the difference between reacting to AI and running it.”

For his part, Trkay is skeptical that rigid guardrails can effectively restrain agentic AI from its most destructive impulses. “They tend to get worked around, either because they slow teams down or they’re too inflexible for legitimate edge cases,” he says. Effective guardrails for agentic AI, he adds, have to be specific enough to be meaningful, and adaptable enough to hold up as use cases multiply, backed by strong architecture, testing, and ongoing monitoring. “The one non-negotiable is the audit trail,” he says. “Whatever autonomy a system has, we need a record of what it did, and why.”

Staying grounded

For CIOs who don’t relish the challenge of setting obstacles and passing points for AI agents scurrying through their maze of networks, there’s always the option of delaying the inevitable by not immediately deploying such applications. Some might not even have the choice, at least for now. “We’re seeing the cost of tokens go up with those new models, because they’re expensive to run,” says Lewis. “But as new models come out, we’re going to see that decrease for some of those older models that were good.”

There is time, then, for CIOs to learn how to keep their head above the torrent of ever more new and powerful agentic AI applications. Whether they’ll be capable of doing so when the next great innovation is sold by Silicon Valley is an open question. Colin Constable, CTO of software development firm Atsign, styles himself as an internet optimist. Even he, however, is dismayed by the decreasing number of junior developers succeeding their more senior counterparts as they retire. That’s a big problem when so many of the former are relying on AI to assist them at work.

“We hand over lots of these decisions to LLMs without making good architectural choices,” says Constable. “If you haven’t been burnt by these things in the past, how would you know the difference?”

For their part, Constable and his colleagues get around this problem with a combination of AI-on-AI oversight of code quality, maintenance of constant dialogue within the team about new coding quandaries, and letting senior developers teach junior counterparts about some of the more avoidable mistakes in their profession. It’s a way of adapting to AI acceleration that points, unequivocally, toward CIOs diffusing responsibility for deeply educating the business about the technology. And if they continue to get it wrong, at least the agent will apologize.