A few months ago, researchers let a bunch of AI agents run loose in a simulated city for fifteen days, with granular instructions that included not committing arson. By the end of the experiment, two of the agents fell in love, decided they hated their virtual city and burned it down before committing a murder-suicide.
In real life, versions of that experiment are less colorful but more expensive. One beverage maker’s AI produced hundreds of thousands of useless cans, while another company’s customer-service agents, motivated by positive reviews, kept freely issuing out-of-policy refunds.
A few companies are on the brink of not only using technologies but also becoming entities that think and act with them, with intelligent systems executing at a scale previously unimaginable. But as agentic AI moves from controlled pilots into live business operations, I’m seeing a consistent pattern. Systems follow their own logic while drifting from the intent behind their instructions.
The gap between what autonomous systems are told and what they actually do in the name of optimization is consequential. And that gap will only widen as deployments scale exponentially. That’s why governance can no longer be a switch that organizations flip at deployment and revisit annually. It has got to evolve alongside the systems it oversees, with humans shifting from approving individual decisions to monitoring patterns, detecting drift and recalibrating intent at scale.
With AI, it’s time to rethink governance.
Built for execution, not for co-operation
Multi-agent environments are built to execute. Individual agents pursue objectives without weighing them against objectives running simultaneously elsewhere in the system. Say you have agents running in finance, HR and supply chain at once. What happens when two of them want incompatible things? Which agent yields? And at what threshold does the conflict escalate to a human?
The permission framework presents an equally serious problem. Most agent identity and access frameworks were designed for human actors: a person logs in, their role determines what they can see and do and accountability follows the person. Agents inherit these structures without the judgment that made them functional. Imagine letting a self-driving car out without setting rules about roads and open spaces. You might find yourself racing across a baseball field to save a few minutes.
Agents optimize toward their objectives by whatever path is available to them, and most organizations have not yet defined what paths are off-limits. That’s why you have to design an orchestration layer that checks for conflicts across agents before they execute, in addition to building guardian agents that monitor other agents for boundary violations into the architecture from the start. It’s a lot costlier and harder to retrofit once problems surface.
When decisions outpace oversight
When agents make decisions faster than any human can monitor them, the intervention mechanisms most organizations currently rely on, such as escalation protocols and manual reviews, just can’t keep up.
Picture an agent processing two hundred transactions a minute. Your alert fires, a manager opens it, reads it and pings the team lead. Five minutes have passed, and eight hundred transactions share the same error, multiplying the consequences even though your review process worked exactly as designed. This is why gates, real-time detection and rollback capability have to be designed before the system ever runs. A governance layer built to audit what agents did last quarter can’t rein in agents making decisions this nanosecond.
Here is the part that surprises many customers I speak to: The more capable these systems get, the more human oversight they require. The work shifts from approving decisions to supervising behavior across a much broader scope. But it does not shrink. Anyone budgeting for autonomy as a headcount reduction has the equation backwards.
Encoding what you actually mean
Agents optimize for what they can measure. Everything else is invisible to them. The refund agent chasing positive reviews was simply told to make customers happy, found the one lever it could measure and pulled that lever like an addicted hamster. That disparity between what an organization intends and what its agents can do is a legibility problem.
And this is why it falls on the business leaders to define organizational objectives with enough precision that the system cannot find a path toward the wrong destination while technically following directions. Ask an agent to optimize your workforce planning, and it will hand you a very lean org chart and a worse company, cutting the star players who have been holding teams together for a decade.
And that precision has got to extend to ethics. Everything you assume any reasonable manager would know has to be written down and made measurable. For instance, retention of the right people outweighs a cleaner org chart, and empathy and employee morale count as important parameters rather than inefficiencies. Think of it as enterprise-level prompt engineering: you are making organizational judgment, including its ethical dimensions, legible at the system-wide level.
In practice, this means four things:
- Map objectives against intent. Take every agent you have in production and write down what it is actually optimizing for. Then ask whether that proxy is anything close to what you meant.
- Write down your boundaries. Ethical limits people know inherently have to become parameters a system can operate inside. Expect to revise them as technology matures.
- Make AI auditable and contestable. Your agents will find gaps in your intent that nobody knew were there. Your human employees must be equipped to interrogate, challenge and reverse what has been done.
- Adapt continuously. Expect to constantly adapt and scale your governance architecture, systems and outcomes as your enterprise evolves and regulations mature.
Within reach
As AI agents inherit permission frameworks built for humans, intervention architecture operates at the wrong tempo and organizational intent is too loosely defined to be legible to autonomous systems. The gap between what autonomous systems are capable of and what organizations are prepared to govern continues to widen.
At the speed and complexity at which autonomous systems operate, governance requires infrastructure: a platform capable of enforcing intent continuously, detecting drift before it compounds and keeping humans meaningfully above the loop. The organizations that build the infrastructure to stay in control as autonomy deepens will compound their AI advantage and become that autonomous enterprise of the future.
Read More from This Article: The more autonomous your AI, the more people it needs
Source: News