Many autonomous agents doomed by governance failures

By next year, 40% of enterprises will have their autonomous AI efforts in part derailed by gaps in governance discovered only after production incidents, a recent report from Gartner predicts.

The reason is that enterprises are treating AI agent governance as binary, either locked down or fully trusted, “and that is the root cause of failure,” said report author Shiva Varma, senior director analyst at Gartner. These failures will force enterprises to demote or decommission some agents.

“Agents operate at different autonomy levels and across different trust boundaries. When the same controls are applied indiscriminately, organizations encounter two common failure modes: over-restriction of simple agents, which slows delivery and drives shadow development, or under-restriction of more autonomous agents, which increases operational, security and compliance risk,” he wrote.

To combat this, Gartner recommends a multi-tiered governance approach, based on agents’ degree of autonomy.

“Autonomy level and scope must be assessed independently,” Varma wrote. “Autonomy level defines an agent’s ability to act, while scope defines the breadth of data, systems and permissions it can access. Governance decisions should consider both dimensions, as risk increases with either expanded autonomy or expanded scope.”

Gartner’s four level governance model, he said, only looks at autonomy level, since access controls scale separately.

In the model, agents that have read-only access to defined data sources, and only display results to the requesting user, are designated Level 1, “Observe”. Governance of these agents, said Varma, should focus on baseline controls: scoped data access, user authentication, usage logging, and “basic functional and security testing.”

Level 2 (“Advise”) agents, he said, also only have read-only access, but generate recommendations to users in activities such as email drafting, report or code generation, or decision support. But since their advice can affect human judgements, Level 1 constraints aren’t enough; they must be extended to include accuracy and hallucination testing, and domain-specific quality evaluation. In addition, user training needs to include information on the appropriate levels of reliance they should place on the results.

At Level 3, “Act with Approval”, agents act with human approval, and perform tasks such as writing data, sending communications, or modifying configurations. It requires even stronger controls, building on Level 2 governance. “At this level, human review is effective only if it remains a meaningful control,” said Varma. “Without strong security testing, clear approval workflows with audit trails and agent‑specific incident response procedures, approvals can degrade under time pressure or approval fatigue, creating a false sense of safety while expanding the attack surface.”

The toughest restrictions must be applied to fully autonomous agents at Level 4, Varma said. They can execute actions independently within defined guardrails, while humans only review exceptions and look at audit logs and aggregated outcomes. That means, in addition to the controls in levels 1-3, these agents should have comprehensive guardrail definitions, rollback capabilities for their actions, continuous monitoring, and a way to stop an agent’s operation if it violates its thresholds. In addition, there needs to be continuous red team testing, clear ownership and accountability for its actions, and business continuity procedures should the agent fail.

Varma advised software engineering leaders to audit agents currently in use and match their governance level to their autonomy.

Sanchit Vir Gogia, chief analyst at Greyhound Research, welcomed Gartner’s recommendations. “Applying one governance model to all agents is rather like applying the same control regime to a receptionist, a finance controller, a database administrator, a claims handler, and a procurement head because all of them use a laptop. It is tidy on paper. It is nonsense in practice.”

To be effective, he said, governance models must recognize that the riskiest thing about an agent is not always what it says, but what it can do next.

Valence Howden, advisory fellow at Info-Tech Research Group, agreed. “At [Level 4] the governance system must be adaptable and the organizations will need to move to more resilient anti-fragile adaptive models.”

Gogia added, “The real governance problem is not model intelligence. It is delegated operational authority moving across trust boundaries faster than enterprises can instrument, constrain, or audit it. Governance is not a brake on AI adoption. It is the precondition for scaling it.”

His advice to CIOs is blunt: “Do not scale agents faster than you can govern their authority. A small number of well-governed agents will create more enterprise value than a sprawling estate of clever, fragile, over-permissioned digital apprentices. The future of AI agents is not autonomy without restraint. It is autonomy inside well-designed boundaries.”