Agentic AI systems don’t fail suddenly — they drift over time

Agentic AI systems don’t usually fail in obvious ways. They degrade quietly — and by the time the failure is visible, the risk has often been accumulating for months.

As organizations move from experimentation to real operational deployment of agentic AI, a new category of risk is emerging — one that traditional AI evaluation, testing and governance practices often struggle to detect.

A subtle pattern

Unlike earlier generations of AI systems, agentic systems rarely produce a single catastrophic error. Instead, their behavior evolves incrementally as models are updated, prompts are refined, tools are added, dependencies change and execution paths adapt to real-world conditions.

For long stretches, everything appears fine: outputs look reasonable, KPIs hold and no alarms fire. Yet underneath the surface, the system’s risk posture may already have shifted, long before failure becomes visible.

This pattern is increasingly being recognized beyond individual implementations. Industry groups such as the Cloud Security Alliance have begun describing cognitive degradation in agentic systems as a systemic risk — one that emerges gradually over time rather than through sudden failure.

In my work evaluating agentic systems moving from pilot phases into real operational settings, I’ve seen this pattern repeat across domains.

Understanding — and detecting — that drift is becoming a central operational challenge for CIOs and CTOs.

Why agentic systems drift differently in production

Most enterprise AI governance practices evolved around a familiar mental model: a stateless model receives an input and produces an output. Risk is assessed by measuring accuracy, bias or robustness at the level of individual predictions.

Agentic systems strain that model. The operational unit of risk is no longer a single prediction, but a behavioral pattern that emerges over time.

An agent is not a single inference. It is a process that reasons across multiple steps, invokes tools and external services, retries or branches when needed, accumulates context over time and operates inside a changing environment. Because of that, the unit of failure is no longer a single output, but the sequence of decisions that leads to it.

In practice, failure shows up in decision sequences rather than individual predictions because behavior is no longer binary but probabilistic and contextual. Two executions of the same agent with the same inputs can legitimately differ, even when nothing is wrong.

This stochasticity is not a bug; it is inherent to how modern agentic systems operate. But it also means that point-in-time evaluation, one-off tests and demo-driven confidence are structurally insufficient for production risk management.

Most agentic systems are still evaluated using familiar techniques: individual executions, curated scenarios and human judgment of output quality. These methods are effective in controlled demonstrations, but they do not translate well to production environments.

This gap between demo performance and real-world behavior has also been observed in recent academic work, including research from Stanford and Harvard examining why many agentic systems perform convincingly in demonstrations but struggle under sustained, real-world use.

In demonstrations, prompts are fresh, tools are stable, edge cases are avoided and execution paths tend to be short and predictable. In production, those conditions change in ways that are harder to anticipate. Prompts evolve, tools change, dependencies fail intermittently, execution depth varies and new behaviors emerge over time. The same system that looked reliable in a demo can behave very differently months later, even though nothing “broke.” The result is often a false sense of confidence. Systems that look reliable in demonstrations may already be drifting operationally.

This helps explain a familiar pattern many enterprises experience: an agent performs well in pilots, passes review gates and earns early trust — only to become brittle, inconsistent or riskier months later, without any single change that clearly “broke” it. From an operational standpoint, this is not a surprise. Rather, it is the predictable outcome of relying on demonstrations instead of diagnostics.

In real environments, degradation rarely begins with obviously incorrect outputs. It shows up in subtler ways, such as verification steps running less consistently, tools being used differently under ambiguity, retry behavior shifting or execution depth changing over time. None of these changes necessarily produce incorrect answers in isolation. By the time output quality degrades, the agent’s behavior has often been unstable for some time.

Lessons from a credit adjudication pilot

In a credit adjudication agent pilot I worked on, we evaluated an agent used to support high-risk lending decisions. The agent didn’t make approvals on its own. It gathered information, ran verification steps and produced a recommendation that a human reviewer could accept or override.

At the start, the behavior looked solid. In pilot reviews, the agent consistently ran an income verification step before producing a recommendation. The outputs were generally conservative and aligned with policy. Based on standard evaluation criteria, there were no obvious concerns.

Over time, several small changes were made. Prompts were adjusted to improve efficiency. A new tool was introduced to handle a narrow edge case. The model was upgraded. Retry logic was tweaked to reduce latency. None of these changes stood out on their own and no single run produced an obviously wrong result.

What changed was only visible when looking across runs.

When I reviewed execution behavior over repeated runs with similar inputs, a pattern started to emerge. The income verification step that had been reliably invoked earlier was now skipped in roughly 20% to 30% of cases. Tool usage under ambiguous conditions became less consistent. The agent reached conclusions more quickly, but with less supporting evidence.

From an output perspective, the system still appeared to be working. Reviewers often agreed with the recommendations and there were no clear errors to point to. However, the way the agent arrived at those recommendations had shifted. That shift would not have shown up in a demo or in spot checks of individual executions. It only became apparent when behavior was examined across runs and compared to earlier baselines.

Nothing failed and there was no incident, but the system was no longer behaving the same way. In a credit context, that difference matters.

Why governance needs diagnostics, not just policy

Governance frameworks are beginning to acknowledge these risks, which is a necessary step. They define ownership, policies, escalation paths and controls. What they often lack is an operational mechanism to answer a deceptively simple question:

“Has the agent’s behavior actually changed?”

Without operational evidence, governance tends to rely more on intent and design assumptions than on observed reality. That’s not a failure of governance so much as a missing layer. Policy defines what should happen, diagnostics help establish what is actually happening and controls depend on that evidence. When measurement is absent, controls end up operating in the dark, creating a governance posture that can look robust on paper while developing blind spots in live systems — precisely where agentic risk tends to accumulate.

In other domains, enterprises already know how to manage this kind of risk by establishing baselines, running repeated measurements, analyzing distributions rather than individual outcomes and looking for persistence instead of noise while separating structural changes from observed effects. Agentic AI systems warrant the same operational discipline. That kind of discipline — establishing baselines, running repeated evaluations and separating signal from noise — has long been standard practice in other high-risk software domains, including how the SEI frames testing and evaluation for complex AI-enabled systems.

Applying this discipline to agentic systems points towards a diagnostic approach that observes behavior without interfering with execution, treats drift as a statistical signal rather than an anecdote, separates configuration changes from behavioral evidence and produces artifacts that operations and risk teams can review. This is not about enforcing behavior; rather, it is about being able to see what’s happening.

No single execution is representative

From an operational perspective, detecting agentic drift looks different from traditional model evaluation.

One of the challenges in detecting agentic drift is that no single execution is representative. What matters is how behavior shows up across repeated runs under similar conditions. Over time, that also means baselines need to be behavioral rather than normative. The goal is not to define what an agent should do in the abstract, but to understand how it has actually behaved under known conditions.

Structural change adds another layer of complexity. Configuration updates — such as prompt changes, tool additions or model upgrades — are important signals, but they are not evidence of drift on their own. What tends to matter most is persistence. Transient deviations are often noise in stochastic systems, while sustained behavioral shifts across time and conditions are where risk begins to emerge.

Taken together, these observations point toward a diagnostic discipline that complements existing governance and control frameworks. Rather than enforcing behavior, it provides visibility into how agent behavior evolves — allowing organizations to reason about risk before incidents or audits force the issue.

The timing of this issue is not theoretical. In 2026 and beyond, agentic systems are being embedded into workflows where subtle behavioral changes carry real financial, regulatory and reputational consequences. In that environment, “it looked fine in testing” is no longer a defensible operational posture.

At the same time, regulators are paying closer attention to AI system behavior, internal audit teams are asking new questions about control and traceability, and platform teams are under growing pressure to demonstrate stability in live environments.

For CIOs and CTOs overseeing agentic deployments, a few implications follow. Single executions are rarely evidence of stability; output quality often needs to be evaluated separately from behavioral consistency and change should be expected even when no visible failures are present. Measurement must take precedence over intuition and agent behavior should be treated as an operational signal rather than an implementation detail.

The goal is not to eliminate drift. Drift is inevitable in adaptive systems. The goal is to detect it early, while it is still measurable, explainable and correctable, rather than discovering it through incidents, audits or post-mortems. Organizations that make this shift will be better positioned to deploy agentic AI at scale with confidence. Those that do not will continue to be surprised by systems that appeared stable — until they weren’t.

From experimentation to trust

Agentic AI systems promise real efficiency and capability gains and many organizations are already seeing value from early deployments. The challenge is that trust in these systems can’t rest on demos alone. As agentic systems move into higher-risk environments, the question shifts from “Does it work?” to “Is it still behaving the way we expect?” That shift doesn’t slow innovation — it gives leaders a way to scale it with confidence.

Organizations that make this transition earlier tend to spot issues sooner, respond with more clarity and avoid being surprised later by systems that appeared stable at first.

This article is published as part of the Foundry Expert Contributor Network.
Want to join?