During a recent advisory engagement, I watched 50 different customers ask the same return-policy question on a Tuesday. The AI support system answered all 50 correctly. It also executed fifty independent calls to a frontier model, ingested the same internal policy document 50 times, and generated essentially the same response 50 times. The company paid full price for every run.
Nothing was broken. No error occurred. No alert fired. The system was working exactly as designed, and it was quietly wasting a meaningful share of the technology budget.
This is how the problem hides. Costs rise gradually, with no new feature or traffic spike to explain them. Finance sees growing variable spend and asks for justification. Engineering checks the dashboards and sees stable, error-free behavior. Nothing appears broken, so nobody knows what to fix.
The system is not failing. It is repeating itself.
In May, I wrote about the routing problem: Paying too much per request by using frontier models for tasks a cheaper one handles. This is the opposite failure, and the harder one to see: paying too many times for the same result. The two problems are independent. A company can route every task to the cheapest capable model and still pay full price to answer the same question ten thousand separate times. Fix what each request costs first. Then fix how many requests you are paying for. This article is about the second problem.
When efficiency hides architectural duplication
The system in that engagement was designed to ingest a customer email, reference internal policy documents and draft a compliant response. It reduced manual effort, improved response times and spread quickly across regional teams. Operationally, the deployment was a success. As a capital allocation, it was a leak.
Every similar request triggered a net-new API call. The system had no mechanism to recognize that it had already done this work, so it recomputed the same result on demand, at full price, indefinitely.
The invisible cost pattern in stateless systems
This type of financial inefficiency is difficult to detect because it does not appear as a system failure. Instead, it shows up as:
- Steadily increasing API usage
- Rising monthly costs without a proportional increase in business value
- Identical outputs generated repeatedly across different user sessions
From a technical perspective, each HTTP request is valid and returns a successful response code. From a financial perspective, the enterprise is paying multiple times for the same result.
BCG’s 2026 research on building an AI-first cost advantage found that 60% of companies report minimal or no value from AI, including cost reductions or revenue gains, despite significant efforts. A significant and often unmeasured portion of that missing value is driven by redundant, unmanaged compute cycles that quietly bloat the operating model. In my own audits of SaaS portfolios, nearly 80% of AI queries are repetitive.
In traditional software, querying a database a thousand times for the same record is inefficient, but the marginal cost of those extra queries is practically zero. In generative AI architectures, this effect is amplified because every request carries a direct, variable price tag based on token consumption.
Why stateless architecture drives capital waste
Most enterprise AI systems are designed to respond to requests independently. They operate as stateless interfaces.
Each input is treated as an entirely new problem to solve, even when it is functionally identical to previous inputs processed just moments earlier. The system has no memory of prior results at the application level. If a thousand employees ask an internal HR chatbot to summarize the new corporate travel policy on a Monday morning, the AI model reads the fifty-page travel policy one thousand separate times.
This creates a default architectural behavior where every request triggers expensive new computation, regardless of whether the correct answer already exists in the system. From a software engineering perspective, building a stateless application simplifies the initial design and accelerates time to market. From a financial perspective, it introduces a permanent mechanism for capital waste.
The scale effect and semantic caching
At a low volume of experimental usage, this inefficiency is negligible and easy to ignore. At full enterprise scale, it becomes a material cost driver that directly impacts corporate gross margins.
Consider a production system handling customer support responses, legal document summaries or internal technical knowledge queries. These use cases inherently involve highly repeated patterns:
- Users ask similar questions
- Systems query similar documents
- Models generate similar outputs
If each instance is processed independently, the organization pays a continuous premium for work that should have been recycled.
Cloudflare’s AI Gateway documentation emphasizes serving identical requests directly from cache instead of the original model provider, cutting both cost and latency by avoiding repeated API calls. Semantic caching extends the same principle to requests that are worded differently but mean the same thing. In AI systems where every token incurs a cost, a caching layer is not just a performance optimization. It is a mandatory financial control.
Why traditional FinOps tooling is blind to redundancy
This issue persists in large organizations because it sits in a blind spot between two operational domains. Engineering teams see correct system behavior, while finance teams see aggregate monthly costs. Neither side possesses the tools required to see the full picture.
Traditional cloud financial operations platforms are highly optimized for managing infrastructure utilization. They are excellent at identifying idle servers, right-sizing databases and recommending reserved instances. However, they are blind to semantic equivalence. A FinOps dashboard can tell you that your API spend increased by 20% this month. It cannot tell you that the increase was entirely driven by your system answering the exact same question ten thousand times.
Without visibility into the actual context of the requests, the redundancy remains hidden from the executives tasked with managing the budget.
The difference between model capability and operational reuse
It is critical to distinguish this redundancy problem from the model routing problem covered in the May column.
Using an overly expensive model for a simple task is a capability mismatch, which inflates the cost per request. However, even when an organization fully optimizes its routing and selects the most cost-effective model available, the system will still bleed capital if it recomputes identical results.
The distinction matters for long-term governance. A model selection problem means the cost per individual request is too high. A redundancy problem means the total volume of requests is unnecessarily high. Upgrading to a faster, cheaper model does not solve the redundancy problem. It simply means you are computing redundant work at a slightly lower price point. The core issue is not technical capability. It is the lack of operational reuse.
The impact on corporate cloud credits and latency
As these unmanaged systems scale, redundant computation creates severe downstream impacts beyond the monthly invoice.
Many enterprises operate on prepaid cloud consumption agreements or allocated AI credits. Systems that suffer from high redundancy burn through these annual allocations prematurely, forcing CIOs to request unexpected budget increases halfway through the fiscal year. Redundant processing also ties up valuable compute bandwidth. When a system is busy re-summarizing a document it has already processed a hundred times, it introduces unnecessary latency for users waiting on genuinely unique queries.
Engineering teams often respond to these friction points by:
- Artificially limiting user access
- Restricting product features
- Demanding higher infrastructure budgets
None of these actions address the root cause.
A different framework for measuring AI unit economics
To solve this problem permanently, organizations need to shift how they evaluate and track their AI spend.
A 2026 study from Harvard Business Review Analytic Services found that while 59% of organizations now have AI in production, ROI is among the least likely performance metrics to have improved. How companies measure AI value has not kept pace with how they deploy it. Instead of focusing entirely on cost per request or cost per thousand tokens, engineering and finance teams need two numbers that neither currently owns.
- Cost per unique outcome: total AI spend divided by the number of distinct results produced. This is the true unit cost of the system.
- Redundancy rate: the share of requests that duplicate work the system has already performed. This is the size of the leak.
A system can have an excellent cost per request and a terrible cost per unique outcome. When the same business outcome is generated repeatedly from scratch, the redundancy rate is where the missing margin went.
The one-afternoon audit
You do not need new tooling to find out whether this is happening to you. Pull your 100 most frequent prompts from last month’s logs. Run them through an embedding model and cluster them by semantic similarity. Count how many clusters contain more than one request. That count, divided by 100, is a rough redundancy rate.
Teams running this exercise for the first time typically find that between half and four-fifths of their traffic is a variation of a question the system has already answered. Every one of those requests is a candidate for a semantic cache. The audit takes an afternoon. The architecture change usually pays for itself in the first month.
The new operational constraint
This inefficiency persists because it has no clear organizational ownership:
- Engineering leaders focus on output correctness and system uptime.
- Finance leaders focus on managing total aggregate spend against the yearly budget.
- No specific role is responsible for identifying and eliminating redundant computation inside the application layer.
Generative AI introduces a rigid economic constraint that traditional software did not emphasize at this level. In the AI era, repeated work has a direct, measurable and unavoidable financial cost.
Your AI system does not need to be broken to be deeply inefficient. At pilot scale, redundancy is a quirk. At enterprise scale, it is a structural cost driver. And because it never triggers an error code, it is one of the easiest architectural problems to overlook and one of the most expensive governance failures to ignore.
This article is published as part of the Foundry Expert Contributor Network.
Want to join?
Read More from This Article: You’re calling the same AI API thousands of times: Why redundant requests are driving hidden AI costs
Source: News