The case and model for real-time AI cost visibility at the infrastructure layer

I spend most of my time inside other companies’ engineering teams, building systems to track and optimize AI spend. The conversation almost always starts the same way. Someone pulls up a dashboard, points at a number bigger than it should be, and says some version of “we know it went up; we just can’t tell you why.”

I use an analogy for it: AI-era CIOs are like city planners optimizing a busy intersection. They can measure the volume and hear the pleas to fix congestion, but can’t tell whether a vehicle is a truck or a bike, or why it’s on the road. Without that, they can’t design the right fix, so they build a highway at great expense when the data would show all it needed was a bike lane.

Every model choice and budget conversation happens against that blurry picture, and the traffic gets heavier every quarter. Gartner expects worldwide AI spending to grow 47% this year, with agentic AI software up roughly 141%. By 2028, it projects an average Fortune 500 enterprise will run over 150,000 agents, up from fewer than 15 in 2025.

Why the cloud playbook can’t answer the AI question

AI presents a fundamentally different problem than cloud cost management, where we answered, “whose spend is this?” largely by tagging the resource. A VM has an owner, a bucket belongs to a team and FinOps optimizes from there. Billing was slow but acceptable, because spend moved inside predictable bands, a human provisioned each resource before it cost anything, and governance capped how fast costs grew. Surprises were unpleasant, rarely existential.

AI took that model, put it to the test and laughed it out of the room. A token call isn’t a resource the way a VM is, so tags have nothing to attach to. To make them affordable, hyperscalers run large models on shared, multi-tenant infrastructure, dropping the per-token price but stripping out the granularity needed to track and control spend.

One API key can carry a dozen workflows across three teams, and the consumer is often an autonomous agent, not a person. The failure modes are also new. An agent can drift off task, loop on a retrieval endpoint hunting for an answer it can’t find, and restart from scratch when it comes up empty. At a few dollars per million tokens, it seems trivial, until that loop runs across thousands of parallel sessions and clears five figures in a day. None of it trips a provisioning gate or maps to a taggable resource. The control points that cloud governance leaned on don’t exist for AI.

The consequences can be severe. Uber spent its entire 2026 AI budget in four months after Claude Code usage ran far ahead of its projection, and they’re far from alone. I’ve worked with plenty of large enterprises hitting the same thing without the headlines. In almost every case, the teams driving the spend weren’t accountable for the budget, and no one saw the scale until it hit an invoice. It’s a credibility destroyer, as these overruns erode margins and stakeholder confidence for the leaders on whose watch it happens, even when their only fault was doing their best with the tools they had.

The data backs this up. Recent DoiT research into enterprise finance leaders found that 89% of the organizations that rate themselves most mature at FinOps still overspent on AI last year, and by the widest margin in the study. Again, the teams with the best cost discipline overspent the most. They built excellent governance for human-gated, taggable resources, then tried to retrofit it for workloads that have neither. They see the overrun, just too late to stop it, and without the insight to know where to focus.

The instrumentation era got us closer, at a real cost

Our best answer was to instrument the application itself: wrap every LLM call in an OpenTelemetry span, inject cost-allocation metadata and propagate that context across a multi-agent workflow so token counts roll up to a budget owner. I’ve done it dozens of times, and it works: accurate per-request attribution, far better than splitting the monthly bill by best guess.

But it was always a workaround. You can only measure what you thought to instrument, and even when you do that, the answer often arrives too late.

One customer I worked with recently had built about ten agents that worked together to automate security and quality evaluation of their software, most of them running constantly. Their bill kept climbing with no obvious cause, and it took a long investigation to find why: one agent, running 24/7, was burning 20 times the tokens of any other, caught in an infinite loop that respawned a never-ending process every time it spun up. The signal was right there in the data, but it only surfaced after weeks of digging and real money out the door.

There’s an irony too. AI is supposed to buy back engineering time, and instrumentation spends it right back. You put senior engineers on measurement plumbing to control the cost of the thing meant to make them more productive. Plus, it’s fragile; every new model, SDK and agent framework is another integration to keep alive. “Instrument everything you might ever build” isn’t realistic when tens of thousands of employees and countless agents launch new workflows daily.

What changes when attribution moves to the infrastructure layer

The approach now emerging, and the one changing how I run these engagements, moves attribution from the application down to the infrastructure. Instead of teams describing their spend through code they wrote, you observe what’s running underneath them.

The mechanism is a kernel-level sensor, the same eBPF technology that security and observability tools use to watch system calls without touching the applications above them. It maps each unit of GPU, CPU, memory and network back to the process, container and request that caused it, then joins every outbound model call with provider cost data so token spend on Anthropic, OpenAI, Gemini or Bedrock is matched to the workload and agent that drove it.

This delivers the same answer OpenTelemetry can give you, at equal or better accuracy, but in real time and without the instrumentation work. Nothing has to be planned in advance, and nothing gets missed. For the first time, the data to manage AI spend is continuously available at an actionable grain. That runaway agent my customer dealt with would have been identified and addressed as soon as it began to snowball.

Governance stops being a top-down audit

Changes in governance is the part I find most interesting, because with continuous, accurate attribution, you can design governance purpose-built for AI rather than retrofitting a system built for an era with different physics.

Strategy still belongs at the top, with the CIO and finance leaders setting the direction, the budgets and the priorities. But daily responsibility for staying inside those lines shifts toward practitioners. Cloud governance was top-down because provisioning ran through the few people with the full business context to weigh it. With AI, spending decisions happen everywhere at once, across shared keys, agents and dozens of teams, so adherence has to sit with the people making them.

Real-time attribution makes that possible. An engineer can see the cost of what they’re building as they build it. A frontier-model call when a smaller model would do, a retrieval loop fanning out across a fragmented data store, an agent retrying a failed tool call in a tight loop — those stop being mysteries at the quarterly review; they’re caught while the work is still warm. And because the same numbers reach leadership, they can weigh spend against value and steer the portfolio rather than litigate a bill nobody can explain. Back at the intersection, the planner can finally tell the trucks from the bikes, and so can every engineer on the road.

What visibility is actually for

So, the biggest foundational roadblock (pun intended) that tripped up even the most disciplined teams is solvable now in a way it wasn’t a year ago. But solving it was never the real goal. Visibility is a means to an end; it just had to be solved first.

Real-time, granular attribution lets companies start treating AI as a managed investment instead of a pay-and-pray experiment. When you connect spend to the work, the work to an outcome and the outcome to the business case that justified it, you can govern AI with intention and put money where it earns its keep.

The last two years rewarded productivity to whoever shipped faster with AI. The next phase gets won on efficiency and impact, by whoever gets the most value per dollar of inference and can prove it. That’s a matter of strategy, and for the first time the data exists to compete on it. What you’re really adding for the customer, and what it costs, stop being a guess.