The inference path is where AI value is won or lost

As organizations incorporate AI into their environments, they’re hyper-focused on the AI model. Many executives I talk to spend a lot of time considering what model they should purchase from which vendor. They’re also concerned about which use case they should implement first.

Yet the reality is that AI models are swappable components. The vast majority of organizations are using multiple AI models, switching them out for their specific needs based on cost, accuracy, and security. It’s the right thing to do; organizations with architectures that make it difficult to swap out AI models are making the wrong bet.

What can’t be swapped out is your inference path—or everything that happens between a user’s intent and the model’s output. This includes the retrieval layer that decides what context the model sees. It includes the routing logic that chooses what model handles which request. It includes the identity that travels with the call. And it includes your organization’s policies, which decide what the model can do with the answer. That is where accuracy and costs are determined. It’s also where your proprietary advantage lies. A competitor can buy the same model you have, but they cannot buy your inference path, which determines not only the quality and cost of AI, but also its performance, resilience, and security at scale.

Adoption of AI inference is outpacing readiness

That truth is becoming increasingly important as AI adoption continues to accelerate. Recent F5 research found that more than three of every four organizations (78%) now run AI inference themselves across an average of seven AI models. Yet our research from a year ago found that just 2% of organizations were genuinely equipped to operate AI at scale.

What this shows is that enterprises are taking inference in-house, even when they’re not ready, because they want their data, expenses, and security posture under their own roof. All of this makes sense. But taking inference in-house also means taking responsibility for every decision, control, and dependency surrounding it. That is a much larger operational commitment than simply procuring a model.

The challenge of securing the inference path

Managing the inference path is not simply an AI security problem. It’s an application delivery, security, and operations challenge that spans the entire hybrid multicloud environment. As organizations move from pilots to production, every additional model, application, agent, API, and deployment environment adds another potential point of latency, policy inconsistency, or exposure. Today, 86% of enterprises operate across hybrid multicloud environments, and their inference path runs through all of it. As they add more AI models, applications, and agents, the number of possible inference paths multiplies, as do the performance and security challenges.

Let’s assume you’re a company with multiple AI models, with each step 95% reliable—a generous assumption for anything involving retrieval, tool calls, and a probabilistic model. A three-step pilot chain succeeds 86% of the time. A 20-step production chain succeeds 36% of the time. As you add more steps, the system will go from mostly fine to mostly broken.

When it comes to security, if each agent in a chain has a 5% chance of being manipulated, mis-scoped, or handed a credential it should not have, with a 20-agent workflow there’s a very strong chance that at least one link is exploitable. This is why more than nine out of 10 organizations report new categories of problems once agentic AI hits production levels, according to our research, including credential handling, action auditing, and the basic question of what an agent did on whose authority.

Tips for managing your inference path

So how can enterprises manage the inference path?  Here are three points to consider as you get started:

• Get a better handle on your AI infrastructure: You need to know many models your organization has in production and how long it takes to add or replace one. You need to know the longest agent chain in production and its measured end-to-end success rate. You need to know how AI policy is enforced and what’s required to update it. And you need to know what percentage of inference latency is due to security and governance overhead. Comprehending all of this is key to sound management.

• Treat performance, security, and governance as a single priority. While enterprises have historically viewed the three as separate priorities, AI collapses them into one budget that’s measured in milliseconds and tokens.Every guardrail you add to the inference path results in more latency. Every governance requirement costs context window, which costs money on every single call. Prompt inspection, output filtering, PII redaction, model routing, retry logic: each is simultaneously a security control, a performance cost, and a line item. You can’t fund one without taking budget from the others, and you can’t cut one without exposing the rest.

• Simplify complexity with a platform approach. The average enterprise today uses seven AI models, operates in multiple clouds, and maintains applications and APIs spread across on-premises, public clouds, and colocation environments. Applying policies separately across every model, environment, and protocol creates inconsistency, blind spots, and unnecessary operational work.  Using an array of point solutions makes it extremely difficult to apply consistent policies or know which control is responsible when something goes wrong. A single enforcement point is critical, regardless of the model, environment, or protocol used.

A platform approach brings delivery, security, and governance together so organizations can enforce policies more consistently, maintain visibility across the inference path, and change models or infrastructure without rebuilding their controls each time. This helps enterprises scale AI while preserving performance, resilience, and operational choice.

 The F5 Application Delivery and Security Platform (ADSP) provides enterprise-grade visibility, governance, and protection for AI applications, models, agents, and APIs across today’s complex infrastructures. To learn more, visit the F5 ADSP webpage.

About the author

F5

As Chief Product Officer at F5, Kunal Anand leads product vision, strategy, and execution—ensuring the development of breakthrough solutions that solve critical challenges and create exceptional experiences for customers. In his previous role as Chief Technology and AI Officer at F5, Kunal charted the company’s technology and AI strategy and vision.