When generative AI (GenAI) first arrived, I was in charge of a large team of data and machine learning engineers. We had built a full ML platform from scratch and had dozens of models in production delivering measurable results. AI was working for us.
But with the novelty of GenAI came the hype. Under pressure from the board, the question was no longer whether the technology could help our product, but how fast we could find a place for it. Our recommender system, which sat at the core of our company’s product, was the obvious candidate.
I pushed back. The Large Language Model (LLM) was trained to predict text, I argued, while our recommender was trained to predict user engagement. The LLM had never seen our proprietary interaction data and had no training signal on our objective. It could not know what our users clicked on, saved or abandoned, because it had never seen it.
It was a close call, but the pushback worked, and our attention moved to other problem spaces where GenAI was a legitimately strong solution.
It doesn’t always work out that way. There needs to be someone in the room who can translate the business strategy into the right technical decision.
I recently talked with the engineering team at a company I used to work for. They were building agentic AI infrastructure, but their algorithms weren’t winning. Under the same hype and the same leadership pressure, they had replaced ML models we built years earlier with AI agents. The results were unreliable, latency was much higher and outputs lacked a confidence score. Worst of all, the token usage was prohibitive.
Even the most powerful tool fails when applied to the wrong problem. LLMs are arguably the most powerful technology ever built, but that does not make them the right tool for every job. Choosing where a problem sits along the continuum from deterministic logic to machine learning to LLMs to fully agentic systems is the engineering skill that separates demos from production.
LLMs for explanation, ML for calibration
I’ve made that mistake too. Our team was building a content moderation model that would flag when customers posted content that was inappropriate or violated our terms of service. After a quick prototype, we decided that this was a perfect application of GenAI, because unlike ML algorithms, which only assign a probability, LLMs could explain why the message was flagged.
As we moved the model closer to production, we hit a snag. The probability value provided by the model was not just useful; it also needed to be accurate. That number was how we decided which content should go to manual review. But we found that self-reported probabilities from LLMs were uncalibrated and essentially unusable in practice.
Eventually we built a solution that leveraged both: the ML model provided the probabilities used for routing, and the LLM supplied the explanation that humans were able to interpret. The answer was to split one job into two, and use the right technology for each.
Simple logic often beats the most powerful agentic system
A key principle when designing agentic systems is to use LLMs as a last resort, and apply deterministic logic everywhere you can. The goal is to reduce the likelihood of an unnecessary mistake, thus increasing overall reliability.
Nowhere is this principle more overlooked than when connecting AI agents directly to the database. Querying a database to pull metrics is often done via SQL, and LLMs can do that by generating SQL on the fly. The problem is that SQL generation is probabilistic, and the results can change from one run to the next. You end up with an agent that confidently returns the wrong number, which introduces the need for human review and defeats the purpose of automation.
There’s plenty of data to support this claim. A 2024 study by Ouyang et al. ran 829 coding problems through the same model five times and found that up to three quarters of them produced no semantically identical results. For data warehouses specifically, the BEAVER benchmark, built by researchers at MIT, Harvard and other renowned institutions, shows that off-the-shelf LLMs perform poorly when querying enterprise data, partly because that data is private and models have never trained on it and partly because of the complexity of real enterprise environments. As of early 2026, the top execution accuracy on the leaderboard is 11.4 percent.
This is why the semantic layer is one of the most useful assets when building agents in the real world. It replaces query generation with a simple retrieval, where the SQL is built by the backend engine in a deterministic manner.
Whether using a semantic layer or building MCP tools that hard-code query parameters, deterministic logic will almost always beat probabilistic SQL generation when querying a database via an agent. Business logic should be written and validated once, not regenerated probabilistically each time.
LLMs solve the cold-start problem
Search systems are complex pieces of engineering. Sophisticated real-world implementations involve several steps, from classifying intent to candidate retrieval, to ranking. Here is where LLMs provided a clever solution that saved us a ton of time when combined with an ML algorithm.
Let’s take intent classification. Imagine multiple categories of products that can be retrieved from the same search bar. Classifying intent here means determining which category the user is searching for, which can be genuinely ambiguous. Our solution was to build one classifier per category, which requires labeled data we didn’t have.
Collecting the data was out of the question for us. It would have been too expensive, so we turned to LLMs instead. It worked great. The classification was high quality, and we felt that we no longer needed labeled data to train an ML model.
But there was a catch. Latency was prohibitive, and so was the token cost. So, we landed on a hybrid approach. We used the LLM to generate labeled data to train an ML classifier. We got the best of both worlds, solving the cold-start problem while avoiding the latency and costs of large language models. That’s the system that ultimately won in production. The LLM earned its place at build time, not at request time.
The continuum, and how to choose
It is tempting to take a powerful solution, such as LLMs and AI agents, and simply apply it to every problem. But when we look at the four use cases above, each of them led to a different decision. In the recommender system, GenAI did not earn its place. In content moderation, it earned part of the job, providing the explanation while the ML model provided the score. In querying the database, the agent did the reasoning, while the semantic layer pulled the right number. And in search, the LLM solved the cold-start problem, but it was never deployed to production.
Behind all of these decisions is the same underlying question: where does this technology earn its place, and does it justify the added complexity?
Production systems often combine multiple levels of algorithmic intelligence. A customer service agent uses an ML model for query routing, a semantic layer for metric retrieval, an LLM for drafting a reply and deterministic logic as enforced guardrails. Four technologies on the continuum, each earning the complexity it brings.
Business pressure is a powerful force, and it can cloud engineering judgment. As leaders, we must filter through the hype and look for the problems where a new technology genuinely adds value, rather than treating it as a tool that will solve all of them. The careless decision satisfies the board but hurts the company; the thoughtful one creates lasting value and still earns the board’s approval.
Read More from This Article: Not every problem needs an AI agent
Source: News